United States: References Class Forecasting and Realistic Tech Contracts

Originally published in Society of Computers and the Law on March 2, 2017.

Michael Bywell suggests one route to overcoming cognitive delusion, deception and strategic manipulation in technology projects. He suggests that reference class forecasting may provide an answer and considers the legal implications.

* * * * *

In this article I revisit the pervasive problem of time and cost overruns in ICT projects – but from a different perspective.

I look at the work of leading scientists, economists and management consultants in trying to understand why these areas remain problematic and establish what can be done to ensure better outcomes. In particular, why is it that project planners routinely fail to produce realistic estimates of how long a project will take and how much it will cost even though they know, or should know, that similar projects have overrun in the past?

I look at reference class forecasting (RCF) and whether asking planners to compare their working estimates to similar past projects is likely to improve outcomes.

Finally, I consider the potential challenges associated with applying RCF in practice together with some of the related legal issues that could arise in the context of ICT projects.

I am a lawyer (who knows his limits) and not a scientist, economist or management consultant. Considerable reliance has therefore been placed on the published work of leading scholars such as Professor Bent Flyvbjerg of Oxford University's Saïd Business School and his colleague, Alexander Budzier. This description of RCF and how it works in practice is fairly rudimentary but I hope that it is true to the underlying theory and serves as a catalyst for discussion on whether RCF should be adopted by the ICT industry.

ICT projects – Overruns and Disputes

One does not need to look too far to find compelling evidence that time and cost overruns remain highly problematic in ICT projects.

Take, for example, the often cited study by Flyvbjerg and Budzier which is summarised in the September 2011 issue of the Harvard Business Review. Flyvbjerg and Budzier looked at 1,471 IT projects from around the world, comparing budgets and estimated performance benefits with actual costs and results. They found that, whilst the average cost overrun was 27%, one in six projects was a 'Black Swan' with a cost overrun of 200% and scheduling overrun (or delay) of 70%.

In a separate article on 'Megaprojects'1Flyvbjerg gave several examples of high-profile projects with substantial cost overruns, including the NHS IT system where the overrun was a staggering 400-700%.

And ICT projects are not alone. Cost overruns for the Channel Tunnel and Sydney Opera House (going way back) were 80% and 1400% respectively.

Other work undertaken by Flyvbjerg (in relation to construction projects) found that, on average, a one-year delay correlates with an increase in percentage cost overruns of 4.64%. Thus a one-year delay on London's $26bn cross-rail project would result in cost overruns of $3.3m per day.2

In a more recent study, by the Queen Mary School of International Arbitration in London and Pinsent Masons,3 49% of respondents said they had encountered 'IT systems development/implementation/integration' or 'outsourcing' disputes in the past five years. And 90% of respondents said that 'delay' was a common reason for 'IT systems disputes.'

Not all such projects are problematic and fail. Many do achieve technological success and make it across the finish line - but often at a significantly higher cost, later than originally planned and in a form different to that originally agreed.

Why is this and how do we explain the tendency of planners to routinely underestimate costs and over-estimate benefits when planning ICT projects? Why do projects get the go-ahead when it is (or at least should be) obvious that they are bound to overrun?

Cognitive Predisposition and the Inside View

According to a number of well-respected scholars (including Flyvberg) one explanation is that key participants involved in the planning of such projects are deluded (but genuinely mistaken) when they prepare their estimates. According to the experts, this deluded state of mind (or optimism bias) is caused by a cognitive predisposition found in most people to judge future events in a more positive light than is warranted by actual experience. This can give rise to a tendency to underestimate the time and cost to complete a project, even where people may know that similar projects have overrun in the past.

Indeed optimism bias pervades our everyday life and affects around 80% of the population. Thus 80% of us will likely predict deriving greater pleasure from a holiday than we subsequently do and although divorce rates in the Western World are nearing 50% most newlyweds estimate their own likelihood of divorce as negligible. Even experts show high levels of optimism bias: financial analysts expect improbably high profits and doctors overestimate the effectiveness of their treatment. Indeed the way in which our brains code positive information (it is good at that) and actual errors in prediction (not so good at that) means that we are programmed this way.4 On the plus side it has also been shown that optimistic people live longer.

Where optimism bias occurs, project planners commonly adopt an 'inside view' when preparing estimates for a new project – a view based on their own past experiences and the task in front of them. They do not pay sufficient attention to actual project outcomes or the 'outside view' which (according to Flyvberg and others) is more likely to produce realistic estimates and outcomes.

In an ICT context this could apply to a planner working for a customer or someone on the bid team of a vendor.

Strategic Manipulators

However, some mistakes can be deliberate (ie rather than genuine or honest) and involve deception or strategic manipulation (in the sense those terms are used by economists and management consultants). This behaviour can arise when planners deliberately and in a strategic manner understate cost estimates and exaggerate benefits in order to obtain approval and funding for a project or, on the contractor/supplier side, to (try to) win the contract.

Such behaviour of individuals is most likely due to political and organisational pressures placed on them in their roles as planners or bid managers. Where such individuals are also 'project champions' and involved throughout they may find it hard to turn back or re-set the project even when it runs into difficulties and they know it is wrong to carry on.

The Outside View and Reference Class Forecasting

One method of overcoming these challenges is RCF.

RCF compares the planned project with a reference class of similar and completed projects. Adjustments to estimates can be made depending on the outcome of the analysis undertaken. RCF relies on hard, statistical data rather than the planners own, ground-up assessment (or inside view) of how the project should be estimated. The detail of the proposed project is largely ignored. According to Flyvbjerg:5

'When both forecasting methods [inside view and outside view] are applied with equal skill, the outside view is much more likely to produce a realistic estimate...That is because the outside view bypasses cognitive and political biases such as optimism bias and strategic manipulation and cuts directly to outcomes.'

Here is a simple illustration:

Company C wants a new IT system. Based on anecdotal information C knows that the type of system they have in mind is commonplace but that getting the solution right for individual businesses can be a challenge. Projects can easily overrun. As part of the planning process C decides to undertake RCF.

C engages a firm of consultants to gather data on similar past projects and prepare the reference class.6The consultants gather the data from their own ad hoc database and other, publicly available, information on similar completed projects.

Armed with this information C then establishes a probability distribution for the selected reference class. This is based on data from ten past projects including actual time and cost to complete.7

Finally, C places the current project at an appropriate point in the reference class distribution so that any probable shortfalls can be identified and related adjustments made to existing estimates.

For example, if the proposed project is an average project (when considered against the reference class), C can expect the final outcomes including cost increases to be in accordance with those experienced by other average projects.

Put another way, the proposed project is bench-marked against past completed projects of a similar type to check whether the planner's own 'inside view' is accurate or not. Adjustments are then made if needed. This could be in the form of an optimism bias budget uplift.

Indeed that is exactly what the UK Department of Transport did back in 2004 (with help from Flyvberg).8

RCF: Not a New Concept

RCF is not a new concept. It has its origins in theories developed by Daniel Kahneman and Amos Tversky which, in due course, helped them win the Nobel prize for economics in 2002.

As we have just seen, RCF was adopted by the UK Department of Transport back in 2004 and, according to Flyvbjerg, similar steps were taken by the Swiss Association of Road and Transportation Experts and the Danish Ministry for Transport – in 2006 and 2008 respectively.9

However, despite these (and other) examples of where RCF has been deployed the overall take-up appears fairly modest – certainly based on research done for the purposes of this article. This may reflect some of the potential challenges involved in applying RCF in practice.

Potential Challenges

Where do willing parties get the RCF data from?

This appears to be a key challenge. The theory is sound but how can willing parties routinely get hold of the data they need to formulate an 'outside view?' Not everyone will have access to the information they need for this purpose.

In the ICT area, governments and large corporate organisations that have undertaken multiple projects may have data sets they can use - provided the data is well-organised (by project type, location, size, time taken and actual cost incurred) and accessible.

That said, even these larger organisations may struggle to find ten or more similar projects (ie a valid reference class) each and every time they want to undertake RCF.

Others with useful data sets might include Flyvbjerg and Budzier themselves. Their study of 1,471 IT projects sounds like a very useful starting point – plus any subsequent data they may have gathered since that study was undertaken.

Organisations such as Gartner may also hold useful information - so too Standish Group and other international organisations and consultancies who, from time to time, gather and/or publish statistics relating to the ICT industry. However, smaller organisations may not have enough or any data to rely on.

Master Data Set?

Notwithstanding the probable existence of ad hoc data sets (such as those just described), there does not appear to be a global or master data set currently available to the ICT industry. In other words, a giant data set that all parties wishing to undertake RCF could access and utilise.

Is the creation of a shared database worthy of further consideration given the potential benefits for the industry as a whole? And could this provide an opportunity for any organisation prepared to take it on?

In my view, unless and until something like this is done, attempts by potential users to deploy RCF are likely to be constrained by the limited availability of reference class data.

The basis upon which data could be shared from a commercial standpoint is beyond the scope of this article. That said, any concerns over confidentiality should be capable of resolution through, for example, the anonymization of data and the appropriate confidentiality agreements.

Overcoming Resistance from Strategic Manipulators

As already noted, some planning mistakes will be genuine. Planners may not realise they are experiencing optimism bias. Where this occurs the chances of getting buy-in to using RCF should be high. Why would an honest planner want to achieve anything other than a better and improved forecast?

On the other hand, where deception or strategic manipulation occurs, there may be resistance to using RCF on a project. If a planner's inaccuracy is deliberate (to win funding or a new piece of business) they may not welcome a process such as RCF if it is likely to reveal the truth (ie that the working estimates are wrong and the project may fail as a result).

Wider adoption of RCF, and making it mandatory where appropriate, should help overcome this challenge. Strategic manipulators would then have no choice but to participate and abide by the outcomes.

Even without blanket adoption there is nothing to prevent senior executives and other decision-makers from insisting that RCF be used on a case-by-case basis.

Risk of Abuse?

There may be a risk that people become complacent in situations where a project budget is increased or time-scales extended as a result of RCF. If there is plenty of added contingency (to put it simply), project participants may simply play to those upper limits rather than maintaining efficiencies and keeping things tight. Is there a chance that the project will still overrun but against higher thresholds or upper limits?

This is where the project managers and lawyers (can and should) earn their keep. Good project management and the inclusion of realistic and enforceable legal sanctions in contracts between the parties to these projects should (in theory at least) help mitigate this sort of situation.

Place More Risk on Contractors?

In a separate article published in 2014, Flyvbjerg and his co-authors10 suggested that 'to ensure responsibility' the financial risk of delay and cost overruns should be placed with contractors who bid on the project – in order to mitigate situations where the winning bidder submits a low price in the expectation that they can make up the difference as the scope increases.11 They appear to suggest that this risk allocation will incentivise contractors to be more honest at the outset.

I agree with this - although the probable reality is that a contractor (or supplier) will fight back and say that the reason for the scope increase was changing requirements on the part of the customer and not under-pricing when the deal was done. This is so often the battleground in ICT legal disputes.

However, if it can be shown that the original pricing and/or timescales (which tend to inter-relate) were always wrong (based perhaps on RCF undertaken 'after the event' to see what the original estimate should have been) then the contractor may be liable, particularly if the original estimates were hopelessly understated and there is limited or no evidence that a robust estimating process was conducted by the contractor when preparing its bid. In that instance, the contractor could face a claim based on fraudulent misrepresentation.

But what happens in a misrepresentation claim where the customer has undertaken RCF? Could this let the contractor off the hook? Could it be said that the customer relied on its own forensic estimates and not those of the contractor?

Is the position further complicated if both parties performed RCF (independently or on a collaborative basis) and agreed pricing and time-scales based on that information?

In these situations does it become harder for customer organisations to argue for compensation (and any other legal remedies sought) based on misrepresentation? If so, should customers de-risk the situation by leaving it entirely up to bidders to undertake RCF? Or would that risk undermining the original purpose of the exercise, namely for the customer, as the project initiator and gate-keeper, to form its own views on time and cost to complete?

Conclusions

RCF is not (nor is it intended to be) a 'silver bullet' solution to the problem of ICT project failure and there are practical challenges associated with it. However the theory does make a lot of sense and one can see how it might help improve outcomes, particularly when coupled with other (well-documented) good practices such as:

(i) breaking up larger projects into smaller, staged projects;

(ii) treating the project as a business endeavour (not a technical one);

(iii) recruiting and keeping good people to perform the work;

(iv) making people and organisations accountable (including those engaged to perform RCF if they get it wrong); and

(v) incentivising project participants to be realistic and honest.

In summary, potential benefits include:

  • preventing parties from embarking on unachievable projects – or at least giving them a better sense of the actual risks involved so that adjustments can be made at the outset;
  • lending more science to the art of estimating;
  • creating more certainty on scope (ie the ability to set realistic requirements based on past similar projects);
  • a transparent approach that levels the playing field by, for example, making it harder for bidders to under-price to win business; and
  • fewer project failures.

Finally, in this age of big data, surely there must be scope to collect, collate and organise historical ICT project data from around the world and make it accessible to project planners across the industry? In other words, if the ad hoc collections of project data that currently exist can be pooled, shared and added to over time, RCF will become more accessible and the benefits for the industry as a whole could be spectacular.


  1. 'What you should know about megaprojects and why: an overview – Draft 9.2, 2014, p 9'
  2. ibid, p 10
  3. 'Pre-empting and resolving Technology, Media & Telecommunications disputes: International Dispute Resolution Survey, November 2016'
  4. These statistics and outcomes are from an article by Tali Sharot of UCL London dated 6 December 2011 and published by Cell Press
  5. 'Quality control and due diligence in project management: getting decisions right by taking the outside view – November 2012,' version 5.2, page 6
  6. The potential challenges associated with getting access to enough reliable reference class data are discussed below.
  7. According to Flyvbjerg ten or more projects should normally make up a reference class. This statement appears in 'The British Department of Transport – Procedures for dealing with optimism bias in transport planning – Guidance Document – June 2004.' That document was authored by Flyvbjerg.
  8. See 'The British Department of Transport – Procedures for dealing with optimism bias in transport planning – Guidance Document – June 2004.'

  9. 'Quality control and due diligence in project management: getting decisions right by taking the outside view – November 2012,' version 5.2, page 23.'
  10. 'Better forecasting for large capital projects – Flyvbjerg, Garbulo and Lovallo, December 2014 published by McKinsey & Co.'
  11. I have also heard this described as 'land and expand'.

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.

To print this article, all you need is to be registered on Mondaq.com.

Click to Login as an existing user or Register so you can print this article.

Authors
 
In association with
Related Video
Up-coming Events Search
Tools
Print
Font Size:
Translation
Channels
Mondaq on Twitter
 
Register for Access and our Free Biweekly Alert for
This service is completely free. Access 250,000 archived articles from 100+ countries and get a personalised email twice a week covering developments (and yes, our lawyers like to think you’ve read our Disclaimer).
 
Email Address
Company Name
Password
Confirm Password
Position
Mondaq Topics -- Select your Interests
 Accounting
 Anti-trust
 Commercial
 Compliance
 Consumer
 Criminal
 Employment
 Energy
 Environment
 Family
 Finance
 Government
 Healthcare
 Immigration
 Insolvency
 Insurance
 International
 IP
 Law Performance
 Law Practice
 Litigation
 Media & IT
 Privacy
 Real Estate
 Strategy
 Tax
 Technology
 Transport
 Wealth Mgt
Regions
Africa
Asia
Asia Pacific
Australasia
Canada
Caribbean
Europe
European Union
Latin America
Middle East
U.K.
United States
Worldwide Updates
Check to state you have read and
agree to our Terms and Conditions

Terms & Conditions and Privacy Statement

Mondaq.com (the Website) is owned and managed by Mondaq Ltd and as a user you are granted a non-exclusive, revocable license to access the Website under its terms and conditions of use. Your use of the Website constitutes your agreement to the following terms and conditions of use. Mondaq Ltd may terminate your use of the Website if you are in breach of these terms and conditions or if Mondaq Ltd decides to terminate your license of use for whatever reason.

Use of www.mondaq.com

You may use the Website but are required to register as a user if you wish to read the full text of the content and articles available (the Content). You may not modify, publish, transmit, transfer or sell, reproduce, create derivative works from, distribute, perform, link, display, or in any way exploit any of the Content, in whole or in part, except as expressly permitted in these terms & conditions or with the prior written consent of Mondaq Ltd. You may not use electronic or other means to extract details or information about Mondaq.com’s content, users or contributors in order to offer them any services or products which compete directly or indirectly with Mondaq Ltd’s services and products.

Disclaimer

Mondaq Ltd and/or its respective suppliers make no representations about the suitability of the information contained in the documents and related graphics published on this server for any purpose. All such documents and related graphics are provided "as is" without warranty of any kind. Mondaq Ltd and/or its respective suppliers hereby disclaim all warranties and conditions with regard to this information, including all implied warranties and conditions of merchantability, fitness for a particular purpose, title and non-infringement. In no event shall Mondaq Ltd and/or its respective suppliers be liable for any special, indirect or consequential damages or any damages whatsoever resulting from loss of use, data or profits, whether in an action of contract, negligence or other tortious action, arising out of or in connection with the use or performance of information available from this server.

The documents and related graphics published on this server could include technical inaccuracies or typographical errors. Changes are periodically added to the information herein. Mondaq Ltd and/or its respective suppliers may make improvements and/or changes in the product(s) and/or the program(s) described herein at any time.

Registration

Mondaq Ltd requires you to register and provide information that personally identifies you, including what sort of information you are interested in, for three primary purposes:

  • To allow you to personalize the Mondaq websites you are visiting.
  • To enable features such as password reminder, newsletter alerts, email a colleague, and linking from Mondaq (and its affiliate sites) to your website.
  • To produce demographic feedback for our information providers who provide information free for your use.

Mondaq (and its affiliate sites) do not sell or provide your details to third parties other than information providers. The reason we provide our information providers with this information is so that they can measure the response their articles are receiving and provide you with information about their products and services.

If you do not want us to provide your name and email address you may opt out by clicking here .

If you do not wish to receive any future announcements of products and services offered by Mondaq by clicking here .

Information Collection and Use

We require site users to register with Mondaq (and its affiliate sites) to view the free information on the site. We also collect information from our users at several different points on the websites: this is so that we can customise the sites according to individual usage, provide 'session-aware' functionality, and ensure that content is acquired and developed appropriately. This gives us an overall picture of our user profiles, which in turn shows to our Editorial Contributors the type of person they are reaching by posting articles on Mondaq (and its affiliate sites) – meaning more free content for registered users.

We are only able to provide the material on the Mondaq (and its affiliate sites) site free to site visitors because we can pass on information about the pages that users are viewing and the personal information users provide to us (e.g. email addresses) to reputable contributing firms such as law firms who author those pages. We do not sell or rent information to anyone else other than the authors of those pages, who may change from time to time. Should you wish us not to disclose your details to any of these parties, please tick the box above or tick the box marked "Opt out of Registration Information Disclosure" on the Your Profile page. We and our author organisations may only contact you via email or other means if you allow us to do so. Users can opt out of contact when they register on the site, or send an email to unsubscribe@mondaq.com with “no disclosure” in the subject heading

Mondaq News Alerts

In order to receive Mondaq News Alerts, users have to complete a separate registration form. This is a personalised service where users choose regions and topics of interest and we send it only to those users who have requested it. Users can stop receiving these Alerts by going to the Mondaq News Alerts page and deselecting all interest areas. In the same way users can amend their personal preferences to add or remove subject areas.

Cookies

A cookie is a small text file written to a user’s hard drive that contains an identifying user number. The cookies do not contain any personal information about users. We use the cookie so users do not have to log in every time they use the service and the cookie will automatically expire if you do not visit the Mondaq website (or its affiliate sites) for 12 months. We also use the cookie to personalise a user's experience of the site (for example to show information specific to a user's region). As the Mondaq sites are fully personalised and cookies are essential to its core technology the site will function unpredictably with browsers that do not support cookies - or where cookies are disabled (in these circumstances we advise you to attempt to locate the information you require elsewhere on the web). However if you are concerned about the presence of a Mondaq cookie on your machine you can also choose to expire the cookie immediately (remove it) by selecting the 'Log Off' menu option as the last thing you do when you use the site.

Some of our business partners may use cookies on our site (for example, advertisers). However, we have no access to or control over these cookies and we are not aware of any at present that do so.

Log Files

We use IP addresses to analyse trends, administer the site, track movement, and gather broad demographic information for aggregate use. IP addresses are not linked to personally identifiable information.

Links

This web site contains links to other sites. Please be aware that Mondaq (or its affiliate sites) are not responsible for the privacy practices of such other sites. We encourage our users to be aware when they leave our site and to read the privacy statements of these third party sites. This privacy statement applies solely to information collected by this Web site.

Surveys & Contests

From time-to-time our site requests information from users via surveys or contests. Participation in these surveys or contests is completely voluntary and the user therefore has a choice whether or not to disclose any information requested. Information requested may include contact information (such as name and delivery address), and demographic information (such as postcode, age level). Contact information will be used to notify the winners and award prizes. Survey information will be used for purposes of monitoring or improving the functionality of the site.

Mail-A-Friend

If a user elects to use our referral service for informing a friend about our site, we ask them for the friend’s name and email address. Mondaq stores this information and may contact the friend to invite them to register with Mondaq, but they will not be contacted more than once. The friend may contact Mondaq to request the removal of this information from our database.

Security

This website takes every reasonable precaution to protect our users’ information. When users submit sensitive information via the website, your information is protected using firewalls and other security technology. If you have any questions about the security at our website, you can send an email to webmaster@mondaq.com.

Correcting/Updating Personal Information

If a user’s personally identifiable information changes (such as postcode), or if a user no longer desires our service, we will endeavour to provide a way to correct, update or remove that user’s personal data provided to us. This can usually be done at the “Your Profile” page or by sending an email to EditorialAdvisor@mondaq.com.

Notification of Changes

If we decide to change our Terms & Conditions or Privacy Policy, we will post those changes on our site so our users are always aware of what information we collect, how we use it, and under what circumstances, if any, we disclose it. If at any point we decide to use personally identifiable information in a manner different from that stated at the time it was collected, we will notify users by way of an email. Users will have a choice as to whether or not we use their information in this different manner. We will use information in accordance with the privacy policy under which the information was collected.

How to contact Mondaq

You can contact us with comments or queries at enquiries@mondaq.com.

If for some reason you believe Mondaq Ltd. has not adhered to these principles, please notify us by e-mail at problems@mondaq.com and we will use commercially reasonable efforts to determine and correct the problem promptly.