United States: References Class Forecasting and Realistic Tech Contracts

Originally published in Society of Computers and the Law on March 2, 2017.

Michael Bywell suggests one route to overcoming cognitive delusion, deception and strategic manipulation in technology projects. He suggests that reference class forecasting may provide an answer and considers the legal implications.

* * * * *

In this article I revisit the pervasive problem of time and cost overruns in ICT projects – but from a different perspective.

I look at the work of leading scientists, economists and management consultants in trying to understand why these areas remain problematic and establish what can be done to ensure better outcomes. In particular, why is it that project planners routinely fail to produce realistic estimates of how long a project will take and how much it will cost even though they know, or should know, that similar projects have overrun in the past?

I look at reference class forecasting (RCF) and whether asking planners to compare their working estimates to similar past projects is likely to improve outcomes.

Finally, I consider the potential challenges associated with applying RCF in practice together with some of the related legal issues that could arise in the context of ICT projects.

I am a lawyer (who knows his limits) and not a scientist, economist or management consultant. Considerable reliance has therefore been placed on the published work of leading scholars such as Professor Bent Flyvbjerg of Oxford University's Saïd Business School and his colleague, Alexander Budzier. This description of RCF and how it works in practice is fairly rudimentary but I hope that it is true to the underlying theory and serves as a catalyst for discussion on whether RCF should be adopted by the ICT industry.

ICT projects – Overruns and Disputes

One does not need to look too far to find compelling evidence that time and cost overruns remain highly problematic in ICT projects.

Take, for example, the often cited study by Flyvbjerg and Budzier which is summarised in the September 2011 issue of the Harvard Business Review. Flyvbjerg and Budzier looked at 1,471 IT projects from around the world, comparing budgets and estimated performance benefits with actual costs and results. They found that, whilst the average cost overrun was 27%, one in six projects was a 'Black Swan' with a cost overrun of 200% and scheduling overrun (or delay) of 70%.

In a separate article on 'Megaprojects'1Flyvbjerg gave several examples of high-profile projects with substantial cost overruns, including the NHS IT system where the overrun was a staggering 400-700%.

And ICT projects are not alone. Cost overruns for the Channel Tunnel and Sydney Opera House (going way back) were 80% and 1400% respectively.

Other work undertaken by Flyvbjerg (in relation to construction projects) found that, on average, a one-year delay correlates with an increase in percentage cost overruns of 4.64%. Thus a one-year delay on London's $26bn cross-rail project would result in cost overruns of $3.3m per day.2

In a more recent study, by the Queen Mary School of International Arbitration in London and Pinsent Masons,3 49% of respondents said they had encountered 'IT systems development/implementation/integration' or 'outsourcing' disputes in the past five years. And 90% of respondents said that 'delay' was a common reason for 'IT systems disputes.'

Not all such projects are problematic and fail. Many do achieve technological success and make it across the finish line - but often at a significantly higher cost, later than originally planned and in a form different to that originally agreed.

Why is this and how do we explain the tendency of planners to routinely underestimate costs and over-estimate benefits when planning ICT projects? Why do projects get the go-ahead when it is (or at least should be) obvious that they are bound to overrun?

Cognitive Predisposition and the Inside View

According to a number of well-respected scholars (including Flyvberg) one explanation is that key participants involved in the planning of such projects are deluded (but genuinely mistaken) when they prepare their estimates. According to the experts, this deluded state of mind (or optimism bias) is caused by a cognitive predisposition found in most people to judge future events in a more positive light than is warranted by actual experience. This can give rise to a tendency to underestimate the time and cost to complete a project, even where people may know that similar projects have overrun in the past.

Indeed optimism bias pervades our everyday life and affects around 80% of the population. Thus 80% of us will likely predict deriving greater pleasure from a holiday than we subsequently do and although divorce rates in the Western World are nearing 50% most newlyweds estimate their own likelihood of divorce as negligible. Even experts show high levels of optimism bias: financial analysts expect improbably high profits and doctors overestimate the effectiveness of their treatment. Indeed the way in which our brains code positive information (it is good at that) and actual errors in prediction (not so good at that) means that we are programmed this way.4 On the plus side it has also been shown that optimistic people live longer.

Where optimism bias occurs, project planners commonly adopt an 'inside view' when preparing estimates for a new project – a view based on their own past experiences and the task in front of them. They do not pay sufficient attention to actual project outcomes or the 'outside view' which (according to Flyvberg and others) is more likely to produce realistic estimates and outcomes.

In an ICT context this could apply to a planner working for a customer or someone on the bid team of a vendor.

Strategic Manipulators

However, some mistakes can be deliberate (ie rather than genuine or honest) and involve deception or strategic manipulation (in the sense those terms are used by economists and management consultants). This behaviour can arise when planners deliberately and in a strategic manner understate cost estimates and exaggerate benefits in order to obtain approval and funding for a project or, on the contractor/supplier side, to (try to) win the contract.

Such behaviour of individuals is most likely due to political and organisational pressures placed on them in their roles as planners or bid managers. Where such individuals are also 'project champions' and involved throughout they may find it hard to turn back or re-set the project even when it runs into difficulties and they know it is wrong to carry on.

The Outside View and Reference Class Forecasting

One method of overcoming these challenges is RCF.

RCF compares the planned project with a reference class of similar and completed projects. Adjustments to estimates can be made depending on the outcome of the analysis undertaken. RCF relies on hard, statistical data rather than the planners own, ground-up assessment (or inside view) of how the project should be estimated. The detail of the proposed project is largely ignored. According to Flyvbjerg:5

'When both forecasting methods [inside view and outside view] are applied with equal skill, the outside view is much more likely to produce a realistic estimate...That is because the outside view bypasses cognitive and political biases such as optimism bias and strategic manipulation and cuts directly to outcomes.'

Here is a simple illustration:

Company C wants a new IT system. Based on anecdotal information C knows that the type of system they have in mind is commonplace but that getting the solution right for individual businesses can be a challenge. Projects can easily overrun. As part of the planning process C decides to undertake RCF.

C engages a firm of consultants to gather data on similar past projects and prepare the reference class.6The consultants gather the data from their own ad hoc database and other, publicly available, information on similar completed projects.

Armed with this information C then establishes a probability distribution for the selected reference class. This is based on data from ten past projects including actual time and cost to complete.7

Finally, C places the current project at an appropriate point in the reference class distribution so that any probable shortfalls can be identified and related adjustments made to existing estimates.

For example, if the proposed project is an average project (when considered against the reference class), C can expect the final outcomes including cost increases to be in accordance with those experienced by other average projects.

Put another way, the proposed project is bench-marked against past completed projects of a similar type to check whether the planner's own 'inside view' is accurate or not. Adjustments are then made if needed. This could be in the form of an optimism bias budget uplift.

Indeed that is exactly what the UK Department of Transport did back in 2004 (with help from Flyvberg).8

RCF: Not a New Concept

RCF is not a new concept. It has its origins in theories developed by Daniel Kahneman and Amos Tversky which, in due course, helped them win the Nobel prize for economics in 2002.

As we have just seen, RCF was adopted by the UK Department of Transport back in 2004 and, according to Flyvbjerg, similar steps were taken by the Swiss Association of Road and Transportation Experts and the Danish Ministry for Transport – in 2006 and 2008 respectively.9

However, despite these (and other) examples of where RCF has been deployed the overall take-up appears fairly modest – certainly based on research done for the purposes of this article. This may reflect some of the potential challenges involved in applying RCF in practice.

Potential Challenges

Where do willing parties get the RCF data from?

This appears to be a key challenge. The theory is sound but how can willing parties routinely get hold of the data they need to formulate an 'outside view?' Not everyone will have access to the information they need for this purpose.

In the ICT area, governments and large corporate organisations that have undertaken multiple projects may have data sets they can use - provided the data is well-organised (by project type, location, size, time taken and actual cost incurred) and accessible.

That said, even these larger organisations may struggle to find ten or more similar projects (ie a valid reference class) each and every time they want to undertake RCF.

Others with useful data sets might include Flyvbjerg and Budzier themselves. Their study of 1,471 IT projects sounds like a very useful starting point – plus any subsequent data they may have gathered since that study was undertaken.

Organisations such as Gartner may also hold useful information - so too Standish Group and other international organisations and consultancies who, from time to time, gather and/or publish statistics relating to the ICT industry. However, smaller organisations may not have enough or any data to rely on.

Master Data Set?

Notwithstanding the probable existence of ad hoc data sets (such as those just described), there does not appear to be a global or master data set currently available to the ICT industry. In other words, a giant data set that all parties wishing to undertake RCF could access and utilise.

Is the creation of a shared database worthy of further consideration given the potential benefits for the industry as a whole? And could this provide an opportunity for any organisation prepared to take it on?

In my view, unless and until something like this is done, attempts by potential users to deploy RCF are likely to be constrained by the limited availability of reference class data.

The basis upon which data could be shared from a commercial standpoint is beyond the scope of this article. That said, any concerns over confidentiality should be capable of resolution through, for example, the anonymization of data and the appropriate confidentiality agreements.

Overcoming Resistance from Strategic Manipulators

As already noted, some planning mistakes will be genuine. Planners may not realise they are experiencing optimism bias. Where this occurs the chances of getting buy-in to using RCF should be high. Why would an honest planner want to achieve anything other than a better and improved forecast?

On the other hand, where deception or strategic manipulation occurs, there may be resistance to using RCF on a project. If a planner's inaccuracy is deliberate (to win funding or a new piece of business) they may not welcome a process such as RCF if it is likely to reveal the truth (ie that the working estimates are wrong and the project may fail as a result).

Wider adoption of RCF, and making it mandatory where appropriate, should help overcome this challenge. Strategic manipulators would then have no choice but to participate and abide by the outcomes.

Even without blanket adoption there is nothing to prevent senior executives and other decision-makers from insisting that RCF be used on a case-by-case basis.

Risk of Abuse?

There may be a risk that people become complacent in situations where a project budget is increased or time-scales extended as a result of RCF. If there is plenty of added contingency (to put it simply), project participants may simply play to those upper limits rather than maintaining efficiencies and keeping things tight. Is there a chance that the project will still overrun but against higher thresholds or upper limits?

This is where the project managers and lawyers (can and should) earn their keep. Good project management and the inclusion of realistic and enforceable legal sanctions in contracts between the parties to these projects should (in theory at least) help mitigate this sort of situation.

Place More Risk on Contractors?

In a separate article published in 2014, Flyvbjerg and his co-authors10 suggested that 'to ensure responsibility' the financial risk of delay and cost overruns should be placed with contractors who bid on the project – in order to mitigate situations where the winning bidder submits a low price in the expectation that they can make up the difference as the scope increases.11 They appear to suggest that this risk allocation will incentivise contractors to be more honest at the outset.

I agree with this - although the probable reality is that a contractor (or supplier) will fight back and say that the reason for the scope increase was changing requirements on the part of the customer and not under-pricing when the deal was done. This is so often the battleground in ICT legal disputes.

However, if it can be shown that the original pricing and/or timescales (which tend to inter-relate) were always wrong (based perhaps on RCF undertaken 'after the event' to see what the original estimate should have been) then the contractor may be liable, particularly if the original estimates were hopelessly understated and there is limited or no evidence that a robust estimating process was conducted by the contractor when preparing its bid. In that instance, the contractor could face a claim based on fraudulent misrepresentation.

But what happens in a misrepresentation claim where the customer has undertaken RCF? Could this let the contractor off the hook? Could it be said that the customer relied on its own forensic estimates and not those of the contractor?

Is the position further complicated if both parties performed RCF (independently or on a collaborative basis) and agreed pricing and time-scales based on that information?

In these situations does it become harder for customer organisations to argue for compensation (and any other legal remedies sought) based on misrepresentation? If so, should customers de-risk the situation by leaving it entirely up to bidders to undertake RCF? Or would that risk undermining the original purpose of the exercise, namely for the customer, as the project initiator and gate-keeper, to form its own views on time and cost to complete?

Conclusions

RCF is not (nor is it intended to be) a 'silver bullet' solution to the problem of ICT project failure and there are practical challenges associated with it. However the theory does make a lot of sense and one can see how it might help improve outcomes, particularly when coupled with other (well-documented) good practices such as:

(i) breaking up larger projects into smaller, staged projects;

(ii) treating the project as a business endeavour (not a technical one);

(iii) recruiting and keeping good people to perform the work;

(iv) making people and organisations accountable (including those engaged to perform RCF if they get it wrong); and

(v) incentivising project participants to be realistic and honest.

In summary, potential benefits include:

  • preventing parties from embarking on unachievable projects – or at least giving them a better sense of the actual risks involved so that adjustments can be made at the outset;
  • lending more science to the art of estimating;
  • creating more certainty on scope (ie the ability to set realistic requirements based on past similar projects);
  • a transparent approach that levels the playing field by, for example, making it harder for bidders to under-price to win business; and
  • fewer project failures.

Finally, in this age of big data, surely there must be scope to collect, collate and organise historical ICT project data from around the world and make it accessible to project planners across the industry? In other words, if the ad hoc collections of project data that currently exist can be pooled, shared and added to over time, RCF will become more accessible and the benefits for the industry as a whole could be spectacular.


  1. 'What you should know about megaprojects and why: an overview – Draft 9.2, 2014, p 9'
  2. ibid, p 10
  3. 'Pre-empting and resolving Technology, Media & Telecommunications disputes: International Dispute Resolution Survey, November 2016'
  4. These statistics and outcomes are from an article by Tali Sharot of UCL London dated 6 December 2011 and published by Cell Press
  5. 'Quality control and due diligence in project management: getting decisions right by taking the outside view – November 2012,' version 5.2, page 6
  6. The potential challenges associated with getting access to enough reliable reference class data are discussed below.
  7. According to Flyvbjerg ten or more projects should normally make up a reference class. This statement appears in 'The British Department of Transport – Procedures for dealing with optimism bias in transport planning – Guidance Document – June 2004.' That document was authored by Flyvbjerg.
  8. See 'The British Department of Transport – Procedures for dealing with optimism bias in transport planning – Guidance Document – June 2004.'

  9. 'Quality control and due diligence in project management: getting decisions right by taking the outside view – November 2012,' version 5.2, page 23.'
  10. 'Better forecasting for large capital projects – Flyvbjerg, Garbulo and Lovallo, December 2014 published by McKinsey & Co.'
  11. I have also heard this described as 'land and expand'.

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.

To print this article, all you need is to be registered on Mondaq.com.

Click to Login as an existing user or Register so you can print this article.

Authors
 
In association with
Related Topics
 
Related Articles
 
Related Video
Up-coming Events Search
Tools
Print
Font Size:
Translation
Channels
Mondaq on Twitter
 
Register for Access and our Free Biweekly Alert for
This service is completely free. Access 250,000 archived articles from 100+ countries and get a personalised email twice a week covering developments (and yes, our lawyers like to think you’ve read our Disclaimer).
 
Email Address
Company Name
Password
Confirm Password
Position
Mondaq Topics -- Select your Interests
 Accounting
 Anti-trust
 Commercial
 Compliance
 Consumer
 Criminal
 Employment
 Energy
 Environment
 Family
 Finance
 Government
 Healthcare
 Immigration
 Insolvency
 Insurance
 International
 IP
 Law Performance
 Law Practice
 Litigation
 Media & IT
 Privacy
 Real Estate
 Strategy
 Tax
 Technology
 Transport
 Wealth Mgt
Regions
Africa
Asia
Asia Pacific
Australasia
Canada
Caribbean
Europe
European Union
Latin America
Middle East
U.K.
United States
Worldwide Updates
Registration (you must scroll down to set your data preferences)

Mondaq Ltd requires you to register and provide information that personally identifies you, including your content preferences, for three primary purposes (full details of Mondaq’s use of your personal data can be found in our Privacy and Cookies Notice):

  • To allow you to personalize the Mondaq websites you are visiting to show content ("Content") relevant to your interests.
  • To enable features such as password reminder, news alerts, email a colleague, and linking from Mondaq (and its affiliate sites) to your website.
  • To produce demographic feedback for our content providers ("Contributors") who contribute Content for free for your use.

Mondaq hopes that our registered users will support us in maintaining our free to view business model by consenting to our use of your personal data as described below.

Mondaq has a "free to view" business model. Our services are paid for by Contributors in exchange for Mondaq providing them with access to information about who accesses their content. Once personal data is transferred to our Contributors they become a data controller of this personal data. They use it to measure the response that their articles are receiving, as a form of market research. They may also use it to provide Mondaq users with information about their products and services.

Details of each Contributor to which your personal data will be transferred is clearly stated within the Content that you access. For full details of how this Contributor will use your personal data, you should review the Contributor’s own Privacy Notice.

Please indicate your preference below:

Yes, I am happy to support Mondaq in maintaining its free to view business model by agreeing to allow Mondaq to share my personal data with Contributors whose Content I access
No, I do not want Mondaq to share my personal data with Contributors

Also please let us know whether you are happy to receive communications promoting products and services offered by Mondaq:

Yes, I am happy to received promotional communications from Mondaq
No, please do not send me promotional communications from Mondaq
Terms & Conditions

Mondaq.com (the Website) is owned and managed by Mondaq Ltd (Mondaq). Mondaq grants you a non-exclusive, revocable licence to access the Website and associated services, such as the Mondaq News Alerts (Services), subject to and in consideration of your compliance with the following terms and conditions of use (Terms). Your use of the Website and/or Services constitutes your agreement to the Terms. Mondaq may terminate your use of the Website and Services if you are in breach of these Terms or if Mondaq decides to terminate the licence granted hereunder for any reason whatsoever.

Use of www.mondaq.com

To Use Mondaq.com you must be: eighteen (18) years old or over; legally capable of entering into binding contracts; and not in any way prohibited by the applicable law to enter into these Terms in the jurisdiction which you are currently located.

You may use the Website as an unregistered user, however, you are required to register as a user if you wish to read the full text of the Content or to receive the Services.

You may not modify, publish, transmit, transfer or sell, reproduce, create derivative works from, distribute, perform, link, display, or in any way exploit any of the Content, in whole or in part, except as expressly permitted in these Terms or with the prior written consent of Mondaq. You may not use electronic or other means to extract details or information from the Content. Nor shall you extract information about users or Contributors in order to offer them any services or products.

In your use of the Website and/or Services you shall: comply with all applicable laws, regulations, directives and legislations which apply to your Use of the Website and/or Services in whatever country you are physically located including without limitation any and all consumer law, export control laws and regulations; provide to us true, correct and accurate information and promptly inform us in the event that any information that you have provided to us changes or becomes inaccurate; notify Mondaq immediately of any circumstances where you have reason to believe that any Intellectual Property Rights or any other rights of any third party may have been infringed; co-operate with reasonable security or other checks or requests for information made by Mondaq from time to time; and at all times be fully liable for the breach of any of these Terms by a third party using your login details to access the Website and/or Services

however, you shall not: do anything likely to impair, interfere with or damage or cause harm or distress to any persons, or the network; do anything that will infringe any Intellectual Property Rights or other rights of Mondaq or any third party; or use the Website, Services and/or Content otherwise than in accordance with these Terms; use any trade marks or service marks of Mondaq or the Contributors, or do anything which may be seen to take unfair advantage of the reputation and goodwill of Mondaq or the Contributors, or the Website, Services and/or Content.

Mondaq reserves the right, in its sole discretion, to take any action that it deems necessary and appropriate in the event it considers that there is a breach or threatened breach of the Terms.

Mondaq’s Rights and Obligations

Unless otherwise expressly set out to the contrary, nothing in these Terms shall serve to transfer from Mondaq to you, any Intellectual Property Rights owned by and/or licensed to Mondaq and all rights, title and interest in and to such Intellectual Property Rights will remain exclusively with Mondaq and/or its licensors.

Mondaq shall use its reasonable endeavours to make the Website and Services available to you at all times, but we cannot guarantee an uninterrupted and fault free service.

Mondaq reserves the right to make changes to the services and/or the Website or part thereof, from time to time, and we may add, remove, modify and/or vary any elements of features and functionalities of the Website or the services.

Mondaq also reserves the right from time to time to monitor your Use of the Website and/or services.

Disclaimer

The Content is general information only. It is not intended to constitute legal advice or seek to be the complete and comprehensive statement of the law, nor is it intended to address your specific requirements or provide advice on which reliance should be placed. Mondaq and/or its Contributors and other suppliers make no representations about the suitability of the information contained in the Content for any purpose. All Content provided "as is" without warranty of any kind. Mondaq and/or its Contributors and other suppliers hereby exclude and disclaim all representations, warranties or guarantees with regard to the Content, including all implied warranties and conditions of merchantability, fitness for a particular purpose, title and non-infringement. To the maximum extent permitted by law, Mondaq expressly excludes all representations, warranties, obligations, and liabilities arising out of or in connection with all Content. In no event shall Mondaq and/or its respective suppliers be liable for any special, indirect or consequential damages or any damages whatsoever resulting from loss of use, data or profits, whether in an action of contract, negligence or other tortious action, arising out of or in connection with the use of the Content or performance of Mondaq’s Services.

General

Mondaq may alter or amend these Terms by amending them on the Website. By continuing to Use the Services and/or the Website after such amendment, you will be deemed to have accepted any amendment to these Terms.

These Terms shall be governed by and construed in accordance with the laws of England and Wales and you irrevocably submit to the exclusive jurisdiction of the courts of England and Wales to settle any dispute which may arise out of or in connection with these Terms. If you live outside the United Kingdom, English law shall apply only to the extent that English law shall not deprive you of any legal protection accorded in accordance with the law of the place where you are habitually resident ("Local Law"). In the event English law deprives you of any legal protection which is accorded to you under Local Law, then these terms shall be governed by Local Law and any dispute or claim arising out of or in connection with these Terms shall be subject to the non-exclusive jurisdiction of the courts where you are habitually resident.

You may print and keep a copy of these Terms, which form the entire agreement between you and Mondaq and supersede any other communications or advertising in respect of the Service and/or the Website.

No delay in exercising or non-exercise by you and/or Mondaq of any of its rights under or in connection with these Terms shall operate as a waiver or release of each of your or Mondaq’s right. Rather, any such waiver or release must be specifically granted in writing signed by the party granting it.

If any part of these Terms is held unenforceable, that part shall be enforced to the maximum extent permissible so as to give effect to the intent of the parties, and the Terms shall continue in full force and effect.

Mondaq shall not incur any liability to you on account of any loss or damage resulting from any delay or failure to perform all or any part of these Terms if such delay or failure is caused, in whole or in part, by events, occurrences, or causes beyond the control of Mondaq. Such events, occurrences or causes will include, without limitation, acts of God, strikes, lockouts, server and network failure, riots, acts of war, earthquakes, fire and explosions.

By clicking Register you state you have read and agree to our Terms and Conditions