An effective and economical evaluation is a vital part of achieving, and demonstrating the achievement of, value for money in a government approach to the market.
Done well, the process will document, on the face of the record, compliance with the legislative and procurement policy framework and legal process and probity requirements.
Our recent experience in numerous evaluations has revealed the following common mistakes in evaluation:
- Not having a proper methodology for treating risk, particularly
double counting risk
Double counting risk will provide an inaccurate summation of the whole evaluation, as emphasis will be placed unnecessarily on a given risk category. risk should be assessed in the areas of technical Worth, price and legal (relating to compliance with the deed of standing offer and issues for negotiation).
A risk matrix detailing ?Measures of likelihood? and ?Measures of consequences/impact? following usual risk management principles can be used.
Our experience indicates that it is important to comply with the iso310000 risk standard, however anZs 4360 is still very useful and practical. Accordingly, a risk description matrix following the anZs 4360 standard, guidelines and handbook modified for the needs of the project and covering Measures of likelihood and Measures of consequence/impact are generally very useful.
- Including a risk description in your rating scale for
evaluating merit.
a response with high technical merit may not always be low risk. the two issues are separate and generally need to be treated separately.
- scoring by numbers rather than by descriptions.
As indicated above, scoring by numbers is often not helpful and fails to describe properly why a particular rating has been determined. Assessing and rating primarily by reference to numbers rather than word pictures opens an evaluation to criticism or challenge.
- Inadequate word pictures - not lining up with the rating scale.
There is always subjectivity in evaluation. Word pictures are critical in demonstrating an absence of bias and form the evidential basis for judgments made. Reliance on numbers can become subjective and any scoring should be adequately described and referenced.
- Not structuring process and templates so assessments,
evaluation reports and debriefings can be easily done from the same
data source.
Utilising the same data source in an assessment is essential given it allows the user to refer back to an agreed base in order to measure appropriateness of evaluation. This will avoid any inaccuracies on measurements and ensure effectiveness of the probity and evaluation process.
- Poor use of evaluation templates and/or evaluation software,
including not properly adapting them to your process/evaluation.
Understanding the software being utilised is essential and will dictate the efficiency and suitability of the evaluation plan. Very often software beyond simple excel spreadsheets (properly checked and verified) does not provide the structure or flexibility required for many evaluation processes.
- Inappropriate rating scales (too many points on the scale or
too few).
Utilising the correct amount of categories is important and this should be adapted for each evaluation having regard to the nature of the requirement and the risk involved.
- Not differentiating between panels and first past the post complex evaluations.
- Not following the evaluation plan or process and failing to manage and record steps and outcomes properly.
- Illegitimately double counting good points or bad points.
As noted above, double counting risk will provide an inaccurate summation of the whole evaluation given emphasis will be placed unnecessarily on a given risk category.
Our experience in designing and implementing evaluation processes highlights the value of having systematically decipherable methods and documentation which includes appropriate flexibility. This can significantly ease work load and increase consistency, efficiency and confidence in the evaluation.
The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.
