Planning & Weighting

...to get the right outcome


Colin Lange - 01 September 2004.

Procurement Plans

Procurement Plans identify what is to be purchased, who the stakeholders are, how the purchase will be arranged, where the supply may come from and most importantly why the purchase is going ahead, i.e. what results (outcomes) are expected. Unless the reasons and anticipated results of a purchase are well understood by all stakeholders, any subsequent work to identify and assign weight to evaluation criteria may be disconnected.

Weighting

Evaluation criteria are usually identified within procurement documents. When trying to choose between competing bids, the bid which best ‘fits’ the evaluation criteria would be preferred. As some criteria ‘mean more’ than others, we reflect this relative importance through assigning weight to individual criteria.

Assigning weight is a key procurement step and must be done in a way that is both (a) efficient; to not unduly delay the process and (b) defensible; should the weights later come under question. More importantly though is the need to assign weight to criteria such that the purpose of the evaluation process to follow is confidently met, i.e. weight should be assigned to assist in differentiating between competing bids.

Before you start

Weight should be assigned to evaluation criteria by staff with good prior awareness of the corporate and / or strategic requirements within which the procurement activity is occurring. I refer to this as ‘procurement context’ and unless this context is well understood weights may be assigned contrary to the corporate and / or strategic requirements.

If evaluation team members are not aware of the results (outcomes) expected to be achieved, then the work they carry out later to assign weight to evaluation criteria may be unduly, but quite reasonably, influenced by their particular specialist skills rather than being balanced by a good understanding of corporate and / or strategic requirements.

A common way to assign weight

Weight may be assigned by:

Seems simple? While this approach seems simple it is in fact open to allegations by a disgruntled bidder that the assignment of weight was quite arbitrary and unduly disadvantageous to them. I refer to weights determined by discussion as ‘manual weights’ to distinguish between them and weights determined by a statistical approach.

If you use manual weights then clearly document the process undertaken and perhaps invite your probity advisor as an observer.

To make the manual weights ‘more effective’ in assisting to differentiate between competing tenders, always try to assign around 80% of weight against the most important evaluation criteria. This ‘rule of thumb’ will mean a tenderer will not be highly ranked when it meets less important criteria to a high degree unless it also ‘does well’ against the most important evaluation criteria.

A better way to assign weight

Pair-wise comparison was developed as a solution to the conundrum of how to “determine the relative importance of discrete objects” – in this case a discrete object is an evaluation criterion. Evaluation criteria are compared a pair at a time and these results are entered into a matrix which ‘produces’ the weights.

Pair-wise comparison is (a) efficient, groups can assign weight exceptionally quickly, and (b) ‘more scientific’ than manual weights as the focus is upon relative importance (something people can reasonably discuss) rather than weights (which can be distracting and cause undue delay and dissent).

The use of pair-wise comparison matrices must ensure they do not fall victim to an internal inconsistency where a first criterion is considered more important than a second, that second criterion is then considered more important than a third but the third criterion is considered more important than the first. This error is referred to as a ‘circular triad’.

Pair-wise comparison matrices may be used crudely or well¹. Weights generated by a crude matrix usually do not adequately reflect true relative importance and are characterised by being roughly similar to each other. Such weights are not as helpful in differentiating between competing tenders, as a tenderer who doesn’t meet the key criteria may compensate by fully complying with less important criteria. I refer to this as the ‘swings and roundabouts effect’.

My company supplies a software system (Apetâ) to manage the assignment of weight to evaluation criteria through both the manual method and pair-wise comparison. Apet’s pair-wise comparison does not suffer from the circular triad problem and is referred to as HWPâ (Hierarchy Weighting Process).

Apet uses a ‘better’ pair-wise comparison matrix generating fully auditable weights geared to produce tenderer rankings, which confidently differentiate between competing tenders.

A new approach to weighting

Emerging procurement evaluation best practice suggests:

This approach, espoused by Dr Des Klass of the Curtin Graduate School of Management, was successfully implemented during many procurement consultancies. It places more emphasis on a criterion’s relative potential to differentiate rather than its relative importance. The fundamental purpose of a procurement evaluation is to reach a point where it is possible to confidently differentiate tenders based on value for money. Hence it is becoming increasingly popular to assign weight to evaluation criteria with this result in mind.

¹ The Analytic Hierarchy Process, Thomas L. Saaty, 1980 p19