Urban Road User Charging Online Knowledge Base
What Appraisal Methods Are Available?
What are the principles of appraisal?
Appraisal of a policy against a single objective is relatively straightforward. Provided that the objective can be measured by an indicator, and performance against it predicted, the policy which performs best will be the one that generates the highest value of that indicator. It would be possible, for example, to appraise road pricing schemes in terms of their ability to generate revenue in this way. Taking this simple example further, it may be that all that is needed is to generate more than a certain level of revenue. This can be reflected by setting an achievement target for the indicator of revenue. All schemes which met that target would be acceptable. Targets may be particularly appropriate for reflecting constraints.
In practice, policies will be appraised against several objectives, which makes the process much more complex. Firstly, different objectives will be measured in different ways. Revenues can be measured in money terms, but it is not clear that pollutants or accidents can, unless money values can be assigned to them. Some objectives, such as liveability, are not even readily quantified. The equity objective, in particular, needs to be measured in terms of a range of values, and any indication of distribution of costs and benefits will be lost if too aggregate a measure is used. Thus objectives differ substantially in the extent to which they can be measured or be assigned money values.
Moreover, if some objectives are more important than others, it may be appropriate to assign weights to them, so that a given value of reduction in congestion can be reflected as being, say, twice as important as a given value of the reduction in pollution. Such a process requires the weights to be specified, perhaps by politicians, stakeholders or the public, and this in turn requires the objectives to be measured in a comparable way.
The appraisal process, and the predicted information which is input to it, will be somewhat uncertain. Uncertainties can be found both in the input data and in the weights assigned to different objectives. The issue of optimism bias, as discussed in Chapter 6, is an important example of uncertainties in input data. Ideally an appraisal process should reflect these sources of uncertainty, and enable the user to assess how robust the resulting decision is.
Finally, the timescale over which costs and benefits arise will affect the appraisal. In general it is assumed that a given benefit obtained in year one will be more valuable than the same benefit achieved in year two or year ten. However, some benefits, such as those to future generations, will only be of relevance in a future year, and cannot be treated in the same way.
Thus ideally any appraisal method needs to reflect:
1. the use of targets for some or all objectives, and particularly for constraints;
2. the differing degrees to which different objectives can be measured;
3. the differing degrees to which different objectives can be valued;
4. the assignment of different weights to different objectives;
5. the uncertainties which arise both in prediction and in the assignment of weights;
6. the relative value of costs and benefits arising in different years.
What are the principal appraisal methods?
There is an extensive literature on appraisal methods (e.g.: Odgaard et al, 2005; Grant-Muller et al, 2001; Mackie and Nellthorp, 2001). We provide only a brief discussion of the principal methods. The principal distinction is between Cost Benefit Analysis (CBA) and Multi-Criteria Appraisal (MCA).
Cost Benefit Analysis makes the simplifying assumptions that performance against all objectives can be quantified, and that all those quantified impacts can be assigned money values. It has been common practice for some time to assign money values to time and accident savings, although there remain concerns about, for example, the treatment of small time savings and the ethics of attempting to value life. More recently some countries have assigned money values to air pollutants, noise and contributors to global warming. However, it remains difficult to value performance against objectives such as liveability, health and economic growth. In particular aggregation of all benefits in money terms makes it difficult to demonstrate the distributional (or equity) impacts of a policy.
Cost Benefit Analysis also traditionally specifies a time horizon over which costs and benefits are predicted, which may range from 20 years to as much as 60 years in current UK practice (DfT, 2008). Standard discount rates are applied to each year’s impacts, so that they may be aggregated as a net present value of costs or benefits. Test discount rates are usually set nationally, and typically range from 2% to 8%. This approach makes it difficult to place any emphasis on the benefits to future generations, since they will occur in a future year, and be assigned a small value once discounted.
Traditional Cost Benefit Analysis makes little attempt to reflect uncertainties in the input data, though recent work has introduced techniques for dealing with optimism bias (DfT, 2008). Weights for different objectives are implicit in the money values, and it is unusual to test sensitivity to different assumptions on money values or discount rates.
Thus CBA makes no attempt to use targets ((1) above); makes simplifying assumptions on (2), (3) and (4) above; only considers uncertainty in input values ((5) above); but does adopt a process for dealing with costs and benefits over time ((6) above).
Multi-Criteria Appraisal, by contrast, avoids the pitfalls of assigning money values and instead uses one or more indicators to measure performance against each objective. Where the objective is readily quantified, as for example with reduction of accidents or air pollution, these quantified indicators are used directly. Even so, there is an extensive debate on the appropriateness of different indicators (Marsden et al, 2006). Where objectives are more qualitative, it is common practice to use semantic scaling to convert them into quantified indicators (Jopson et al, 2007; Kelly et al, 2008).
It is then possible to set targets for each of these indicators, and to appraise a policy in terms of its ability to achieve this set of targets. Whether or not this is done, the performance of a policy can be presented separately, usually in tabular form, against each indicator. The appraiser then has to compare the sets of results across objectives and across policy options to decide which options are the most acceptable. This can be a challenging task if the data is extensive, but this approach has the advantage of leaving the responsibility for deciding on the relative importance of different impacts to the decision-maker.
As an alternative, it is common practice to assign weights to different objectives, or to their related indicators, and to generate a weighted sum of the performance against all objectives. In a similar vein, weights could be assigned to results occurring in different years. Such methods can produce a single performance indicator, much as CBA does, but with a more explicit process of valuation. This makes the selection process easier, but may disguise some of the assumptions involved.
Several MCA methods incorporate sensitivity tests which allow the effect of differing input values, valuation methods and weights to be tested, either to produce different ranked lists of options or to assess the robustness of a preferred scheme to different assumptions. Such methods offer a much more robust approach to appraisal, but can be more complex in use.
Thus MCA methods are able to reflect the use of targets ((1) above), though this is rarely done; they can address differing abilities to quantify and value different impacts ((2) and (3) above); and they have well developed approaches for dealing with weighting and uncertainty ((4) and (5) above); but they are typically less effective in dealing with the distribution of impacts over time ((6) above).
Finally, some established methods adopt a combination of CBA and MCA approaches. This is particularly the case with the UK’s NATA method (DfT, 2008). One of the latest studies of appraisal methods in the EU states (excluding Luxemburg) and Switzerland is from the HEATCO project (Odgaard et al 2005). It concluded that the standardisation of principles for project appraisal varies considerably across countries and modes. In addition, differences also can be found in the way CBA approaches are employed. In most of the countries, CBA is used as a means to choose between different project alternatives (including “doing nothing”), to demonstrate the need for a measure and/or to prioritise between different variants. These findings support the earlier work of Grant-Muller et al (2001) who in addition stated that “All appraisal frameworks contain a mixture of monetized impacts, impacts measured in both physical and qualitative forms”.
Obviously variations in appraisal approaches and assumptions, even within the EU and Switzerland, can affect the transferability of predicted performance from one city to another. This issue is taken up in Chapter 13 of this report.
How might the appraisal of road pricing schemes differ from appraisal of other policy instruments?
Recent draft guidance on the appraisal of road pricing schemes in the UK (DfT, 2008) suggests that in many ways the appraisal of road pricing can follow the principles applied to the appraisal of any policy instrument. However, it highlights five ways in which road pricing appraisal raises specific issues.
The first issue relates to the complexity of the responses to a road pricing scheme, which in turn affect the performance in terms of efficiency. This is particularly a challenge for prediction, but the extent of transfers between modes and times of day, and the potential for suppression of journeys mean that the calculation of benefits and disbenefits is a complex process.
The second issue concerns the wider economic benefits, which were addressed in more detail in Chapter 9. The processes by which changes in travel costs and in accessibility affect business and residential location are poorly understood, and the extent to which benefits and losses to businesses exceed those calculated directly from changes in travel costs are uncertain. This issue relates to the treatment of agglomeration and productivity benefits, which was considered in more detail in Chapter 9. Once again, it is a particular challenge for road pricing schemes because of the extent of the influences, both positive and negative, on business costs.
The third issue relates to the treatment of changes in accessibility. The DfT guidance raises the specific issue of changes in severance, but in practice this will be only one of a large number of changes in accessibility for the users of all modes. These will in turn be one of the principal factors affecting the distribution of benefits.
The distribution of benefits and the equity implications are the fourth issue raised. The DfT guidance recommends an approach to the modelling and appraisal of equity implications which has proved extremely challenging for the cities involved.
Finally the guidance notes that the treatment of value for money (as reflected in UK practice by the comparison of CBA net present benefits and MCA outcomes with net present costs) needs to be modified to reflect the fact that road pricing will usually generate surplus revenue. Despite this observation, guidance on the ways in which the value of revenues generated, for example using shadow pricing (May et al, 2000, May et al, 2001) is still limited.
What can we learn from past approaches to appraisal and evaluation?
Appendix D includes an example of the appraisal of one of the road pricing proposals for London (GOL, 2000). It demonstrates that it has attempted to reflect the full range of policy objectives, and has used a combination of CBA and MCA approaches. Little attempt has been made to weight the different objectives, and the treatment of distributional and equity impacts remains weak.
No information on this theme is currently available from the case studies