The methodology for this evaluation comprises a performance evaluation and an economic analysis. The economic analysis while not the central focus of the evaluation, is a pinnacle aspect of the evaluation. It seeks to discover whether investments financed by the MCC created the intended Economic Rate of Return (ERR). The evaluation also seeks to discover if and how economic benefits, in the form of reduced vehicle operating costs and time savings, are experienced and captured by stakeholders. In general terms, the economic analysis is a comparison between the costs necessary to improve roads in Mozambique to the expected benefits that result. Costs include rehabilitation and construction costs, and routine and periodic maintenance costs. The nature of benefits that are realized from improving roads can vary. As a proxy, and in line with MCC's Principles to Practice research, the economic analysis estimates economic benefits by calculating the decline in vehicle operating costs and time savings for users of the roads built with MCC grants. A portion of the data collected under all research areas will feed into the assumptions that inform economic analysis.
The economic analysis identifies and unpacks the underpinning drivers of Economic Rate of Return changes over time and provides learning regarding what MCC can do to promote better ERR for future similar road investment projects. In HDM-IV, the evaluation's economic analysis model, the primary drivers of the economic analysis are traffic count, vehicle operating cost (VOC), travel time, IRI, and investment costs. Examples of potential lessons the analysis may generate include ways to avoid under-designed roads that result in excessive maintenance costs or over-designed roads that result in excessive construction costs.
The performance evaluation will identify factors in project implementation that influenced economic results. It will also provide depth and narrative around economic analysis results so that economic results are better understood by the project's stakeholders. The performance evaluation also complies with MCC's requirement that each project conduct an independent evaluation that measures the achievement of results.
The evaluation design uses a mixed method approach, comprising qualitative and quantitative data collection from primary and relevant secondary sources. Primary data collection utilizes targeted sampling for the various proposed surveys, for example: Origin-Destination surveys, engineering assessments, traffic counts, journey times calculations, and axle loading measurements. Appropriate sampling strategies have been developed which take the evaluability assessment findings into consideration.
Research Question 0:
Was the project implemented according to plan? Were there any deviations from the original design? If so, these should be documented to the greatest extent possible.
Research Question 1: What is the economic return – calculated in terms of vehicle operating cost (VOC) savings and travel time (TT) savings – of the road investment? What factors drove changes to the Economic Rate of Return (ERR) over time? How could the project have been designed to result in a higher ERR?
Research Question 2A: What are the relevant road authority's current maintenance practices and what is the likelihood that MCC's investment will remain adequately maintained for the life of the investment? Based on this assessment, what set of maintenance assumptions should be used in the HDM-4 model to yield the best estimate of the costs and benefits of the road investment?
Research Question 2B: In cases where MCC invested in improving maintenance practices or included a maintenance Conditions Precedent in the Compact (applicable to Mozambique), what were the effects of those efforts and why?
Research Question 2C: What political, and economic incentives are shaping road maintenance decisions in the country? And what other key factors are influencing actual maintenance practices?
Research Question 3A: Who is traveling on the road, why, what they are transporting, what they are paying for transport, and how long does it take to move along key routes?
Research Question 3B: Have road usage patterns changed, in terms of who is traveling on the road, why, what they are transporting, what they are paying for transport, and how long it takes to move along key routes?
Research Question 4: How is the transportation market structured and what is the likelihood that VOC savings will be passed on to consumers of transportation services?