The key research questions guiding our design of the RRRP evaluation are:
• Did rehabilitating roads affect the quality of roads?
• Did rehabilitating roads improve access to markets and social services?
• Did rehabilitating roads affect agricultural productivity and profits, and if so, by how much?
• Did rehabilitating roads improve household well-being for communities served by these roads, especially income and poverty?
As is frequently the case for large scale infrastructure projects, it was not feasible to randomly select the roads that would be rehabilitated in the MCA RRRP from amongst the eligible. Instead, MCA planned to fund projects based on estimated economic rates of return (ERRs). The ERR is calculated from several inputs, including the vehicular traffic, vehicle operating costs, and the cost of the project, among others. An ERR was estimated for each of the road links under consideration for rehabilitation, and in order to be funded, a road’s ERR had to meet or exceed 12.5 percent. However, the MCC Board subsequently made the decision to discontinue funding any further road construction and rehabilitation under the Compact due to concerns about Armenia's democratic governance. Starting in 2009, the Armenian government instead accessed loans from the World Bank to rehabilitate many road sections that were included in the RRRP plans before MCC discontinued funding, using and updating the road project designs developed by MCA-Armenia. These roads were selected based on estimated ERRs as well as other factors such as the size of the population who would benefit.
The evaluation of rural road rehabilitation uses a comparison group design. The evaluation design identifies the counterfactual by defining a comparison group of roads selected from those initially proposed by MCA that share similar characteristics as those selected by the World Bank, but that were not rehabilitated. The evaluators adopted a hybrid approach that matches road links based on a limited number of key observable characteristics and then uses regression modeling to control for other characteristics that may impact outcomes. The most fundamental criterion used in identifying potential comparison roads is that it is among the roads that were originally considered for rehabilitation by MCA, indicating that the road conditions are poor enough to consider rehabilitation.
The data for this evaluation come from Armenia’s Integrated Living Conditions Survey (ILCS), an annual household survey fielded by Armenia’s National Statistical Service (NSS). MCA funded a considerable increase in the rural sample, which was in part aimed at facilitating the RRRP evaluation. The sampling plans were developed and implemented based on the evaluation design described above. The evaluation includes households served by 27 roads that were rehabilitated with World Bank financing based on designs funded by MCA and 28 comparison roads.
