The performance evaluation of the Kigoma solar activity was designed to answer questions about the implementation of the program and about outcomes that may have been affected by the program. Specifically, the evaluation sought to answer the following questions:
1. How was the Kigoma solar activity implemented?
2. How did outcomes differ at follow-up and change over time for the targeted group selected to receive the Kigoma solar activity versus the nontargeted group?
We sought to answer the first question by examining several implementation domains: (1) implementation processes and experience with the program, (2) installation of MCA-T-funded solar systems, and (3) service quality of solar systems. To answer the second question, we examined several performance domains: (1) electric and non-electric energy consumption and expenditures, (2) investment and economic activities, (3) human capital accumulation, and (4) economic growth.
The evaluation covered all seven types of potential beneficiaries of this activity, and included respondents selected to participate in the activity (targeted respondents) and a comparison group of respondents who were not selected to participate (nontargeted respondents). Nontargeted respondents were selected from distinct geographic areas to minimize the possibility of contamination. No additional efforts were made to select nontargeted respondents to be similar to targeted respondents; as a result these two groups of respondents may differ in significant ways on characteristics that might affect our outcomes of interest. The beneficiary types are (1) schools, (2) health centers, (3) dispensaries, (4) businesses in village markets, (5) fishers, (6) businesses that received loans from local credit institutions to purchase solar systems, and (7) households that received loans from local credit institutions for solar systems.
To address the first research question, covering implementation, we looked at differences in the implementation outcomes between the targeted and nontargeted groups and changes over time in those outcomes. We also examined changes in solar PV use between the interim survey and the follow-up survey among all targeted respondents. In addition, we compared the use and service quality of MCA-T and non-MCA-T PV systems at follow-up, comparing all MCA-T users with all non-MCA-T users, regardless of their group assignment. Last, we examined the challenges and benefits of participation in the Kigoma solar activity among those who reported using MCA-T systems at follow-up.
To address the second research question, covering outcomes in the performance domains, we compared outcomes between the targeted and nontargeted groups and analyzed how key outcomes changed over time. For most outcomes, this meant looking at change between the interim and follow-up surveys. We calculated an adjusted difference at follow-up and a "net change," which is defined as the difference between targeted and nontargeted groups at follow-up minus the difference between targeted and nontargeted groups at interim. This is similar to a difference-in-difference analysis. In some cases, we collected information during the follow-up survey that was not collected during the first round of data collection. For these questions, we present only differences between the targeted and nontargeted groups at follow-up.
The performance evaluation utilized data collected from an interim survey (2013) and data from a follow-up survey (2015). The follow-up data are provided in this package. Key findings related to implementation include:
-
Implementation generally occurred according to plan, but some key implementation challenges, such as the apparent limited use of an SMS marketing component and the lack of participation by surveyed fishers, limited our ability to evaluate some components of the activity.
-
Solar PV use was common among those who received MCC-funded systems.
-
As use of solar PV systems grows, so do expectations for their performance and capacity.
-
Solar PV systems face quality issues, and a lack of maintenance and repair training may limit their utility and popularity.
Key findings related to performance include:
-
The use of solar PV systems increased over time, among both targeted and nontargeted respondents, and was consistently high in the targeted group at interim and follow-up.
-
Liquid fuel use was lower among targeted respondents than nontargeted respondents both at interim and at follow-up.
-
Findings from the performance domains provide limited evidence of association between the Kigoma solar activity and improvements in outcomes related to investments, economic activities, and human capital accumulation for specific beneficiary types.
More details can be found in the final evaluation report: Vohra, Divya, Edith Felix, Duncan Chaplin, and Arif Mamun. "Evaluation of the Kigoma Solar Activity in Tanzania: Final report." Report submitted to the Millennium Challenge Corporation, Washington, DC: Mathematica Policy Research, February 2017.