In this evaluation design report, we describe the design for the evaluation of the technical, vocational education and training (TVET) activity (Activity 2) of the Guatemala Education Project funded by the Millennium Challenge Corporation (MCC) through a Threshold Program focusing on secondary education in the country. The TVET activity includes the GOPA Training for Work for MINEDUC and ENCA activities. Our research questions focus on assessing mainly the short-term outcomes, the fidelity of program implementation, and justification of program resource allocation and are grouped around three main questions: (1) To what extent did the TVET program meet its short-term outcomes?; (2) To what extent was the TVET program implemented according to plans? Why or why not?; (3) To what extent do the results of the TVET program justify the allocation of resources? To answer these research questions, we plan to conduct a mixed-methods performance evaluation with three components.
First, for each activity, we will conduct a fidelity of implementation study that will (1) explore how implementation is being conducted by implementers; (2) examine whether implementers adhered to the original program design, how and why changes to the design were made by implementers; (3) help contextualize short-term outcome results by describing the geographic, social, and policy environment in each of the Departments as well as each TVET center; and (4) provide a deeper understanding of how the project interventions may contribute (or not) to short-, and medium-term outcomes. The implementation study will draw on administrative data from MINEDUC and the TVET centers on enrollment, interviews with key stakeholders, focus groups with students in the first cohort of the programs, and reviews of project and stakeholder documents, including the labor market diagnostic studies conducted by the departmental and national working groups. If the timing and the health situation allows, we will also conduct observations of the trained teachers delivering the new content in classrooms.
Second, we will conduct an outcome analysis. While no longer a major component of the evaluation, our team will use any quantitative data we gather to assess the extent to which short and medium outcomes are met by this program. The analysis aims to shed light on potential benefits from the investments in the TVET centers. The outcomes analysis will draw primarily from administrative data collected through the GOPA and MCC M&E plans, data on number of instructors trained, student enrollment, etc. As part of the key informant interviews with businesses, we will discuss the extent to which the TVET curricula aligns with the labor market demands and whether they believe students graduating from the TVET centers will meet the labor market demands. If the timeline and the health situation allows, we will conduct observations of classroom instruction to determine to what extent instructors demonstrate the new skill sets covered by the TVET training.
Third, we will conduct a multi-site case study of 4 TVET centers (ENCA, implementing careers in agroindustry plus 3 MINEDUC centers, implementing careers in hospitality, transportation, and informational technology) across departments. We will work with MINEDUC, PRONACOM, and MCC to identify a set of TVET institutions with careers that are representative of the revised programs and meet varying degrees of expected outcomes. To obtain a diverse sample of TVET institutions, we will seek to include institutions in urban and rural localities, varying sizes and infrastructural conditions, and institutions offering a range of programs of study. The data for the case studies will be incorporated into the interview and focus group protocols for the implementation study and outcomes analysis to ensure efficiency in our data collection, analysis, and writing processes.
Primary and secondary data collection will span from mid-2019 until mid-2022.