Abt Associates evaluated Indonesia's Procurement Modernization Project using mixed methods. The qualitative approach used the 5-S framework (modified from McKinsey's 7-S framework) to analyze organizational change within procurement service units (PSUs) related to shared values (perception about corruption), structure (permanency of PSUs), systems (procurement processes adopted), skills (staff procurement skills), and staffing (staff preference for careers in procurement). To evaluate changes along the 5-Ss and several key procurement outcomes, the evaluation used a quasi-experimental impact evaluation design - a weighted difference-in-difference that compared outcomes for Phase 2 treatment PSUs with outcomes for comparison PSUS. The comparison PSUs were shortlisted in Phase 2 but not selected to receive the program's core services and were weighted to look similar to the treatment PSUs. Data for the evaluation come from structured surveys with PSU and spending unit (OPD) staff at the baseline (2016) and endline (2019), semi-structured interviews with PSU staff at endline (2019), and semi-structured interviews with key stakeholders after the project ended (November 2018-January 2019). The quantitative analysis also used time series data from the Procurement Management Information Service (PMIS) on two tender-level final outcomes: time efficiency and cost-efficiency.
Evaluation questions -
Shared Values
a. Are there any issues related to the political economy (or other aspects) of the procurement system and its actors not addressed by the project that may have impacted the project's ability to achieve its intended results?
b. Did the program result in a change in culture or shared values?
Structure
What types of organizational or operational changes are taking place at the PSU level?
a. Have PSUs adopted the Maturity Model (a set of self-assessment and development tools) as an approach to supporting their organizational development goals?
Systems
What types of procedural changes are taking place in the conduct of procurements?
a. What was the quality of policies and procedures developed by the project (e.g., public private partnerships - PPPs)?
b. Are there changes in policies, procedures, or otherwise that could lead to quality improvements in ultimate procurement (contract) outcomes? How so?
c. Are there changes in policies, procedures, or otherwise that could lead to savings (financial or total lifecycle) in government procurements? How so?
d. Are PSUs using e-catalog for standard purchases?
e. Are PSUs using the lifecycle PMIS?
f. What was the quality of PMIS?
g. Has the PMIS contributed to changes in procurement planning or implementation?
h. Does the design of PMIS meet the needs of the UKPBJs and other procurement actors?
i. Have PSUs developed their own framework contracts?
j. Have PPPs been conducted in accordance with the policies and procedures developed by the project?
k. Has there been an increase in PPP transactions?
Skills
a. Are the skills/knowledge emphasized in the training spreading within the PSU? How so?
b. Has the procurement knowledge and skill of trainees improved?
c. What was the quality of training and mentoring?
Are staff now permanent staff?
a. Do staff seem committed to and engaged in pursuing a procurement career path?
b. Are trained or “permanent” staff retained?
c. Do staff feel more supported administratively and legally?
d. Was there a gender inclusive strategy for recruiting procurement staff?
Overall
Were the Activities/Sub-Activities implemented as designed?
a. What were the implementation challenges and successes?
b. Is there evidence that the interventions have resulted in the outcomes outlined in the project logic?
c. Was the set of activities designed the most strategic intervention for the Indonesian procurement context or to improve procurement?
d. Has framework contracting/e-catalog resulted in time and/or cost savings?
e. Is there evidence for cost savings in the program PSUs?
f. How has budget absorption [execution] in the PSUs changed over time?
g. Are there detectable improvements in budget execution and efficiency of procurement execution in the UKPBJs and spending units?
In terms of treatment exposure, Phase I PSUS received treatment from 2013-2018 and Phase 2 from 2016-2018. Through efforts that had nationwide reach and through PSUs that became centers of excellence (COE), it is possible that the comparison PSUs received some treatment at the same time, but less intensively.
The evaluation found evidence of positive impact of PM project's intensive activities in only a few areas. The project improved staff skills-as measured by their quiz scores on procurement processes-but absolute quiz scores remained quite low. Additionally, more PSUs achieved permanency as a result of the PM project. The evaluation found that the relative odds that a PSU was permanent increased almost nineteen-fold because of the project. However, we did not find clear evidence that the project's intensive activities improved the adoption of improved procurement processes, maturity model, e-catalogues, PMIS, and/or framework contracting. Adoption of policies and procedures and PMIS increased over time, even in comparison PSUs. Some of these efforts were simultaneously being promoted by nationwide policies, which can explain improved performance in the comparison PSUs. Conceivably, COEs could have also helped improve the outcomes in comparison PSUs but our data suggests that majority of comparison PSUs did not receive COE input. Overall, despite the small gains in staff skills and greater permanency of PSUs, the PM project's intensive activities did not improve the quality, cost, or time efficiency of procurement. These outcomes did not improve in the comparison group either.