Impact analysis assessments are designed to carefully determine how impact, i.e. social or economic change that ultimately results from project activities, can and will be demonstrated, measured, and analyzed during the course of the project. Impact analysis seeks to guide USAID in telling a credible and compelling story about project impacts on the societies and economies in the countries that USAID assists.
This report reviews the components of the COTS project in order to identify opportunities for impact analysis — where analysis includes the identification, measurement, and attribution and may involve simple data collection or more complex studies. This report makes a variety of suggestions of specific measures and studies and of areas of further inquiry where impact analysis may be possible.
This report is not intended to be prescriptive so much as to start the project team thinking and planning around impact analysis. The next step is for the project team to determine which analysis it wants to undertake and then identify and assign resources for those efforts.
This assessment explores a variety of ways to identify and assess impact. In some cases, there may indeed be a quantifiable indicator of impact, for which data can be collected on a periodic basis. In other cases, the project may be limited to case studies or individual interviews in order to draw direct links between project activities and the kinds of behavioral changes that we can call outcomes (as opposed to inputs and outputs). For all components, this assessment presents ways that the impacts can ultimately be monetized so that a return on investment indicator can be calculated.
A general conclusion in this report is that the project will only be able to measure and analyze impact in a comprehensive way if impact analysis is integrated into the project activities for each component. COTS is a complex project composed of a variety of activities, many with separate and distinct objectives. While a few possibilities exist for indicators that can capture impact across a set of activities, in many cases attribution will only be possible if project staff looks for ways to identify impact at the activity level. This will require follow-up efforts for activities; for example, interviews of training participants some time (say, three months) after a training to determine whether their behavior has changed consistent with the training. As noted above, some of these analyses will not produce “indicators” but rather case studies or impact success stories that can still be enormously useful for the project to demonstrate impact.
An integrated approach will require all members of the COTS staff to be involved. The M&E specialist can assist by coordinating, monitoring, and compiling analysis, but each technical specialist will need to be involved in identifying impact and collecting data and information to analyze it. This need not be overly time consuming, and again, project staff will need to decide which impacts it wants to capture.
It should be noted that the measures considered in this report that can be monetized are, in most cases, not economic value-added indicators. That is, just because they are measures that can be represented in dollar terms (and therefore compared to the dollar cost of project activities), they do not necessarily represent increases in GDP. They should each be reported for exactly what they are and not summed unless otherwise recommended in this assessment.