Executive Summary

Objective

The objective of this feasibility assessment (FA) is to assess the possible evaluation options for the Health, Ecosystems, and Agriculture for Resilient Thriving Societies (HEARTH) Eastern Kafue Nature Alliance activity (“Kafue Activity”) in Zambia. The assessment considers design options, including impact evaluation (IE) and performance evaluation (PE), that meet Agency-wide HEARTH and Mission learning interests, with the goal of determining the most rigorous options that can be applied given implementation, resource, and other constraints for this activity.

Kafue Activity Overview

The Kafue Activity will be implemented in several Game Management Areas and conservancies that border the eastern side of Zambia’s Kafue National Park, by a consortium of public and private sector partners led by The Nature Conservancy. High poverty rates in the area lead to a dependence on natural resources and income from forests for many households, contributing to deforestation and forest degradation from wood extraction, agricultural expansion, and fires. To address these issues and impact both conservation and human well-being outcomes, the Kafue Activity is comprised of four strategic approaches (SAs):

• SA1: Strengthen natural resource compliance and management systems
• SA2: Develop inclusive ecosystem-based markets for local prosperity
• SA3: Strengthen community maternal and child health and improve access to clean water
• SA4: Develop effective land and resource use planning, tenure, and governance systems

Evaluation Approaches Considered

IEs measure the causal impact of a program, or the difference in outcomes caused by the program and not by other external factors. While assessing the feasibility of an IE was one of the primary objectives of this assessment, at USAID’s direction, the FA team considered PE approaches. In particular, a variety of evaluation methods were considered including (1) experimental approaches, which measure the causal impact of programs through randomized assignment (e.g., randomized control trials); (2) quasi-experimental, which also attempt to measure causal impacts but without randomization (e.g., difference-in-difference [DID] and statistical matching); and (3) non-experimental approaches, which can answer descriptive questions about differences but cannot measure causality with the same degree of rigor or confidence. Non-experimental approaches include PEs, which generally include before-after comparisons without a rigorously defined counterfactual, and case studies, which include in-depth learning from an instance through extensive description and analysis. A mixed-method evaluation integrates two or more evaluation methods, usually drawing on both quantitative and qualitative data. Generally, mixed methods evaluations can provide a deeper understanding of why change is or is not occurring and capture a wider range of perspectives.

Summary of Findings

The FA team finds that the Kafue Activity presents an important opportunity to improve USAID’s understanding of conservation and biodiversity programming through a mixed methods evaluation, including both IE and PE components. Specifically, several components of the program being implemented in Year 1 (SA1 resource protection and SA2 agricultural markets and out-grower support) might be amenable to evaluation through a quasi-experimental difference-in-difference (DiD) approach, with matching to improve rigor. DiD is a quasi-experimental evaluation design that estimates programmatic impact by comparing (1) changes in outcomes among program participants with (2) changes in outcomes among non-participants and is one of the most commonly used designs for IEs. For some individual SAs, experimental methods like a randomized lottery around eligibility cut-off for health and water (SA3) activities can be further explored after the needs assessments and situational analyses have occurred. It should be emphasized that final decisions about the evaluation design and methodology for activities under SA3, SA4, and some components of SA1 and SA2 can only be made when the interventions and sites are determined at the end of Year 1.

Given the dearth of counterfactual-based studies on the Kafue Activity strategic approaches, even knowledge generated through a well-designed PE for some components would advance USAID’s and the HEARTH portfolio’s learning agenda. Furthermore, an evaluation would add value by strengthening the program’s theory of change and promoting a deeper understanding of where to focus on intervention integration and quality. Baseline data will provide a key source of monitoring and evaluation (M&E) data and provide important contextual information that can be used to promote more effective, adaptive programming.

In addition, the Kafue Activity presents a unique opportunity to measure the effect of conservation programming on biodiversity outcomes. This is due to the large amount of biodiversity monitoring that will take place as part of the program implementation; this large-scale wildlife monitoring makes it feasible to pursue a cost-effective, rigorous, and long-term study of biodiversity outcomes. Extensive observation data collection through a combination of SMART monitoring, spoor surveys, and camera traps may provide the necessary data to measure biodiversity outcomes in the context of an IE approach.

Recommendations

In addition to the key findings above, the FA team recommends the followings to USAID:

• Baseline data collection at the end of Year 1: Given the phased implementation design, the FA team recommends waiting until the end of Year 1/ beginning of Year 2 to conduct one comprehensive baseline household data collection effort after all activities and locations have been determined.
• Need for Pause and Reflect: The site locations and content of most interventions will not be finalized until the end of Year 1. The FA team recommends a series of regular coordination and information exchange meetings as implementation information becomes available. In addition, at the end of Year 1 the MERL plan will need to be updated and there should be a Pause and Reflect of all stakeholders.
• Biodiversity Measures: Overall, we recommend a combination of approaches for monitoring biodiversity outcomes that leverages existing datasets and data streams from partners with new data sources. Remote sensing data, particularly forest cover, provides rich and readily available proxies for biodiversity, and as well as important habitat outcomes. Additionally, the most likely direct biodiversity indicators will involve changes in wildlife behavior or spatial distributions near treatment sites. Camera traps will provide an efficient balance between cost and field effort, yielding high-quality data for a broad diversity of large mammals, and the long record of aerial censuses and recent SMART monitoring activities make valuable baselines for understanding biodiversity outcomes for common large-bodied species.
• Long term evaluation: The FA team recommends a long-term evaluation, including follow-up data collection about five years after the end of the activity. The primary biophysical outcomes of interest will take a longer time to materialize than the standard USAID program cycle.
• Strong coordination and collaboration are required throughout design and implementation: A rigorous evaluation will require detailed M&E tracking of inputs, outputs, and specific site locations, along with significant coordination among the IPs and between the IPs and the evaluation team, to ensure the design is appropriate as implementation plans evolve.
• USAID and IP focus on integration and quality of programming: Integration is a key underlying assumption for the whole of project theory of change, as well as for the theories of change for several SAs. Thus, site selection for activities should prioritize overlapping implementation to the extent possible, to answer key learning questions related to integrated programming.