Document Type: Monitoring and Evaluation

As stated in USAID’s scope of work for the Food and Enterprise Development (FED) Impact Survey, the “objective of the impact study is to capture the magnitude of the impact of USAID/Liberia’s value-chain investments in FED on agricultural productivity and profitability, investment and earnings, and quality of life”. The evaluation has three primary goals: 1) to measure the difference in key indicators between FED beneficiaries and a comparison group of farmers, making use of standardized Feed the Future (FTF) performance indicators to the maximum extent possible; 2) to disaggregate this impact across four value chain commodities; and 3) to estimate to the extent practicable the impact of FED programming by year of initial program intervention.

The FED Impact Survey focused on five evaluation questions:

1. Have beneficiary farmers applied new practices, and technologies to selected value chains?
2. How much have beneficiary farmers invested (cash or equivalent in-kind) compared to the comparison group?
3. What are the gross profits and productivity of beneficiary farmers compared to the comparison group?
4. What are the household incomes of beneficiary farmers compared to the comparison group?
5. What is the quality of life of beneficiary farmers and their families compared to the comparison group?

PROJECT BACKGROUND

In 2011, USAID launched the implementation of FED, a five-year, $75 million cost plus fixed fee completion type contract under a consortium led by Development Alternatives, Inc. (DAI). The objectives of the program, as stated in the contract, were to:

1. Increase agricultural productivity and profitability and improve human nutrition;
2. Stimulate private enterprise growth and investment; and
3. Build local technical and managerial human resources to sustain and expand accomplishments achieved under objectives one and two.

The project was implemented in six counties that collectively include 75% of Liberia’s households, more than two-thirds of all farming households; and nearly 70 percent of the country’s population living below the poverty line; these six counties, which include Grand Bassa, Bong, Margibi, rural Montserrado, Lofa, and Nimba, make up the Liberia Feed the Future Zone of Influence (ZOI).

FED worked in four product value chains (rice, cassava, vegetables, and goats) in partnership with farmers, agribusinesses, non-government organizations (NGOs) and the Government of Liberia. The FED Fiscal Year (FY) 2015 Annual Report estimated that the project had directly benefited at least 102,679 rural households of which approximately 60% participated in the cultivation of rice, 30% in cassava, 5% in goats, and 5% in vegetables.

In 2015, performance monitoring results reported by FED were questioned when an Independent Data Quality Review (DQR) “revealed serious deficiencies throughout the data management process, i.e., instrumentation, collection, collation, monitoring and quality checks, and how information was calculated and selected for reporting purposes.” It concluded that much of the data reported by FED, particularly on the most critical outcome/impact indicators, was of “poor quality and cannot be used for reporting or decision-making purposes.” As a result of the DQR, this impact survey was commissioned by the USAID mission in Liberia, and implemented by International Development Group LLC, to estimate the impacts of FED on beneficiary outcomes in light of unusable performance monitoring information.

This is a report on the mid-term performance evaluation of the Georgia Economic Prosperity Initiative (EPI) project funded by the United States Agency for International Development (USAID) Mission in Georgia. The project is implemented by Deloitte Consulting.

The evaluation of EPI was conducted during the period April – May, 2013, by a team assembled by Mendez, England & Associates (ME&A) and NORC, both with headquarters in Bethesda, Maryland. The team consisted of three experts – one international and two locals – with experience in agricultural development, private sector and business enabling environment.

EVALUATION PURPOSE AND EVALUATION QUESTIONS

The purpose of the evaluation was to assess the contributions of the EPI toward achieving the USAID/Caucasus’ Development Objective – “Inclusive and Sustainable Economic Growth.” Specifically, the goal was to measure the effectiveness of EPI’s interventions in targeted sectors and value chains. The evaluation’s main objective was to determine the progress of EPI in improving the overall competitiveness of the Georgian private sector through consideration of the project’s design and implementation to date. Other objectives included: 1) assessing the contribution of EPI’s activities toward achieving its high-level results (Productivity, Access to Finance/Domestic Investment, Exports, and Employment); and 2) advising on the practicality of measuring “the success of EPI as a whole” against such targets. The evaluation covered the first two of the project’s four-year term, the period of September 2010 through September 2012.

As requested in the scope of work (SOW), the main questions that the evaluation had to address were:

1. To what extent has the EPI project contributed to improving the business enabling environment in Georgia and the Government of Georgia’s (GoG) adoption of such improvements?
2. What are the main effects of the project on targeted agriculture and non-agriculture value chains and the business enabling environment, including on overall country-level competitiveness?
3. With respect to EPI’s targeted sectors, what has been and what will likely be the contribution of the project toward increasing the four high-level results: Productivity, Employment, Access to Finance/Domestic Investment, and Exports.
4. As a result of EPI initiatives, have businesses changed business practices, as well as increased productivity and the value of their enterprises, thus contributing to the overall competitiveness of value chains and the Georgian private sector?
5. What are the constraints/challenges/issues that inhibit the project’s contribution toward achieving the high-level results during the remaining term of the project?
6. What are the opportunities to improve impact and enhance the implementation and management of similar projects in the future; i.e. gender equity and sustainability?

PROJECT BACKGROUND

EPI is designed to improve enterprise, industry, and country-level competitiveness in Georgia. EPI’s assistance to firms in agricultural, manufacturing and the service sectors aims to increase investment, open new markets, raise productivity, drive domestic and export sales, and create jobs. The goal is to strengthen and institutionalize positive changes in the business enabling environment in the country.

EPI’s activities are organized around four major components:

• Component 1: Business Enabling Environment (BEE), which includes business regulation/licensing, strengthening property rights, investment sector economic governance, trade and customs economic governance, tax administration, procurement/privatization, and agricultural policy.
• Component 2: Agriculture (AG), which includes: a) hazelnut, mandarin (tangerine), greenhouse and open field vegetable value chains (VCs); and b) technical assistance programs provided by agricultural service and input providers to benefit farmers and agribusinesses, associations, agricultural vocational colleges, financial service providers and non-governmental organizations (NGOs).
• Component 3: Manufacturing and Services (M&S), which includes: a) a number of VCs, such as transport and logistics, packaging, Information Communication Technology (ICT), apparel, wine, and Meetings, Incentives, Conferences, and Exhibitions (MICE) tourism; and b) productivity enhancement, investment promotion and market linkages strengthening.
• Component 4: Cross Cutting Activities (CC), which includes upgrading workforce skills in targeted VCs; youth/women employment issues; access to finance; improving the quality of business and economic information; and creation of a sustainable capacity to implement a targeted approach to investment promotion.

MAIN FINDINGS

• EPI successfully integrated many project components and activities and cross-cutting support among project components.
• One of EPI’s strong points is promoting public-private dialog and partnerships, as well as dialog and cooperation among private sector entities.
• EPI is credited with being flexible and responsive to clients’ expressed needs and opportunities.
• Work on development of major pieces of legislation is always uncertain but EPI has had a fair share of success.
• EPI’s work in tax administration filled an immediate need, was demand driven, and is highly appreciated by the client.
• EPI was able to establish working relationships with important project counterparts, including: municipal, village and Adjara regional administrations; Farm Service Centers; and Ferrero’s subsidiary – Agrigeorgia.
• Across all agricultural products, the logistics, organization and quality of training was excellent. Thematic coverage of agriculture production and the quality of training materials was also excellent. Information gathered during the evaluation shows that the Greenhouse training is one of the most notable achievements of the AG Component.
• Prior to EPI’s involvement, the potential of MICE was not widely viewed as an area for development by the GoG or the tourism industry. With EPI’s help, the Georgian National Tourism Agency (GNTA) declared MICE tourism a priority and, for the first time, Georgia appeared as a MICE destination on the global map.
• EPI was instrumental in facilitating the private-public partnership (PPP) that led to the involvement of IT companies in the development of some e-government products and services.
• EPI’s main activity in workforce development – upgrading the skills of garment sector workers through partnerships with vocational educators – is an effort with the potential to positively impact employment, product quality, promotion and export volumes, as well as increased investment in the sector.
• Most project outputs and performance targets have been achieved, some with qualifications.
• EPI’s work to achieve international accreditation of agricultural testing laboratories has the potential to positively impact the success of the local businesses.
• EPI’s work in development of a modern construction code may not succeed but this is largely for reasons outside of the project’s control.
• The work in intellectual property protection was generally well implemented, highly valued by the beneficiaries and has had, or is likely to have, positive impacts.
• The investment sector economic governance sub-component was limited in scope; therefore, specific accomplishments are hard to identify at this time.
• The Georgian state procurement system was essentially a Georgian effort aided by European donors; EPI has had limited influence on or interaction with the system.
• There remains a good deal of skepticism that working with small Georgian farmers is a long-term strategy.
• Work remains to be done in the AG sector on improving business practices and market access.
• While the work on crop insurance was highly successful, there remain serious issues of sustainability.
• The Transport & Logistics (T&L) subcomponent of the project was a priority. The Trade Facilitation System (TFS) is a very good example of EPI’s success with Public Private Dialog (PPD) and PPP and will likely turn out to be an important part of the project’s legacy.
• Enterprise owners are highly appreciative of the analytical work and recommendations provided by EPI’s international and Georgian consultants. However, there is a risk that many recommendations will stay on paper given factors such as poor access to finance and low capabilities among enterprise management.
• Most beneficiaries in the apparel VC are optimistic about results, which they expect in project years 3 and 4. Beneficiaries highly value EPI’s activities in addressing low productivity in the apparel companies as a major impediment.
• The social capital or economic benefits derived from the cooperation among value chain actors seems to be low.
• EPI’s work on financial leasing has been useful and successful. However, results for farmers are not yet visible and some significant challenges remain.
• EPI’s work on financial advisory services resulted in a minor increase in volume of loans going to farmers and small, medium, enterprises (SMEs); however, there remain many challenges to increase the flow of credit to agriculture and small business.
• EPI has contributed to the development of an indigenous capacity in investment promotion. However, concrete results of these efforts have not yet been realized and there are mixed views on its success and viability
• Key stakeholders in government have generally positive but mixed perceptions about the project; the project’s rough start colored perceptions, which continues to this day.

MAIN CONCLUSIONS

• EPI has made, and continues to make, important contributions to business climate improvement.
• Most major contributions of EPI have been in assisting in the implementation of good policies already established and, particularly, in providing concrete, tangible means of implementation that increase transparency and administrative efficiency, thereby increasing certainty and reducing transaction costs.
• EPI’s emphasis on PPD has been instrumental in introducing to the GoG a new way of doing public business that can enhance the government’s ability to make good policy; however, there are questions of sustainability.
• EPI has had moderate success in promoting useful laws and regulatory acts, which may increase over the remaining years of the project.
• EPI has provided no formal training in policy analysis and development techniques to GoG staff.
• Among the most significant effects the project has had on the business enabling environment have been increased transparency and decreased transaction costs of administrative processes, and more open communications through PPD. Secondary effects include modernization of some key laws and regulations.
• Improvements in farming practices are visible and over time should lead to better quality products and enhanced country-level competitiveness.
• EPI has made and continues to make important contributions towards establishing Georgia as a Caucasus Transit Corridor, which can have a direct impact on high-level indicators and overall country competitiveness.
• EPI was instrumental in the support of very important promotional activities, which were trend-setting for the wine tourism and MICE sectors in Georgia. Those activities arguably put Georgia on the wine tourism and MICE maps for the first time.
• There are some short-term positive impacts on high-level results, but likely not in the magnitudes suggested by EPI. Most impacts are very difficult to estimate and the project was not structured for high-level evaluation. It is reasonable to expect medium- and long-term positive impacts on some high-level indicators.
• Few visible changes in business practice have been observed in the AG Component at this time.
• Changes to business practices are more visible in M&S VCs. However, even there, utilization is slow and affected by other variables such as the availability of financing. It is likely that the businesses will change their practices provided that EPI continues and, perhaps, intensifies its support to some value chain enterprises.
• The change in the government may put on hold some activities but should not be considered as a constraint that inhibits achievement of the project’s high-level objectives during the remaining term.
• Sustainability of PPD, one of EPI’s signature accomplishments, may depend on further institutionalization of the practice.
• The competitive advantage of Georgia’s apparel sector and, therefore, its long-term impact on economic growth is questionable. However, the medium-term impact on the sector’s economic indicators can be significant.
• Access to finance, and in particular finance for small farmers and SMEs, remains problematic.
• Sustainability of investments in small farmers is open to question.

MAIN RECOMMENDATIONS

• Analytical work could be done in a shorter period of time in order to start focusing on implementation in the earlier period of the project.
• Emphasize time of delivery of promised products and services.
• Avoid selecting project activities based on likelihood of achieving performance targets.
• Continue to work on good performance indicators.
• Emphasize flexible planning.
• Going forward, emphasize proven strengths and phase out less productive initiatives in business climate reform.
• Emphasize institutionalization of PPD – Competitiveness Council, business associations, administrative procedure laws, etc.
• Provide more formal training in policy analysis and development to GoG.
• Current initiatives in the AG Component should be enhanced including, for example, training in marketing and aid to cooperatives.
• To organize better trading channels for farmers, the project should concentrate on the last link of the value chain, marketing.
• Involve more agribusinesses in AG Component activities; develop special training needed for businesses.
• Elaborate workable schemes for the formation of commercially oriented Farmers’ Groups; define incentives and, at the same time, highlight the importance of obligations that they will have when joining such a group.
• It is necessary to intensify technical assistance to some VC enterprises or risk losing the value of the work that has already been done.
• Some companies require permanent, daily assistance in management but remain skeptical of business service providers (BSPs); a solution is needed.
• EPI should focus more on small scale networking activities that aim at the development of market linkages within the value chain and also broaden and deepen the value chain.
• Emphasize Private Sector Leadership.
• Rethink the long term potential of the apparel sector VC.
• Vocational schools should become market oriented.
• Look more closely at alternatives for access to finance.

This document provides an assessment of the use of IT in Georgia’s logistics industry and a corresponding action plan. During this assessment, representatives from all areas of logistics were interviewed, including sea, rail, road, air, freight forwarders and logistics associations. The plan describes the vision for a trade net for all players involved in the logistics industry. The action plan was developed using the IT Standard Systems Develop Life Cycle (SDLC) and Deloitte’s Portal Methodology modified to reflect the specific requirements from this assessment.

Georgia’s current building codes are outdated. They are a mix of Soviet, American, as well as British, German, and other European standards. The most recent are the seismic codes that were last updated in 2009, but they were mainly based on versions of codes from the 1980’s and need substantial revision. Soviet Construction Norm and Rules (SNIPs) are the most widely used codes in Georgia. But even SNIPs have not been updated for a decade. Although SNIP structural and seismic codes are used, there are no Mechanical, Electrical, and Plumbing (MEP), fire protection, existing buildings, energy conservation, and accessibility codes. Lack of knowledge; awareness; and importance of building safety, design, construction material use, energy efficiency, water efficiency, and operation and maintenance are contributing to poor quality, inefficient, and unsustainable Georgian building stock. Therefore, new and modern building codes are urgently needed.

Reforming the building codes will have a profound impact on the development of construction sector in Georgia. The implementation requires some initial investments that would be fully compensated due to the gains related to the quality of buildings. The consumers of the buildings will benefit as well because of the reduced expenses related to the heating.

The implementation of International Building Codes (IBC) will have important safety and environmental effects as well. IBC is an important first step towards implementation of green building policies that, based on experts’ estimations, will bring energy savings in average of 37%, savings in CO2 emissions by 72%, savings in water use by 40% and savings in solid waste by 70%.

This report outlines the activities and results of a two-week assignment aimed at reviewing and assessing the readiness of the Georgia Revenue Service’s Risk-Based Audit Selection (RBAS) system and recommending improvements to the risk assessment model and the RBAS system.

Currently, Sakpatenti functions as a centralized organization but has a long history of operational redundancies and inefficiencies, which are exemplified by ongoing institutional challenges. In order to improve Sakpatenti’s operational efficiency, reduce its indirect costs, better allocate its scarce funding, and maintain the support of stakeholders, Sakpatenti and EPI initiated a reform supporting a new Electronic-filing (EFS) system. This reform’s goal is to reengineer, standardize and automate 300 plus business processes while consolidating departmental functions to provide 100 plus future processes that will implement the EFS system.

Sakpatenti wishes to modernize the EFS in keeping with the emerging e-Government initiative. The EPI report entitled the Business Process Reengineering Analysis provides information on a set of optimized processes to achieve efficiency, implement cost cutting measures and implement EFS.

The reform envisions a number of long-term improvements to Sakpatenti’s organizational efficiency and customer service as well as economic benefits for Georgia. These benefits include: reducing costs related to time and money for the applicants, decreasing costs related to the IPR application processing and filing in Sakpatenti, increasing the number of applications, and further developing Sakpatenti’s revenue generation capacity as well as various qualitative economic impacts for Georgia.

Currently, Georgia lacks an internationally accredited laboratory to evaluate and test local produce prior to export. Tests conducted by existing local laboratories are not recognized outside of Georgia. As a result, Georgian hazelnut processors are unable to provide credible validation of their products’ quality to international buyers. The result is international skepticism regarding Georgian quality. Only a small cadre of international buyers has the knowledge and risk tolerance to operate in such an environment. As part of EPI’s mandate to serve as a catalyst to increase productivity and employment in export driven agricultural sectors, it is considering working with a local laboratory to assist it to obtain international accreditation.

To assess the potential benefit of such an endeavor, EPI contracted with Coldbrook Ventures LLC, a US based international consulting agency. Coldbrook Ventures LLC was tasked with assessing the potential economic impacts of developing a domestic internationally accredited laboratory for Georgia’s hazelnut market. Thomas C. Beck, the EPI consultant, used both qualitative and quantitative methods to identify areas of potential impact and monetize the effects. The consultant found that an internationally accredited laboratory would spur economic expansion and quality improvements throughout the hazelnut sector. The existence of such a laboratory would also lead to higher productivity by increasing quality assurance. These changes would eventually enable Georgia’s hazelnut sector to attain price parity with the hazelnuts of other nations.

This report, conducted by EPI’s Standards and Compliance Consultant, assesses the Georgian apparel sector’s compliance with minimum health and safety requirements and provides suggestions for improvement. The report was prepared after visiting nine apparel companies in December 2011. The health and safety recommendations for all apparel companies focus on shifting priorities to safe production processes, a healthy working environment, and enhanced morale. These factors will help the apparel sector gain long-term sustainability and be competitive through adoption of international standards.

Due to the accelerated review method of processing applications and increase in the number of major organizations conducting businesses in the area, this consultancy was requested to perform a BPR assessment to streamline future application efforts within Sakpatenti.

To implement an efficient e-filing and patent management processing system, this trip finalized processes for national and international processing for future processes for the following units — Inventions/Utility Models, Geographical Indications, and Finance. Current and future processes for Trademark, Design, External Application, Copyright, and Legal were conducted during the first trip.

In addition to this, a local requirement specialist (Eka Kathamadze) was hired to conduct a full-scale requirements analysis of all business processes that are currently underway. A quality review of those requirements has been necessary to make sure that the requirements gathered are in line with the future processes and the vision provided by Sakpatenti. In addition to the requirements gathering session involving various experts and specialists representing Sakpatenti, external applicant focus groups were initiated, which consisted of patent attorneys and trademark applicants from various pharmaceutical industries. These external applicants were brought in to glean ideas on what they would like to see established for the processing of future applications via the e-filing system.

After the analysis and design of future processes were complete, to implement the requirements analysis phase, it was determined that training needed to be provided to various departmental units within Sakpatenti.

In addition to the above training sessions, to enable BPR facilitation within Georgia, upon request from the Data Exchange Agency, supplementary training was provided to 20+ Georgian organizations to facilitate information sharing between Data Exchange Agency and the 20+ identified organizations within the public sector. This training was provided to help promote the transfer of information between agencies hence facilitating the advent of the e-government initiative within Georgia.

Within the past year, the state procurement system of Georgia has undergone significant changes in terms of the legal framework for state procurement, the procedures and practices applied in the course of conduct procurement, and some of the institutional and organizational arrangements for the system. Foremost among those changes has been the introduction of the Electronic Procurement System (EPS), through which most procurement is now being conducted. That move, which is aimed at boosting efficiency and transparency in state procurement, as well as promoting competition and integrity, is widely acknowledged as being a very significant and bold forward step. While the introduction of the EPS and the other new aspects of the state procurement system are still quite recent, and attempting to make comprehensive and definitive assessment would be premature, it is nevertheless already possible to identify areas in which some midcourse adjustments and corrections and adjustments would be merited, as well as to identify strategic ways in which to solidify and build on the progress achieved, so that further strengthening and evolution of the system could be achieved. The report makes findings and recommendations in particular as to ways in which the legal framework can be further developed and improved, prioritize ways in which procurement procedures and practices can be further improved to maximize quality and value for money outcomes, steps to ameliorate low rates of supplier participation, and measures to increase the capacity of the procurement workforce.