EPI Mid-Term Performance Evaluation

This is a report on the mid-term performance evaluation of the Georgia Economic Prosperity Initiative (EPI) project funded by the United States Agency for International Development (USAID) Mission in Georgia. The project is implemented by Deloitte Consulting.

The evaluation of EPI was conducted during the period April – May, 2013, by a team assembled by Mendez, England & Associates (ME&A) and NORC, both with headquarters in Bethesda, Maryland. The team consisted of three experts – one international and two locals – with experience in agricultural development, private sector and business enabling environment.

EVALUATION PURPOSE AND EVALUATION QUESTIONS

The purpose of the evaluation was to assess the contributions of the EPI toward achieving the USAID/Caucasus’ Development Objective – “Inclusive and Sustainable Economic Growth.” Specifically, the goal was to measure the effectiveness of EPI’s interventions in targeted sectors and value chains. The evaluation’s main objective was to determine the progress of EPI in improving the overall competitiveness of the Georgian private sector through consideration of the project’s design and implementation to date. Other objectives included: 1) assessing the contribution of EPI’s activities toward achieving its high-level results (Productivity, Access to Finance/Domestic Investment, Exports, and Employment); and 2) advising on the practicality of measuring “the success of EPI as a whole” against such targets. The evaluation covered the first two of the project’s four-year term, the period of September 2010 through September 2012.

As requested in the scope of work (SOW), the main questions that the evaluation had to address were:

1. To what extent has the EPI project contributed to improving the business enabling environment in Georgia and the Government of Georgia’s (GoG) adoption of such improvements?
2. What are the main effects of the project on targeted agriculture and non-agriculture value chains and the business enabling environment, including on overall country-level competitiveness?
3. With respect to EPI’s targeted sectors, what has been and what will likely be the contribution of the project toward increasing the four high-level results: Productivity, Employment, Access to Finance/Domestic Investment, and Exports.
4. As a result of EPI initiatives, have businesses changed business practices, as well as increased productivity and the value of their enterprises, thus contributing to the overall competitiveness of value chains and the Georgian private sector?
5. What are the constraints/challenges/issues that inhibit the project’s contribution toward achieving the high-level results during the remaining term of the project?
6. What are the opportunities to improve impact and enhance the implementation and management of similar projects in the future; i.e. gender equity and sustainability?

PROJECT BACKGROUND

EPI is designed to improve enterprise, industry, and country-level competitiveness in Georgia. EPI’s assistance to firms in agricultural, manufacturing and the service sectors aims to increase investment, open new markets, raise productivity, drive domestic and export sales, and create jobs. The goal is to strengthen and institutionalize positive changes in the business enabling environment in the country.

EPI’s activities are organized around four major components:

• Component 1: Business Enabling Environment (BEE), which includes business regulation/licensing, strengthening property rights, investment sector economic governance, trade and customs economic governance, tax administration, procurement/privatization, and agricultural policy.
• Component 2: Agriculture (AG), which includes: a) hazelnut, mandarin (tangerine), greenhouse and open field vegetable value chains (VCs); and b) technical assistance programs provided by agricultural service and input providers to benefit farmers and agribusinesses, associations, agricultural vocational colleges, financial service providers and non-governmental organizations (NGOs).
• Component 3: Manufacturing and Services (M&S), which includes: a) a number of VCs, such as transport and logistics, packaging, Information Communication Technology (ICT), apparel, wine, and Meetings, Incentives, Conferences, and Exhibitions (MICE) tourism; and b) productivity enhancement, investment promotion and market linkages strengthening.
• Component 4: Cross Cutting Activities (CC), which includes upgrading workforce skills in targeted VCs; youth/women employment issues; access to finance; improving the quality of business and economic information; and creation of a sustainable capacity to implement a targeted approach to investment promotion.

MAIN FINDINGS

• EPI successfully integrated many project components and activities and cross-cutting support among project components.
• One of EPI’s strong points is promoting public-private dialog and partnerships, as well as dialog and cooperation among private sector entities.
• EPI is credited with being flexible and responsive to clients’ expressed needs and opportunities.
• Work on development of major pieces of legislation is always uncertain but EPI has had a fair share of success.
• EPI’s work in tax administration filled an immediate need, was demand driven, and is highly appreciated by the client.
• EPI was able to establish working relationships with important project counterparts, including: municipal, village and Adjara regional administrations; Farm Service Centers; and Ferrero’s subsidiary – Agrigeorgia.
• Across all agricultural products, the logistics, organization and quality of training was excellent. Thematic coverage of agriculture production and the quality of training materials was also excellent. Information gathered during the evaluation shows that the Greenhouse training is one of the most notable achievements of the AG Component.
• Prior to EPI’s involvement, the potential of MICE was not widely viewed as an area for development by the GoG or the tourism industry. With EPI’s help, the Georgian National Tourism Agency (GNTA) declared MICE tourism a priority and, for the first time, Georgia appeared as a MICE destination on the global map.
• EPI was instrumental in facilitating the private-public partnership (PPP) that led to the involvement of IT companies in the development of some e-government products and services.
• EPI’s main activity in workforce development – upgrading the skills of garment sector workers through partnerships with vocational educators – is an effort with the potential to positively impact employment, product quality, promotion and export volumes, as well as increased investment in the sector.
• Most project outputs and performance targets have been achieved, some with qualifications.
• EPI’s work to achieve international accreditation of agricultural testing laboratories has the potential to positively impact the success of the local businesses.
• EPI’s work in development of a modern construction code may not succeed but this is largely for reasons outside of the project’s control.
• The work in intellectual property protection was generally well implemented, highly valued by the beneficiaries and has had, or is likely to have, positive impacts.
• The investment sector economic governance sub-component was limited in scope; therefore, specific accomplishments are hard to identify at this time.
• The Georgian state procurement system was essentially a Georgian effort aided by European donors; EPI has had limited influence on or interaction with the system.
• There remains a good deal of skepticism that working with small Georgian farmers is a long-term strategy.
• Work remains to be done in the AG sector on improving business practices and market access.
• While the work on crop insurance was highly successful, there remain serious issues of sustainability.
• The Transport & Logistics (T&L) subcomponent of the project was a priority. The Trade Facilitation System (TFS) is a very good example of EPI’s success with Public Private Dialog (PPD) and PPP and will likely turn out to be an important part of the project’s legacy.
• Enterprise owners are highly appreciative of the analytical work and recommendations provided by EPI’s international and Georgian consultants. However, there is a risk that many recommendations will stay on paper given factors such as poor access to finance and low capabilities among enterprise management.
• Most beneficiaries in the apparel VC are optimistic about results, which they expect in project years 3 and 4. Beneficiaries highly value EPI’s activities in addressing low productivity in the apparel companies as a major impediment.
• The social capital or economic benefits derived from the cooperation among value chain actors seems to be low.
• EPI’s work on financial leasing has been useful and successful. However, results for farmers are not yet visible and some significant challenges remain.
• EPI’s work on financial advisory services resulted in a minor increase in volume of loans going to farmers and small, medium, enterprises (SMEs); however, there remain many challenges to increase the flow of credit to agriculture and small business.
• EPI has contributed to the development of an indigenous capacity in investment promotion. However, concrete results of these efforts have not yet been realized and there are mixed views on its success and viability
• Key stakeholders in government have generally positive but mixed perceptions about the project; the project’s rough start colored perceptions, which continues to this day.

MAIN CONCLUSIONS

• EPI has made, and continues to make, important contributions to business climate improvement.
• Most major contributions of EPI have been in assisting in the implementation of good policies already established and, particularly, in providing concrete, tangible means of implementation that increase transparency and administrative efficiency, thereby increasing certainty and reducing transaction costs.
• EPI’s emphasis on PPD has been instrumental in introducing to the GoG a new way of doing public business that can enhance the government’s ability to make good policy; however, there are questions of sustainability.
• EPI has had moderate success in promoting useful laws and regulatory acts, which may increase over the remaining years of the project.
• EPI has provided no formal training in policy analysis and development techniques to GoG staff.
• Among the most significant effects the project has had on the business enabling environment have been increased transparency and decreased transaction costs of administrative processes, and more open communications through PPD. Secondary effects include modernization of some key laws and regulations.
• Improvements in farming practices are visible and over time should lead to better quality products and enhanced country-level competitiveness.
• EPI has made and continues to make important contributions towards establishing Georgia as a Caucasus Transit Corridor, which can have a direct impact on high-level indicators and overall country competitiveness.
• EPI was instrumental in the support of very important promotional activities, which were trend-setting for the wine tourism and MICE sectors in Georgia. Those activities arguably put Georgia on the wine tourism and MICE maps for the first time.
• There are some short-term positive impacts on high-level results, but likely not in the magnitudes suggested by EPI. Most impacts are very difficult to estimate and the project was not structured for high-level evaluation. It is reasonable to expect medium- and long-term positive impacts on some high-level indicators.
• Few visible changes in business practice have been observed in the AG Component at this time.
• Changes to business practices are more visible in M&S VCs. However, even there, utilization is slow and affected by other variables such as the availability of financing. It is likely that the businesses will change their practices provided that EPI continues and, perhaps, intensifies its support to some value chain enterprises.
• The change in the government may put on hold some activities but should not be considered as a constraint that inhibits achievement of the project’s high-level objectives during the remaining term.
• Sustainability of PPD, one of EPI’s signature accomplishments, may depend on further institutionalization of the practice.
• The competitive advantage of Georgia’s apparel sector and, therefore, its long-term impact on economic growth is questionable. However, the medium-term impact on the sector’s economic indicators can be significant.
• Access to finance, and in particular finance for small farmers and SMEs, remains problematic.
• Sustainability of investments in small farmers is open to question.

MAIN RECOMMENDATIONS

• Analytical work could be done in a shorter period of time in order to start focusing on implementation in the earlier period of the project.
• Emphasize time of delivery of promised products and services.
• Avoid selecting project activities based on likelihood of achieving performance targets.
• Continue to work on good performance indicators.
• Emphasize flexible planning.
• Going forward, emphasize proven strengths and phase out less productive initiatives in business climate reform.
• Emphasize institutionalization of PPD – Competitiveness Council, business associations, administrative procedure laws, etc.
• Provide more formal training in policy analysis and development to GoG.
• Current initiatives in the AG Component should be enhanced including, for example, training in marketing and aid to cooperatives.
• To organize better trading channels for farmers, the project should concentrate on the last link of the value chain, marketing.
• Involve more agribusinesses in AG Component activities; develop special training needed for businesses.
• Elaborate workable schemes for the formation of commercially oriented Farmers’ Groups; define incentives and, at the same time, highlight the importance of obligations that they will have when joining such a group.
• It is necessary to intensify technical assistance to some VC enterprises or risk losing the value of the work that has already been done.
• Some companies require permanent, daily assistance in management but remain skeptical of business service providers (BSPs); a solution is needed.
• EPI should focus more on small scale networking activities that aim at the development of market linkages within the value chain and also broaden and deepen the value chain.
• Emphasize Private Sector Leadership.
• Rethink the long term potential of the apparel sector VC.
• Vocational schools should become market oriented.
• Look more closely at alternatives for access to finance.