Abstract
This study evaluates USAID's experience with evaluation since 1995, when Agency policy changed from requiring every project to be evaluated to recommending that evaluations only be done in response to management need. Key findings are noted below. Considerable evaluation work is being used in decisionmaking and program planning. Some 60% of respondents to the study questionnaire reported that evaluation has been critical to their understanding of activity performance. Fieldwork presented many other examples of the use of evaluation in mission decisions and debates. However, USAID is doing markedly fewer evaluations, and only a small percentage of those completed is being submitted to Washington. Of most concern is the very limited number of in-depth program evaluations. The monitoring/evaluation balance has swung too much toward performance monitoring. Evaluation differs from performance monitoring and has been ignored too long. Most current field evaluations focus on activity performance, especially by implementing partners, whereas strategic objective (SO)/R4 monitoring systems assess progress at the strategic level. There is a disconnect between the two that cannot be closed by more performance monitoring. SO leaders, in the main, have not been planning for strategic evaluations at the SO level. USAID managers, particularly busy team leaders, skim reports and react to the most pressing actions needed. But only the most committed managers find time to do evaluations at all -- one USAID manager stated that he spent 90% of his time on things that are required and that evaluation had a hard time making it to the top of the 10% list -- and most evaluations do not circulate beyond the partner and the strategic objective (SO) team. Professionals in the field need more support if they are to develop effective programs. The dominant use of the "fly-in" method for USAID evaluations is having a serious effect on the quality of the evaluations. In this approach, a team comes for 4 to 6 weeks and interviews the mission, the activity staff, and "representatives" of the local people. There is not enough time to obtain a representative sample. The team frequently tells the USAID manager pretty much what he or she already knows. Insufficient knowledge and understanding of evaluation planning and methods have led to this approach, which is not worth the time and money being spent on it. Fairly minimal advice and guidance from evaluation experts prior to the evaluation could lead to significant improvement. Refining evaluation questions and identifying suitable and timely methods are the key. By the time an evaluation team arrives, it is too late to affect the methods in any significant way. Professional commitment is the primary incentive for evaluation. In general, professional staff do not see any Agency-wide incentive to advance learning through evaluations. Also, too many evaluations are linked with success stories. Continuing this practice will call into question the objectivity and independence of evaluation findings and conclusions. The biggest structural change in USAID's work is the increasing reliance on non-career staff, few of whom have any long-term stake in Agency learning. Much of the current evaluation work is being done by partner organizations. They are learning from the experience; USAID is not. Privatizing monitoring and evaluation undercuts the flow of information as organizations recognize information as an element of competitive advantage. Moreover, contracting and granting rules create serious barriers to learning from experience and applying lessons learned. The key recommendation is that senior Agency managers make learning about what works and what doesn't of primary importance. Only with visibility and high-level support will evaluation, along with performance monitoring, be a means by which "truth speaks to power." More importantly, USAID needs to find a way to "use truth to empower." (Author abstract, modified)