eVALUation Matters - Independent Development Evaluation (IDEV)

eVALUation Matters A Quarterly Knowledge Publication on Development Evaluation

The Problem with Development Evaluation and what to do about it First Quarter 2017

Independent Development Evaluation African Development Bank From experience to knowledge... From knowledge to action... From action to impact

eVALUation Matters

Evaluator General:

Is a quarterly publication from Independent Development Evaluation

Rakesh Nangia

[email protected]

at the African Development Bank Group. It provides different perspectives and insights on evaluation and development issues. Editor-in-Chief:

Managers: Rafika Amira

[email protected]

Samer Hachem

[email protected]

Jacqueline Nyagahima, Knowledge Management Consultant.

Karen Rot-Munstermann [email protected]

Acknowledgments:

Talk to us:

IDEV is grateful to all contributors, reviewers, editors, and proofreaders who worked on this issue, in particular: ❙❙ Mireille Cobinah-Ebrottie, Evaluation Knowledge Assistant ❙❙ Kobena Hanson, Knowledge Management Consultant ❙❙ Aminata Kouma, Junior Consultant ❙❙ Najade Lindsay, Junior Consultant ❙❙ Karen Rot-Munstermann, Manager, IDEV 3 ❙❙ Kate Stoney, Communication Consultant ❙❙ Prosper Yepdi, French editing and translation Design & Layout: Visual Identity Creative Ltd. Director: Jonathan Newham Editorial design: Francisco Marrero About Independent Development Evaluation The mission of Independent Development Evaluation at the AfDB is to enhance the development effectiveness of the institution in its regional member countries through independent and instrumental evaluations and partnerships for sharing knowledge. Disclaimer: The views expressed in this publication belong solely to the authors, and not necessarily to the authors' employer, organization or other group or individual.

© 2017 – African Development Bank (AfDB)

Phone (IDEV)

+225 2026 2041

Phone (AfDB standard)

+225 2026 4444

Write to us: 01 BP 1387 Avenue Joseph Anoma, Abidjan 01, Côte d’Ivoire e-mail us: [email protected] Find us online: idev.afdb.org afdb.org

First Quarter 2017

Development evaluation has been around for a while now. However, the perception is that it does not garner its expected level of influence. Why is there poor assimilation of the lessons learned and recommendations from evaluation? Where do the problems really emanate? Is it from the users of evaluations, or from the evaluators? Is it from the policy or the process? Contributors to this issue approach the challenges of development evaluation from various perspectives, drawing on their wealth of evaluation knowledge and experience, and their grasp of the evaluation landscape in Africa.

News in pictures, page 80 4

From the Evaluator General’s Desk Rakesh Nangia, African Development Bank “If history repeats itself, and the unexpected always happens, how incapable must Man be of learning from experience”, said G.B Shaw, the noted playwright. Alarming as the assertion is, there are innumerable instances I have observed over two decades in development that resonate with this sentiment. While perusing the large number of evaluations that IDEV has conducted over the years, I am surprised to find that certain recommendations have persisted over several evaluations, some spanning 10 years.

8

For better use and impact of high‑level independent evaluations: a user’s perspective Marc Cohen, African Development Bank This paper takes the perspective of the users, that is, management and staff in development agencies who should benefit from and use research produced by independent evaluation offices. It argues that current practices in the independent evaluation function are not sufficiently conducive to the integration of the user’s perspective at the various phases of the evaluation process, which can ultimately limit the effective use of the evaluation. The paper proposes an enhanced collaboration and complementarity between self-evaluation and independent evaluation processes.

Table of contents

20

Independence in development evaluations: Getting the balance right with stakeholder engagement Akua Arthur-Kissi, African Development Bank Considering the importance of stakeholder engagement in ensuring credible and useful evaluations, maintaining independence while achieving this has become a recurring challenge in development evaluations over the years. This paper focuses on particular challenges of possible tensions among key actors in both ensuring independence during the evaluation and applying adequate stakeholder engagement efforts in the evaluation. The paper suggests that there is not one single approach that fits in all circumstances in dealing with this challenge.

32

Challenges in Conducting Development Evaluation: Dealing with perceptions Pindai M. Sithole, Centre for Development in Research and Evaluation International Africa The paper reflects on the challenges that evaluators sometimes encounter when they conduct development evaluations. The context of the discussion is the author’s own development evaluation experience, particularly in Africa and Asia. The challenges that the project or program funders and implementers bring into the evaluation process include fear of evaluation, contextual detachment of projects/programs, limited resource commitments to evaluation, language and cultural barriers as well as weak or misleading evaluation sensitization to project/program services recipients/beneficiaries and other stakeholders.

42

Rethinking Knowledge Management and Independence – Promoting the Use of Evaluation to Support Organizational Learning Erika Ismay MacLaughlin, African Development Bank This paper considers learning in the context of organizations. It focuses on two distinct user groups: senior managers and executive directors; and operational staff. Whereas the AfDB’s current evaluation strategy does not clearly identify how different user groups are expected to consume and use evaluative information, the academic literature suggests that organizational learning is distinct from individual learning and is driven by a unique set of processes. In order to better leverage independent evaluation to support institutional learning, it is necessary to critically examine how organizations learn.

56

Elaboration and running of a development evaluation policy Bertrand Savoye, French Development Agency The development evaluation policy of a technical and financial partner (tfp ) often has several objectives, which are sometimes difficult to reconcile because they require different organizational methods and evaluation methods for their achievement. The TFP will seek to resolve these problems through arbitrations and compromises that can evolve pragmatically over time, according to its priorities.

66

Evaluating for Development: Reshaping our evaluation criteria? Zenda Ofir, Evaluation specialist This paper argues that “development evaluation” does not attend sufficiently to “development”, especially considering the challenges facing the Global South. It is necessary to shift to a more dynamic approach that will enable us to evaluate for development. This, in turn, will compel us to make sure that the criteria that determine “what” and “how” we evaluate are refined in principle and in practice, cognisant of what defines an intervention that truly fosters development.

Published March 2017 Editorial Calendar 2017 Q2 2017 Comprehensive Evaluation of Development Results: Behind the scenes Q3 2017 Evaluation in the era of the SDGs Q4 2017 Evaluation as a driver of reform in International Financial Institutions Q1 2018 Building supply and demand for evaluation in Africa Call for contributions: Evaluation Matters is a quarterly magazine issued in both English and French, by IDEV, Independent Development Evaluation of the AfDB. We welcome topical articles from writers’ original work and will be pleased to consider contributions for the themes in the above Editorial Calendar. See details at http://idev. afdb.org/en/document/ editorial-calendar-2017

First Quarter 2017


eVALU

From the Evaluator General’s Desk, page 4

Evaluation Matters: The Problem with Development Evaluation and what to do about it

“Despite years of evolution in development evaluation, systematic learning and its application continues to present challenges”.

The Problem with Development Evaluation and what to do about it First Quarter 2017

Find eVALUation Matters at http://idev.afdb.org/ en/page/evaluationmatters-magazine

eVALUation Matters First Quarter 2017

“If history repeats itself, and the unexpected always happens, how incapable must Man be of learning from

From the Evaluator General’s Desk

experience”, said G.B Shaw, the noted playwright. Alarming as the assertion is, there are innumerable instances I have observed over two decades in development that resonate with this sentiment. While perusing the large number of evaluations that IDEV has conducted over the years, I am surprised to find that certain recommendations have persisted over several evaluations, some spanning 10 years. Evaluations highlight the same problems over and over again, making similar recommendations to address them. And yet, for some inexplicable reason, the problems persist. Take, for instance, the issues of quality of supervision and weak monitoring of country level portfolios. These were highlighted in recommendations from the country assistance evaluation, “Review of 2000–2004 Country Portfolio Review Reports” in June 2005 and they resurfaced in the Comprehensive Evaluation of the Development Results of the African Development Bank Group (2004–2013), almost a decade later. Although I am gratified that measures are finally underway to address these issues, it is hard not to consider the cost of this “missed learning” to the Bank and more importantly its regional member countries.



No one size fits all – an overused statement amongst development practitioners. The intervention should be tailored to suit the beneficiary. Even if an intervention is repeated, it is likely to differ from the original form. It is thus imperative to assess previous interventions and to draw lessons to inform the subsequent stage(s). Given the obvious nature of these statements,

“Despite years of evolution in why then is it that although develdevelopment evaluation, systematic opment evaluation has been active learning and its application for several decades, the knowledge continues to present challenges”. it generates is not adequately influencing learning and decisions made by organizations who have committed serious resources? Why is there inadequate assimilation of the lessons learned and recommendations from evaluation? Are we truly incapable of learning from our own experiences and destined to make the same mistakes, or is there a clever way to break the cycle? Use of evaluations is a salient aspect that thread through this edition of eVALUation Matters. The reasons of poor uptake stem from various sources. These include getting the balance right between independence and stakeholder engagement, between project or program funders and implementers, limited resources and capacity, and language and cultural barriers. How does evaluation lend itself to learning; and how evaluation policies are put into practice. Finally, against the backdrop of the transition from the Millennium Development Goals to the “no one left behind” Sustainable Development Goals, evaluators are asking themselves whether the evaluation criteria we employ are still the right ones.


5


The contributors to the current issue of eVALUation Matters seeks to address these challenges from various perspectives, drawing on their wealth of evaluation knowledge and nuanced grasp of the landscape in Africa. This edition presents a more informed view on the problems and advice from development and evaluation practitioners. I would encourage you to draw on the vast experience and knowledge of our contributors to enrich our individual and collective intellectual capital and use it to shape our respective work agendas and collective lived experiences. Happy reading!

About the Evaluator General Rakesh Nangia is the Evaluator General for Independent Development Evaluation at the African Development Bank. Prior to joining the AfDB, he spent 25 years at the World Bank, where he held several positions including Director of Strategy and Operations for the Human Development Network and Acting Vice-President for the World Bank Institute. He attended the Indian Institute of Technology in Delhi and Harvard University and holds degrees in business administration and engineering.


7

For better use and impact of high-level independent evaluations: a user’s perspective

This paper takes the perspective of the users, that is, management and staff in development agencies who should benefit from and use research produced by independent evaluation offices. It argues that current practices in the independent evaluation function are not sufficiently conducive to the integration of the user’s perspective at the various phases of the evaluation process, which can ultimately limit the effective use of the evaluation. The paper proposes an enhanced collaboration and complementarity between self-evaluation and independent evaluation processes.


Marc Cohen, African Development Bank

“D

evelopment practitioners rarely appreciate and use the full benefit of evaluative research” and “too little rigorous research of relevance to development gets done” (Ravallion, 2008). These views, candidly expressed by a World Bank researcher, are commonly advanced by development practitioners in agency circles. But why is it so? Why this perception that evaluation may fail to feed actual practices in development agencies? And what can be done about it? This paper takes the perspective of the users, i.e. management and staff in development agencies who should benefit from and use research produced by independent evaluation offices. It focusses on high level evaluation, i.e. sectoral, corporate or special studies, often aimed at promoting reforms and influencing strategic directions of a development agency. The paper argues that current practices in the independent evaluation function are not sufficiently conducive to the integration of the user’s perspective at the various phases of the evaluation process, which can ultimately limit the effective use of the evaluation. Users are not commonly associated with the evaluators’ choices regarding the topics, questions and methods of the studies, elements that have obvious bearings on the findings and results. Issues related to the sustainability and the actual feasibility of the evaluation’s recommendations – whether financially, institutionally, culturally or else – are rarely considered by evaluators who tend to leave the choice of

the detailed implementation modalities at the discretion of the management of the agency. The resources of self-evaluation systems are not systematically captured and its findings insufficiently addressed through independent evaluation. In fact, the independent evaluation function may be subject to a tension between (i) the production of evaluative research that is relevant to users and effective in influencing their practices and managerial decision, and (ii) a strict adherence to the principles and quality standards of development evaluation as defined by the OECD/DAC (OECD, 1991; OECD, 2010). As a means for releasing this tension – and at the same time strengthening the utility of development evaluation – this paper proposes an enhanced collaboration and complementarity between self-evaluation and independent evaluation processes.

Setting the evaluation agenda: a process driven by policy considerations, politics or perceptions? The evaluation agenda, the set of high level studies that composes the work program of independent evaluation, normally emerges as the outcome of consultations with different stakeholders who may naturally have different perspectives. In the case of Multilateral Development Banks (MDBs), independent evaluation units will consult the shareholders through the board of executive directors (to whom the unit usually


9


reports), client countries, and management and staff of the institution. In this process though, the political context and considerations of the larger shareholders will tend to prevail. The choice of topics of high-level evaluations is thus rarely fortuitous: it reflects an underlying agenda, driven by priorities of a political nature that are not necessarily shared by practitioners whose main concern remains technical and focused on development policy issues. One could argue that the growing skepticism regarding the role of aid and the ability of agencies to make good use of the aid money has largely dominated a politicized development agenda over the past decade. “Development results”, “value for money”, “institutional efficiency”, “donor coordination”, “operational quality”, “sector and thematic selectivity”, to name a few, are the recurrent themes of high level evaluations. This feature is a necessary accountability mechanism: development practitioners must provide credible responses to (political) decision-makers who are accountable to their citizens. Development policies are subject to public scrutiny, like any other field of public policy, and this is where the role of independent development evaluation is critical. Furthermore, a (politically-motivated rather than policy-oriented) process for setting the agenda, has proven effective to generate relevant topics for development research and practice alike. For instance, the fact that parliaments, in recipient as well as in donor countries, demand greater scrutiny on aid budgets largely influenced the global push towards aid data transparency and audit exercises. Tense fiscal situations in OECD countries coupled with the diminishing “aid appetite” in political platforms and the public opinion at large, stimulated a wave of research on aid effectiveness which gave rise to the “results agendas” implemented by many agencies. Evaluative research prompted the development

of a generation of sophisticated techniques and tools at the micro level (such as evidence-based project ratings, quality-at-entry standards, logical frameworks, etc.) as well as corporate level (such as corporate results frameworks, balanced scorecards, quality assurance and portfolio dashboards, etc.) which now constitute the backbone of agencies’ self-evaluation systems. One issue, as further elaborated in the following sections, is that the mass of information generated by the self-evaluation systems remains insufficiently synthesized, analyzed and used by both the providers and the users of evaluations.

“The choice of topics of highlevel evaluations is thus rarely fortuitous: it reflects an underlying agenda, driven by priorities of a political nature that are not necessarily shared by practitioners whose main concern remains technical and focused on development policy issues”. The political interference with the evaluation agenda becomes problematic when some form of dominant development thinking or ideology biases the evaluation questions and methods, which in turn can influence the scope and nature of findings. There can be a perception among development practitioners that evaluation can be used to justify decisions that have already been made in higher-level circles sometimes outside the development sphere. A joint donor study on the question of aid exit seems to support this opinion (Slob and Jerve, 2008). A key message of this study is that aid entry and exit are essentially part of foreign policy: “Aid policy is increasingly being coupled to foreign and security policy concerns and aid is increasingly being used as a foreign


11


policy instrument… The politicization of aid, thus, has led to a higher level of aid volatility and an increased number of exits running counter to basic principles of good donorship, emphasis on predictability and concern for the rights of beneficiaries”. Often guided by independent evaluation results, MDBs have operated important shifts in their portfolios in the past decade, typically towards infrastructure and away from the social sectors or soft interventions; some sectors or work areas have had “ups” and “downs” and “ups” again depending

“In practice thus, the experience and knowledge of the evaluators will influence the evaluation design much more than an objective review of the pros and cons of the various methods”.

on their perceived relevance and level of priority in the broader political agenda; the use of specific instruments, such as budget support, experienced successive (and costly) periods of favor and disgrace without clear entry and exit strategies and in the absence of solid evidence (Rønsholt, 2014). While recognizing the necessary and positive role of a politicized development evaluation agenda, there is a risk that the technical and policy-focused perspective of the development practitioners could be neglected in the process. The key element thus, is to achieve a right balance between political and technical perspectives through consultations with a wide variety of stakeholders on an equal footing. Practitioners or “users” of evaluation are arguably less prone to external pressures; of course, they have their own interests and preferences, but they can help frame

12

the evaluation topics and questions on the basis of their experience and knowledge and with a clear focus on the evaluation utility principle.

Developing the evaluation methodology: an objective choice from a toolbox or the evaluators’ preference? The quality standards for development evaluation normally require that the evaluation questions inform the development of the methodology (OECD, 2010). The role of the evaluator is thus to ensure that the methodology will be fit for the purpose, responding to the evaluation questions in the most objective manner based on the information that can be reasonably gathered within the constraints of time and budget. Evaluation though, is an art rather than a science: alternative or mixed methods can be retained by the evaluators and evaluators do tend to have their own preferences and set of skills. “There are no evaluations that can be done without the evaluator’s own experiences, values, beliefs, and expectations playing an important role…” (McDavid, Hawthorn, 2006). In practice thus, the experience and knowledge of the evaluators will influence the evaluation design much more than an objective review of the pros and cons of the various methods. To quote Ravallion again: “far too often though, it is not the question that is driving the research agenda but a preference for certain types of data or certain methods; the question is then found that fits the methodology and not the other way round” (Ravallion, 2008). Whether one agrees or not with this view, it highlights the need to address the (unavoidable) biases from the producers of evaluations in setting the evaluation question and in choosing the evaluation methods.



Because development evaluation is a relatively recent field with few people trained in complex evaluation techniques, design and methodological choices have been rather limited in practice. Robust but somewhat sophisticated and costly quantitative methods or combined qualitative and quantitative techniques are barely used in high level evaluations. As a result, observations, desk reviews of documentation and basic qualitative techniques such as interviews and focused group discussions have so far been the preferred methods of investigation. Broadening the spectrum of techniques and methods used in high level evaluation would certainly mitigate the biases in the design phase and later enhance the robustness of findings. Another relevant question is whether an upfront involvement of users in the selection of the evaluation methodology would mitigate the risks of biased choices. Indeed, while the methodological choices are critical to the robustness of findings, there is no established procedures for associating users in the study design. Using self-evaluation findings – normally the reflection of users’ concerns – to help formulate the evaluation questions is not considered a quality standard either. One reason is the view that closely associating users with the methodological choices could be detrimental to the independence principle. Nonetheless, the perception of a lack of consultation and transparency in critical evaluation choices can be a cause for misunderstanding: where evaluation findings are in large part the reflection of the evaluator’s choices made earlier in the process and with little involvement of users, there is a risk that users look at some of the evaluation results as partial or inaccurate.

Getting the evaluation findings right: an independent informed opinion or a negotiated process? A reconciliation between independent and self-evaluation? As argued above, the potentially biased mechanisms for setting the evaluation topics, questions and methods may lead to a set of findings that are grounded on partial evidence and/or not fully understood by users. Some of the evaluation findings can be somewhat disconnected from the actual experience and the insider’s knowledge of the decision makers and staff of the institution. To prevent such a situation, a process of consultation and validation of the findings, through which a draft report is discussed with users and screened for possible errors, inconsistencies and omissions, usually takes place. Of course, users and producers of evaluations do not have to agree on an independent report, but in practice, a common understanding of what the critical issues are and how they can be addressed is necessary; it can also strengthen the user’s buy-in of the study in view of its actual use. Through this consultative process thus, evaluation findings can be refined to emerge as the reconciliation of two independent and possibly divergent opinions and systems: the opinion of the evaluator on the one hand (informed by own knowledge and independent evaluation findings) and the opinion of the user on the other hand (informed by own knowledge and self-evaluation findings). One can argue that this process, sometimes lengthy and belated, could have been largely avoided if user’s perspectives and concerns had been better integrated early on and if self-evaluation had been an


13


integral part of the scope of the independent evaluation. In fact, the two evaluation systems tend to be run in parallel with too little synergies and interactions. Self-evaluation, by definition produced by an entity engaged in the activity to be evaluated, presents the obvious advantage of being of direct relevance to the users. The utility principle normally requires that an independent development evaluation reviews the relevant self-evaluation resources.

“From a user’s perspective, selfevaluation resources and findings should be explicitly part of the independent evaluation process.” Yet, self-evaluation resources can often be dismissed or insufficiently tapped by evaluators. Indeed, a recent World Bank study found that “staff do not view self-evaluation systems as a source of timely, credible, and comprehensive evaluation… and engage with the systems with a compliance mindset” (World Bank, 2016). The lack of confidence in the self-evaluation system – its weaker rigor and questionable candor – is a cause for neglect by independent evaluation. Yet, because self-evaluation is routinely used for guiding performance management and for capturing knowledge, it has potential to trigger change. From a user’s perspective, self-evaluation resources and findings should be expli citly part of the independent evaluation process. This can be achieved by ensuring that (i) the findings of self-evaluation, where available, are specifically considered by the independent evaluation (and possibly responded to through an evaluator’s opinion), and (ii) the strengths and weaknesses of self-evaluation are 14

systematically assessed by the independent evaluation. This approach would lead to an increased convergence between self and independent processes (making the process of review and informal validation of evaluation findings rather superfluous) without compromising the independence principle.

Implementing evaluation recommendations: a realistic path or mission: impossible? Whether evaluators should use findings to formulate recommendations is a question that has been debated for some time and will still be. Some evaluators have expressed reluctance to enter into this territory, leaving it to the users to decide how to best use the findings. Others have attempted to define frameworks aimed at guiding whether recommendations are needed or not (Iriti et al. 2005). From the user’s perspective, the recommendations are an essential part of the evaluation process. A development practitioner will find little use of commissioning an evaluation study if it does not lead to some form of policy action or prescription aimed at improving a situation. The dominant view thus, is that evaluation studies should include recommendations for due consideration by users. The systematic response to and follow up on recommendations is also one of the standards of quality for development evaluation (OECD, 2010). In many agencies, this entails the preparation of a formal response from Management and the implementation of agreed follow up action. There are no clear guidelines or standards however, aimed at guiding the recommendations formulated by independent evaluators. Practices do vary. What constitutes a “good” recommendation is a question that has apparently not received much attention or that has not received



consensus to date. The number of recommendations understandably varies from one evaluation to another, sometimes as few as a couple but more often as much as a dozen. At the corporate level though, the question of the consolidation and coherence of recommendations can be raised. For instance, the management response tracking system recently established by the AfDB currently counts some 430 actions that management has committed to undertake in response to 230 recommendations formulated in 26 independent evaluation studies carried out since 2012. Are these recommendations (and related actions), accumulated over just a few years, a coherent set for driving change in the institution? How can they be phased or prioritized? Are they potentially conflicting? Are they manageable? Can progress on actions be tracked at a reasonable costs? And obviously, how much is the implementation of these actions going to cost? These questions, as legitimate as they may sound, remain unanswered in the current practice of independent evaluation. As the portfolio of reforms – triggered by independent evaluation but also by audit reports or by external commitments – is rapidly growing in development agencies, both producers and users of evaluation will have to pay more attention to the issues of phasing, prioritization, and implementation of reforms. The level of detail or the specificity of recommendations, also greatly varies from one study to another. In the case of high level thematic and corporate evaluations, recommendations tend to be labelled in a rather generic form. From the perspective of user, a vague phrasing is often insufficient for deciding on a specific course of action. Typical recommendations for instance are worded as “improve the efficiency of process XXX” or “enhance the quality of product YYY” or “more strategically select priorities ZZZ”. These formulations

present the advantage of being agreeable to the producers and users alike as they leave much room for interpretation and manoeuver. Management and staff of the agency are thus in a position where they can comfortably agree on most recommendations and respond by pointing to a number of actions. The question thus, is about the additionality or real impact: what changes have been triggered by the evaluation and what would have happened in the absence of the evaluation? Of critical importance to the users is of course the capacity of the institution to use the evaluation findings and to propose actions in response to the set of recommendations. Recommendations are usually formulated as statements normally derived from findings but not accompanied by cost estimates or feasibility analysis. Budget and human resources considerations are obviously critical aspects; feasibility also refers to institutional, cultural and technical factors. From the perspective of the user, an insufficient “contextualization” can be a striking limitation of independent evaluation; a consequence is that evaluation research does not necessarily provide useful knowledge to guide practice in other settings. Evaluators are often tempted to recommend a “best practice” or an optimal solution that has been adopted in certain contexts whereas a “second best” option – perhaps less satisfactory from a technical perspective but easily implementable – could be more appropriate in the specific context of the study. For instance, recommendations of high level evaluations often consist of introducing a new process e.g. develop quality standards, conduct operational fragility assessments, introduce evidence-based project ratings, delegate authority for procurement and disbursement, etc. The recommended process may by technically sound to address the specific issue at stake in the evaluation study (when considered in isolation), but it may be redundant


17


with other business processes, require skills that are not available, conflict with the corporate culture or the management style, etc. Typically, evaluations do not elaborate on the “how much” and the “how” of the recommendations, although these are the key concerns of the users who will become responsible for implementation.

“Intuitively, it seems that users are best placed to contextualize the evaluation results and screen the recommendations based on feasibility criteria.”

Intuitively, it seems that users are best placed to contextualize the evaluation results and screen the recommendations based on feasibility criteria. Thanks to their intimate knowledge of the institution they work in, its culture, its formal and informal networks, the management and the staff members can provide useful insight into the quality and realism of recommendations. Through evidence generated by the self-evaluation system, they can also propose concrete solutions for consideration of the independent evaluation. Self-evaluation has indeed, the advantage

18

of being much closer to action than independent evaluation and it offers more potential for fixing relatively simple problems (Heider, 2016). There is little doubt that an open dialogue on the evaluation recommendations between users and producers of evaluation would have several virtues in itself, such as promoting knowledge, buy-in and commitment in the institution.

To conclude From the perspective of users, the quality standards and the independence principle can be seen as obstacles to their role and inputs in the independent evaluation process. Users may thus tend to consider the robustness and utility of independent evaluation as imperfect. Self-evaluation systems remain their main source of information for making management decisions, addressing performance issues and capturing knowledge. The dominant point of view of evaluators on the other hand, is that a close association of users in the evaluation process, from the design phase to the formulation of recommendations, may compromise the principle of independence. Producers of evaluation tend to have a low level of confidence in self-evaluation and they may see user’s perspective as partial and of limited interest in the search for solutions.



In practice, aid agencies have developed formal and informal ways of mitigating the twin tension between the independence and the usefulness principles of development evaluation and between the potentially divergent results or interests of the independent and the self-evaluation systems, respectively. The final outcome though, is certainly sub-optimal, both in terms of effectiveness (actual use and impact of the evaluation) and efficiency (complex processes mean that a high level evaluation typically takes 18 to 24 months to be completed).

Three directions for enhancing the impact of independent evaluation are proposed in this paper: (i) an upfront involvement of users in the design phase of the evaluation study, (ii) improved synergies between independent and self-evaluation (in particular through the use and assessment of self-evaluation resources and techniques by the independent evaluation) and (iii) a systematic feasibility assessment of independent evaluation recommendations as a basis for user’s action.

References Heider, C. (2016), independent versus self-evaluation: is there a place for both? World Bank Blog. https://ieg.worldbankgroup. org/blog/independent-versus-self-evaluation-there-place-both Iriti, J.; Bickell, W.; Nelson, C. (2005), Using Recommendations in Evaluation: A Decision-Making Framework for Evaluators, American Journal of Evaluation, 26. McDavid, J.; Hawthorn L. (2006), Programme Evaluation and Performance Measurement, an Introduction to Practice. OECD (1991), DAC Principles for the Evaluation of Develop-

Author’s profile

ment Assistance.

OECD (2010), Quality Standards for Development Evaluation. Ravallion, M. (2008), Evaluation in the Practice of Development, World Bank Policy Research Paper 4547. Rønsholt, F.E. (2014), Review of Budget Support Evaluations, DANIDA. Slob, A. and Jerve, A. (2008), Managing Aid Exit and Transformation, Joint donor evaluation. SIDA, NORAD, DANIDA. World Bank (2016), Behind the Mirror, A report on self-evaluation systems at the World Bank Group.

Marc Cohen is currently Manager, Quality Assurance, at the African Development Bank (AfDB). Between 1998 and 2009, he worked for the Asian Development Bank (ADB), successively as team leader in operational departments, principal specialist of the results unit, and head and focal point for assistance in fragile and conflict-affected situations. Prior to that, he was a sector analyst and social development expert with UNESCO where he coordinated the first UNDP-led inter-agency Human Development Initiatives. Marc has operational experience in some 30 countries in transition and/ or reconstruction in Asia-Pacific, Africa and Eastern Europe. He obtained post-graduate degrees in Economics, Political Science and Management from the Universities of Oxford, the Institut d’Etudes Politiques de Paris, Paris IX Dauphine and Paris I Sorbonne.


19

Independence in development evaluations – Getting the balance right with stakeholder engagement

Considering the importance of stakeholder engagement in ensuring credible and useful evaluations, maintaining independence while achieving this has become a recurring challenge in development evaluations over the years. This paper focuses in particular on the challenge of possible tensions among key actors in both ensuring independence during the evaluation and applying adequate stakeholder engagement efforts1 in the evaluation. It builds on discussions from literature, experiences and observations of evaluation practices among the multilateral development institutions, including the African Development Bank, to shed light on possible ways to deal with the challenge. The paper suggests that there is not one single approach that fits in all circumstances in dealing with this challenge. It argues that there is always the need to adopt and explore the organisational context in our evaluation approaches. This implies an approach that combines the use of an appropriate evaluation system arrangement supported by evaluators’ experience, judgement and beliefs around the context and expectations of the evaluation.


Akua Arthur-Kissi, African Development Bank

Introduction

T

here is widespread recognition in the development community of the importance of independence in development evaluations. This emanates from the crucial role that evaluations play in strategic orientation of evidence-based policy and program decision making. Evaluation has become an essential component of good governance and has been a good support for reflection and periodic adaptations in the project, program and policy cycle (Organisation for Economic Cooperation and Development – Development Assistance Committee [OECD-DAC]). Mayne (2012) describes the principle of independence as a function in evaluation systems. In his opinion and similarly described by OECD, this is a principle that instils trust in the evaluation function as related to governance/oversight. To the DAC community, independence of evaluations is the degree of autonomy and separation from policy and operations. This allows evaluators to be able to report truthfully and without fear of repercussions or compromising to any demands of operations managers or entities being evaluated. There is a common argument about this practice (independence) that if the evaluation body is too remote from the everyday operations of the institution or concerned parties, then the evaluations may not be closely aligned with required information.

Following from the above, independent evaluations2 are considered part of sound management practices and they are reflected on three dimensions3: a) organisational independence where the evaluation function should be separated from those responsible for the design and implementation of the entities being evaluated. It also ensures that evaluation staff have full access to the information needed; b) behavioural independence, an expectation of ethics based conduct that seeks to prevent or manage the potential for conflict of interest; c) and functional independence requiring mechanisms in place to ensure freedom from influence in the planning, funding and reporting of evaluations. Literature indicates the need for a balance to be struck between independence and isolation. The OECD-DAC review report (2016) on evaluation systems cautions on the danger of isolation when seeking evaluation independence: a) lack of connection with operational units hence problems to access data and information, b) a reduced sense of ownership of the evaluation results specifically by management/ policy makers hence lower uptake; c) risk of being perceived as disloyal and untrustworthy or bringing in unwelcome criticisms considered as potentially damaging to uphold confidence. In practice, these issues consistently contend with good practice standards of stakeholder engagement that are important and usually required for ensuring credible and useful evaluations.


21


Experience shows that involving decision makers and stakeholders in the evaluation process enhances the prospects of the evaluation being more salient, legitimate and credible to ensure use and influence. In the same realm within the development evaluation profession and from diverse evaluation literature, stakeholder participation in evaluation processes is considered important to attaining sustainability, ownership and mutual accountability. However, a recent survey by OECD DAC 4 among its member networks showed that only a small number of agencies confirm that frequent or close stakeholder participation is encouraged during evaluation planning and design phases. While each stakeholder engagement effort will require a tailored approach based on the scope, scale and desired results, the expectation is that evaluation approaches should be consistent with the guiding principles of stakeholder engagement5. Morra & Rist (2009) affirm the importance of engaging stakeholders early enough in the evaluation process, from planning through to completion of the evaluation. In practice, it is not evident finding the right balance between remaining independent and engaging meaningfully with clients, management, program owners and others. One may ask how do we remain independent if we have to engage that much with stakeholders? Where do we place the boundary to keep the potential consequences at bay? Different literature and observed practices in multilateral development banks (MDBs), including the AfDB, point to a widespread recognition among the development community on the importance of stakeholder engagement in evaluation processes. At the AfDB, the approach to stakeholder engagement is embedded in the planning and review stages of all evaluations. The 22

experiences show the powerful role that the different stakeholders play in the success of an evaluation. This stems from clients in the regional member countries and staﬀ in the Bank who have interest in the subject of evaluation. Considering the criticality of these key principles and the effect they have on the ability of development evaluations to achieve usefulness and influence, this paper focuses on bringing to light the challenges that evaluators have to endure to ensure a good balance of applying these two principles in a mutually connected way. It also seeks to explore possible ways of approaching the challenges through an analysis of the different scenarios and the possible context where they manifest. It concludes by proposing possible options to explore by evaluators in the search for the appropriate approach to deal with the potential challenges.

The issue of staying independent in development evaluation As mentioned earlier, the principle of independence in evaluation has seen a growing prominence in both evaluation discussions and in literature. Although there is contention about its clarity and usage by some, it is mostly considered as a key principle to ensuring the credibility and quality of an evaluation. There is no clear definition of evaluation independence (Kaiser & Brass, 2010). A number of development evaluation agencies adopt the OECD guide, DAC Quality Standards for Development Evaluation (OECD, 2010) which considers independence as a key overarching principle. It defines independence as when “the evaluation process is transparent and independent from policy makers and program management”.



Figure 1: Framework of the tricky triangle by Bastiaan de Laat

era

t

mu n

me n TO

R,

Re

ss se

ce

an

y ilit

ta b

Evaluation

un

As

co

id Gu

Ac

tio

n

Evaluation Commissioner

Information

Enquiry

Evaluator

Evaluand

Source: Adapted from Loud & Mayne (2014) to illustrate the relationships among the three roles in the evaluation process.

Loud and Mayne 2014 agree on this principle and further allude that the credibility of an evaluation will mostly be at least dependent on the level of independence accorded to the evaluation function. This notion places evaluation close to the audit function and proposes to ensure independence in the implementation and management of evaluation processes. There is however the need for the right policy and practical guidelines for decision-making and follow up to be in place. In addition, experienced evaluators highlight the criticality of independence to ensuring evaluations credibility. For example, Robert Picciotto, former head of the World Bank's independent evaluation group (IEG) argues that independence remains an issue that affects an evaluation’s credibility despite all efforts made in an evaluation for ensuring better quality of work (2003). Equally, Scriven (1975) in his discussion on preventive measures to reduce risks of bias in evaluations proposed independence in

24

evaluations as one of the ways to reduce the tendency of bias. Experiences and literature show that credibility matters a lot to internal and external audiences of evaluations. It is particularly a key factor that affects the usefulness and use of evaluation recommendations (Penny Hawkins in Loud and Mayne 2014). For example, a lack of perceived independence is most often used as reason to dismiss or contend evaluation findings even when there could be other types of possible issues around information type or a strong opinion by an influential entity as reasons. It is obvious that strengthening independence goes along with the recognition of credible and useful evaluation work. The ability to sustain full independence without becoming too remote from realities of practice and policy remain a challenge here but experience shows that



this is to an extent achievable when the evaluation policy and manuals clearly guide the evaluation processes to attaining this as a goal.

How does stakeholder engagement matter in achieving credible evaluations? Engaging stakeholders early on in the evaluation process gives the evaluator a better understanding of the intervention or issue of evaluation, what it was intended for and the issues and challenges they encountered in the process. This helps the evaluation team to be better informed about the issues to be covered in the evaluation as well as specific information needed, when the information will be required and who can provide it. Engaging with key stakeholders helps to ensure that the evaluation does not miss any major crucial issue. Stakeholders´ engagement is expected to help increase their support of and interest in the evaluation which forms the potential for buy-in and effective use of the evaluation. Although there are many benefits claimed for participation, there is a growing cynicism amongst evaluators and stakeholders who feel let down that sometimes these claims yields suboptimal outcome in terms of efficiency (longer evaluation lifespan) and subsequently the usefulness of the evaluation. This also relates to the caution described by Loud and Mayne (2014) in the analysis of the “tricky triangle” concept by Bastiaan de Laat , where they illustrate the importance of the relationship and at the same time the dangers with the different configurations of the relationship with respect to independence, bias and control in evaluations. The question now is what constitutes a “good” balance between independence and stakeholder engagement to achieve the design and completion of credible

and useful evaluations? This is a question that incites a contentious uptake among the evaluation and development community and it has not reached consensus to date among evaluation bodies. From an evaluation user’s point of view, maintaining this balance depends on several factors. This includes the existence of an approach and set up that seeks to capture political and technical perspectives of the different stakeholders, and the ability to consider them objectively in the evaluation questions. Bryson et al 2010 provides interesting guidance on appropriate tools to achieve these. This similarly touches on the potential tension between the different roles in evaluation and the influence of their relationship configuration on evaluation use described by Bastiaan de Laat in the tricky triangle concept (figure 1). Understanding the concepts of evaluation actors’ relationship configurations In the tricky triangle concept by De Laat in Loud and Mayne (2014) three different roles exist in evaluation settings. They are defined as follows; a) the “evaluand” referred to here as the one being evaluated; b) the evaluator, the one performing the evaluation; c) the evaluation commissioner, the body or entity that calls for the evaluation and finances the evaluation service that is to be delivered. The analysis argues that maintaining a certain level of balance to achieve credibility and ability to minimise bias and control depends on the type of configuration and relationship between these three roles. From the discussions, what makes the triangle tricky is that there is an anticipation from the different actors with respect to the evaluation outcome which has to be considered for an effective evaluation. The scenarios illustrated in figure 1 point to the fact that the three roles (actors) involved in the evaluation process


25


do seek at any point to have a level of control or influence which relates to final judgements and outcome of the evaluation. This is critical to its credibility and usefulness. Previous mention of the dangers of the tricky triangle illustrate that maintaining an ideal set up or configuration does not necessarily work particularly in dealing with the challenges of potential influence of the different actors and its detrimental effect to sustaining credibility. Rather, having agreed policies and evaluation procedures that manage the relational factor is key to managing the risks. Understanding how these roles are configured in an institutional setting matters a lot when trying to plan for the ideal balance between independence and stakeholders’ involvement. No matter how neutral each actor may be there is the temptation to adapt the evaluation objective to one’s own agenda or specific expectation which could influence the evaluation questions or the approach. In the same way, the evaluand (one being evaluated) in his engagement may seek to push out certain topics from the evaluation or direct the focus to only the bright spots. A closer look at practical evaluation settings and set ups in some MDBs The usual practice in most MDBs, including AfDB, presents two modes of evaluation setups. There is an internal evaluation where all roles are within the same organisation (particularly undertaking self evaluations). There is also a hybrid configuration where the role of the evaluation commissioner and the evaluator coincide and the evaluand is separate through independent evaluations. The practice is a combination of the two modes. With the hybrid configuration, the evaluation unit may be the commissioner of external evaluations or studies but the evaluation unit’s portfolio

of programs is commissioned indirectly by the Board of Directors (BoD) who have to endorse the workplan. This endorsement makes the BoD the defacto commissioner of the evaluations. The endorsed workplan becomes a contract between the evaluation unit and the BoD to carry out the planned evaluations within a given time and resources. With the former mode, the practice is often within operations units of MDBs in undertaking self evaluations in the form of self-assessment notes, monitoring reports and project completion reports. Generally, there is no discrete evaluation commissioner but rather these self evaluations are directed by required internal or external organisational procedures thereby acting by default as commissioners and guidance for the implementation of the evaluation though there is a clear line of authority. Consequently this mode seems to place a stain on the validity and use of the evaluation depending on the extent to which evaluators are honest and objective in these self evaluations. It also relates to the perception of the use of self evaluations as being only formative or developmental by its nature (Patton, 2011) rather than for accountability as there is a percieved higher risk of bias and therefore often only used as data sources in broader independent evaluations as practiced in most MDBs. With the latter mode of practice, the three roles are within the same organisation but they are generally separated through a separate dedicated independent evaluation unit (evaluators) who only report to executive board members (the commissioner) and not to senior management who manages the financing and implementation of the organsation’s programs (the evaluand). The hybrid configuration often involves a combination of two modes in practice; both fully fledged internal evaluations, with the support of external expert evaluators or sometimes externalised evaluations through the evaluation unit who plays a supervisory role. Within the AfDB, both


27


are practiced but the experiences are characterised by issues around the tensions of the tricky triangle and its possible risks to independence and the evaluation outcome and use. Some literature criticises this mode of operation as having the potential to cause role pressures within the configuration and a high tendency towards positive bias (Mathison, 2005, Wenar 2006, Loud & Mayne 2014). It is however argued that these challenges could be addressed with the adoption and application of appropriate evaluation policies and measures that ensure credibility of evaluations and guarantees independence. The development evaluation community have done well to promote the adoption and institutionalisation of this practice among its members. The Evaluation Cooperation Group6 (ECG), proposes a set of “good practice standards” to guide members to establish independence of the internal evaluation units. This standard practice guideline maintains that the independence of evaluation units is not a given, but rather, it should be nurtured and protected regularly in all activities to achieve success.

Reaching the balance of independence and engaging actors for a useful evaluation Scriven (1975) reiterated that seeking to be objective and independent in evaluations often leads to the use of “external” evaluators. Though externality could be relative, nevertheless, the possibility for the external evaluator to give in to pressures from a client does not disappear if the chances of getting another contract is at stake. This is also affects independent judgment in evaluations (Mathison, 1999). In the same realm, building long-term relationships between evaluators and program managers is perceived potentially harmful either as it would lead to the evaluator increasingly identifying with the 28

organisation she or he works for, therewith, increasingly behaving in a “biased” way as an internal evaluator. Following from these points, it is evident that as much as evaluation practices encourage effective engagement of all actors, how the evaluation will be used and for which purpose should be the key yardstick in determining a framework for their collaboration in dealing with the relationships. This also serves to test the extent to which the policies guiding the evaluation practices in our institutions support efforts to mitigate the risks. Let us look at two typical instances in the AfDB's independent evaluation unit (IDEV) concerning such risks. IDEV practices are guided by its established policies and principles on its independence (in behaviour, set up and functions) which were approved by the Board of Directors. The first instance was an independent country program evaluation which had to be cancelled halfway through its implementation. This was a result of limited collaboration among key actors and other stakeholders on the purpose, timing and usefulness of the evaluation. In this case, the practice of independence over shadowed the required level of engagement among key actors particularly the evaluands and their relationship thereof. A contrary situation however was the independent evaluation of the operations procurement policy of the AfDB. This evaluation achieved the buy – in and effective use of outcome by the key stakeholders i.e. the Board of Directors, operations staff and procurement officers of the AfDB, as well as clients in the regional member countries. The evaluation received full cooperation and access to information following a good engagement process during and throughout the evaluation activities (from planning to completion). The engagement process was driven by the common agreement on the purpose and use of the evaluation by the key



actors. With a clear sense of objectivity for the evaluation, the potential diverse interest and influence of the key actors (i.e. the evaluation commissioner, the evaluators, the external evaluators and the evaluand) were considered and curbed to the evaluation focus and use to achieve independence in the planning and implementation process. An example of the actions was the fact that the evaluand was engaged to follow up and ensure that the key issues emerging from self-assessment reports were considered within the broader questions of the evaluation. These elements facilitated effective engagement and a common sense of purpose and use for the evaluation among the stakeholders. Basically, the underlying factors in both cases points to issues around the following but not limited to: a) key actors’ collaboration on the purpose and usefulness of the evaluation; b) management of the interest and influence of key actors by the evaluators in the planning and implementation process; and c) existence and application of appropriate policies and frameworks

guiding practices for independence in the institutional setting.

Managing extent of independence and engagement in evaluations In practice, knowing the importance of being inclusive, it is also similarly important not to be so at the expense of independence. By this, it implies the important trade-offs to be made. One has to be cautious of the likely danger of tension between all stakeholders’7 interest about certain information so that the scope remains relevant to the objective and purpose of evaluation use. There have not been any clear standards or guidance on how to manage this tension. However evaluation experiences point to enabling factors for consideration, some of which are identified and described in the AfDB experiences above. To conclude, the emphasis remains that in evaluation planning and implementation, recognizing the factors that have positive


29


effects on managing stakeholder interest and influence goes a long way to deal with the challenge. The concept of stakeholder management also calls on evaluators to be wary of the elements of different types of stakeholders’ power and interest so as to take appropriate actions to manage their weighting of influence towards the outcome. In all these, acceptable standard practices of evaluators' independence should not be overlooked.

A take away for evaluators or practitioners In summary, three morals emerge in the above discussions for consideration by evaluators or evaluation practitioners, as they seek to manage tensions and maintain a balance between the extent of independence and engagement of stakeholders for credible and useful evaluations:

a. Firstly, stakeholders have to be engaged to the extent possible throughout the evaluation process for credibility and use. However, an appropriate organisational setting for evaluations through its policies and strategies will be the catapult for the evaluator’s ability to preserve a balance of engagement and independence; b. Secondly, an understanding of the evaluation configuration and awareness of common pitfalls in applying good standard practices presents options to explore for effective independent evaluations. c. Finally, practising independence in evaluations is as important as managing relational issues and common purpose among key actors of the evaluations. Ultimately, the main focus should be on how credible and useful the final product will be to the users, not only commissioners but those whose lives may be affected by decisions informed by the evaluations.

Endnotes 1

Stakeholder engagement: Defined here as a management

3

practice aimed at capturing knowledge, increasing ownership of the project by users, reducing conflict, encouraging innovation and facilitating spin‐off partnerships among

Evaluation Cooperation Group (ECG), 2008. 4

involving them meaning fully to enhance inclusive deci-

5

forum, learn about each other’s values, reflect on these values and create a shared vision and shared objective. The practice is useful in increasing awareness, changing

Summarized in “Seven guiding principles of stakeholder engagement”. (Gautrey, 2003).

sion making, promoting equity. From a social perspective it is a process where diverse stakeholders share a common

Evaluation Systems in Development Cooperation: 2016 Review, OECD 2016.

actors. The practices involves identifying the key actors through an analysis of their interest and influence and

According to good practice standards of evaluation by

6

The peer group of MDB’s and MFI’s Evaluation Units.

7

All actors with interest and influence on the evaluation Including users of evaluation outcome.

attitudes and affecting behaviors (Bryson, 2004). 2

Independent evaluation: An evaluation carried out by people or entities free of the control of those responsible for the design and implementation of the development intervention. (OECD 2002, P.25).

30



References & bibliography AfDB (2005), Report on the independence of the department of

Mathison, S (1999) Rights, Responsibilities and duties: A

Operations Evaluation (IDEV).

comparison of ethics for internal and external evaluation

Bernstein, D.J., Whitsett, M. D.& Rakesh M. (2002), Addressing

(Special Issue). New Directions for Evaluation.

Sponsor and Stakeholder Needs in the Evaluation Authorizing

Mayne, J. (2012). Independence in Evaluation and the Role of

Environment: Trends and Implications (New Directors for Eval-

Culture. Evaluation cultures: Sense-making in complex times,

uation, No. 95, Fall 2002, Wiley Periodicals).

105-138.

Bryson, J.M. (2004). What to do when stakeholders matter: stake-

Morra – Imas ,L & Rist R. (2009), The Road to Results: Designing

holder identification and analysis techniques. Public manage-

and Conducting Effective Development Evaluations.

ment review, 6(1), pp.21-53.

OECD (1991), DAC Principles for the Evaluation of Develop-

Bryson J.M, Patton M.Q & Bowman R. A. (2010), working with evaluation stakeholders: A ration, step-wise approach & tool kit, Journal of Evaluation and Program Planning – Elsevier 2010. Evaluation Cooperation Group – ECG (2010). Good practice standards on independence of international financial institutions’ central evaluation departments. Gautrey, C. (2003), ‘Seven guiding principles of stakeholder engagement’, Blogs at ATD, blog post, viewed 15 March 2017, «https:// www.td.org/Publications/Blogs/Management-Blog/2013/06/ Seven-Guiding-Principles-Stakeholder-Engagement». Kaiser, F.M. & Brass, C. T. (2010). Independent evaluators of federal programs: Approaches, devices and examples. Washington D.C. Loud M. & Mayne J. (2014), Enhancing Evaluation Use, Insights

Author’s profile

from Internal Evaluation Units.

ment Assistance. OECD (2010), Quality Standards for Development Evaluation OECD (2016), Review of Evaluation Systems in development cooperation. Patton, M.Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, Guilford Press. Piciotto, R. (2003). International trends and development evaluation: The need for ideas. American Journal of Evaluation. Scriven, M (1975); Evaluation bias and its control. Berkeley: University of California. Wenar, L. (2006) Accountability International Development Aid. Ethics and International Affairs.

Akua Arthur-Kissi is an evaluation officer in the Independent Development Evaluation (IDEV) of the African Development Bank. She has been working on corporate, country program and thematic evaluations for the past 8 years. She co-task managed a recently completed country strategy and program evaluation of South Africa (2004–2015), and also takes credit for her contributions to previous influential evaluations by IDEV: The Paris Declaration Review (2010), Fragile States (2011–12) and Policy Based Operations evaluations (2010) of the Bank. Akua previously worked in sector and country operations departments and the knowledge management unit of the Bank for 8 years. Before joining the Bank, she worked momentarily with GIZ – Regional Aids Program for Africa in Ghana. Akua holds an MSC degree in Project and Programmes management from Warwick University, UK, and certification in monitoring, development evaluation and capacity building.


31

The paper reflects on the challenges that evaluators sometimes encounter when they conduct

Challenges in Conducting Development Evaluation: Dealing with perceptions

development evaluations. The context of the discussion is the author’s own development evaluation experience, particularly in Africa and Asia. The challenges that the project or program funders and implementers bring into the evaluation process include fear of evaluation, contextual detachment of projects/ programs, limited resource commitments to evaluation, language and cultural barriers as well as weak or misleading evaluation sensitization to project/program services recipients/ beneficiaries and other stakeholders. The major conclusions are that clients who are associated with these challenges have limited knowledge of the essence of evaluation and/or they do not plan and budget for evaluation. The key recommendation is that before any project or program is implemented, the implementers and administrators concerned should undergo training in monitoring and evaluation. This can also be a pre-requisite for anyone to receive funding for any development project/ program.


Pindai M. Sithole, Centre for Development in Research and Evaluation International Africa

Introduction

T

he discussion in this paper centers on some of the challenges which evaluators often face when they conduct development evaluations. The issues highlighted are largely based on the author’s own experiences in development evaluations in Africa and Asia. The analysis of the issues as well as the solutions that are presented were enriched through discussions that the author had held with fellow evaluators, on the subject, and through the documents reviewed. This is against the backdrop that evaluation is not a ritualistic exercise but a lifeblood of every development initiative. Thus, evaluation is an integral component of every development project/program as it enables one to systematically establish the worthiness, value-addition or transformation brought into people’s lives (Gertler et al. 2011, Lee 1999; Royse, Thyer & Padgett 2006 and Scriven 1991).

Challenges evaluators often experience Reflective analysis was used to conceptualize and synthesize the challenges that are experienced in evaluation as well as their solutions. The key challenges which are presented here focus on the behavioural issues in evaluation that are attributed to a client. In this context, a client refers to the individual or group of individuals who commissioned the evaluation. In

other words, this is an evaluation stakeholder whose project or program is being evaluated. The main challenges that are identified are: (i) Fear of evaluation; (ii) Detachment of project/program design from the context; (iii) Limited resource commitment to evaluation work; (iv) Culture and language in data collection; and (v) Sensitization and mobilisation of the evaluation and bias in the selection of participants. Fear of evaluation Observably, some project or program implementers as well as funding partners generally fear evaluation almost in the same way that some people fear financial audits. Three factors seem to explain this unfortunate fear: (a) the thought that the evaluation findings can turn out negative and can lead to loss of their employment; (b) human behaviour to naturally feel ashamed whenever one performs below the expected level; (c) the perception that evaluation is a highly technical process which very few people in an organization would understand. Together, these three explanations clearly illustrate a distorted understanding of the essence of evaluation. With this kind of evaluation distortion or dislocation, it is likely that project/ program administrators and implementers would be interested to see an evaluation report which highlights positive aspects about their project/program (Aucoin and Jarvis 2004). Clearly, the fear of evaluation described here indicates a lack of appreciation of the role of evaluation


33


in development on the part of the program administrators and implementers. In this situation, evaluation or evaluators are normally received with less enthusiasm and cooperation. To address this challenge, two interrelated solutions can be considered. Firstly, would be to have monitoring and evaluation exposure and skills training for everyone in the organization. The depth of the training content would vary with the levels of people in the organization. Those in the implementation units would obviously require more detailed training than those in the top management. Complementary to this, during the evaluation, evaluators should deliberately include debriefing meetings with the client at strategic points throughout the evaluation process up to the submission of draft report. From the author’s own experience, debriefings reduce or eliminate client’s fear as these become platforms for ongoing feedback and validation between evaluator and client. Detachment of project/program design from the context To date, the pattern of resources for development has been that the developed Western countries are the major funders of development initiatives in the developing regions of the world including Africa. This is one explanation for the “cultural rainbow” situations that are often found in project conceptualization, design, and implementation and reporting. If not carefully examined, the project design may suffer a misfit from diverse dimensions thereby reducing the chance to achieve transformative results (Lauring 2008). The misfit or contextual irrelevance is, for the most part, linked to the diverse cultural, religious, economic and political situations as well as the eco-political histories of countries across Africa. Wynn, Dutta and Nelson (2005) emphatically expresses this point when

they postulate that “Interventions that are effective in developed countries may not be effective in developing countries that have differing social, economic, cultural, and infrastructure factors that may affect how a project is implemented and the project’s outcomes”. Again, evaluation experience on community-based programs shows that funding partners and implementers sometimes do not value community entry protocols with indigenous and political leaders. This is evident in that the time allocated to the evaluator to collect data in the communities rarely factors in community protocols. In reality, the evaluator would have to carry out the required protocols anyway but the client may not be willing to adjust fieldwork time, even if some of the protocols are lengthy in nature. For instance, it could be logistically time consuming to get to the traditional leader’s homestead. The logistical challenges may arise from the bad state of roads, or no roads at all, and in that case the evaluator would then resort to travel on foot. Sometimes, the indigenous leaders would require the client to do the protocols or at least introduce the evaluator to them and confirm that indeed the named evaluator is sanctioned to collect data for project/ program evaluation in their jurisdictions (Sithole 2016). Honadle (1986: 17) argues alongside this pitfall in community-based evaluation and stresses that “the ability to manage people and resources must be indigenized if a project is to contribute to lasting success”. Chambers (1995) concurs with Honable when he asserts that effective project or program design, implementation and sustainability are quite possible if the people whose lives require the positive change are contextually understood and genuinely involved in the project. In fact, it is argued that projects/programs that have sound contextual fitness are more likely to succeed in achieving theory of change than those which are detached; a view also found in Matunhu (2011)


35


and Rodney (1972) in their separate landmark analyses of how European colonizers underdeveloped Africa through a systematic process which this author calls ‘suﬀocation of African culture’. A possible solution to the project/program contextual detachment is for the funding partners, administrators and implementers to conduct a comprehensive contextual analysis to inform design, implementation, monitoring & evaluation, learning as well as reporting. A number of techniques are available to do so which include PESTLE. The technique PESTLE stands for Political, Economic, Social, Technological, Legal and Environmental. In brief, the relevance in applying these six pillars of PESTLE in contextual analysis lies in that the analysis of: (i) political situation provides appreciation and understanding of the political dimensions of the country including sensitive issues and what is considered 'political correctness' for a development project/program to succeed; (ii) economic dimension allows one to gain appreciation of how the national economy is structured and functions as well as how that links to the political sphere of the country; (c) social aspects focuses on the normative socio-cultural frameworks in the country that either facilitate or impede positive development; (d) technological situation provides insights on the extent to which the various forms of technology in the country foster or limit development; (e) legal framework is vital to ensure that any development initiative is designed and implemented in line with the attendant laws and policies of the country; (f) environment means that one has to appreciate and understand the broad range of the geo-political parts of the country to include weather patterns, topography and environmental conservation issues and attendant policies. In short, it is important for funding partners and implementers alike to have a good grasp and appreciation of the indicators 36

across these six pillars in a country where a development project/program is to be implemented. Limited resource commitment to evaluation work Even though most funding partners of development initiatives value evaluation, they sometimes allocate limited resources to it especially money and time. Among the resources that evaluators sometimes find limited are the budget for evaluation as well as the time within which the evaluation has to be conducted. One of the reasons for inadequate funds allocated to the evaluation process emanates from the fact that some organizations do not have a proper plan for evaluation and learning. In this instance, evaluation tends to become an activity on which to spend the budget surplus and not a priority and deliberate undertaking to transform people’s lives. Such an attitude or institutional culture makes it difficult for evaluators because the level at which the evaluation at hand is valued differs between the evaluator and the client. On the aspect of time as a resource, at times the evaluator is given tight deadlines with unrealistic time allocated to the various tasks in the evaluation process. This usually happens when either the funding partners or the implementers commission an evaluation too late in the life of the project/program concerned. That is to say, evaluation is rushed to meet the timeline before the project/program is officially wound up as per the funding agreement. The late commissioning of an evaluation could be a result of delays encountered during the implementation process some of which could be beyond the control of the implementers. In addition, the delay in commission evaluation could be emanating from the need to buy time to complete pending project/program activities. Generally, Africa is a synchronous



culture where it is not uncommon for people to make time on unexpected events and adjust everything else accordingly. However, the perspective of the Western culture views time as a sequential phenomenon which means that little or no room is given to accommodate the unexpected events (Amponsah 2012). It is possible that when time adjustments are not considered in the event of unexpected circumstances, one way to cope with the situation might be to reduce the sample size of the evaluation and/or the number of data collection tools. Of course, reduction in the number of data collection tools compromises the power of triangulation and by implication, the validity of the evaluation data. The solution to the challenge above is envisaged in two ways. The first one is to ensure that every development project/ program has a proper plan for evaluation with a clearly defined budget. The second one is for funding partners and implementers alike to make concerted efforts to understand how Africans view time in development discourse as this will help in allocating time in the evaluation plan. Most importantly, to also understand how time is valued at various periods of the year in a given country or region in Africa. For example, in Zimbabwe or the Southern African region in general, people in the rural areas value their time for farming during the period October to April. In this scenario, community members are most likely to invest little time with the evaluator if data collection is conducted during this period as they will be busy in their fields. Culture and language in data collection In data collection, there are instances where the evaluator finds that the evaluation participants speak only in their native languages. On its own, this is not a problem but only becomes a concern if funding partners and/or implementers are not fluent in the native language or they

partially understand or write it. In such a situation, it is highly probable that the data collection tools would have been conceptualized and developed in a language foreign to the people from which the data will be obtained. The differential effect of the language and cultural influence of the evaluation design increases the probability of meaning loss during data collection due to translation as well as missing of cultural cues (Adeyemo 2013; Amponsah 2012; Bankov 2000; Harris 1988 and Imberti 2007). This is usually evident during focus

“In data collection, there are instances where the evaluator finds that the evaluation participants speak only in their native languages. On its own, this is not a problem but only becomes a concern if funding partners and/or implementers are not fluent in the native language”. group discussions and interviews and the scenario confirms the modern axiom that “The limits of our world are the limits of our language”; an aspect which has been extensively researched and documented by Mpofu-Hamadziripi et al. as documented in Mararike and Vengayi (2016). It can be seen that limited understanding due to language barrier in evaluation tends to compromise authenticity of data and leads to wrong conclusions and recommendations. Related to language challenges, some natives culturally cannot spend a long time (sometimes beyond an hour) with a stranger (the evaluator) as what normally happens in focus group discussions and interviews. In other words, long data collection tools tend to make people uncomfortable. The possible solution to the above is to ensure that evaluation team


37


composition includes some individuals who are fluent in the native/indigenous languages of the people from which data will be obtained. On the culture of short conversations, it is advisable to make data collection tools short especially the focus group discussion and interview guides if the cultural understanding suggests so.

“The key recommendation is that before any project or program is implemented, the implementers and administrators concerned should undergo training in results-based monitoring and evaluation”.

Sensitization of the evaluation and bias in the selection and mobilisation of participants

A suggested solution is twofold. On the evaluation stakeholder sensitization, the client should carry out an initial sensitization and in the process introduce the evaluator/evaluation team. The mobilisation of participants should, therefore, be guided by a sampling matrix carefully and jointly developed by the evaluator and the client. It should be noted that sensitization does not mean that the client tells the stakeholders to rehearse their answers to the possible questions the evaluator is likely to ask. When sensitization is done well, it tends to increase evaluator acceptance and promotes cooperation among the stakeholders throughout the evaluation process. Furthermore, when the client takes the responsibility to sensitize stakeholders, a misconception that evaluation is a “witch-hunting” exercise is minimized.

Conclusions and recommendations

The final challenge is that some clients think that to prepare and bring together stakeholders in the evaluation process is the responsibility of the evaluator. This is problematic because the evaluator would be limited in understanding the cultural dynamics, logistical details and institutional protocols required. On the selection and mobilization of evaluation participants, the challenge is when a client intentionally selects and mobilises evaluation participants for the evaluator in order to positively influence evaluation findings.

It can be concluded that clients who are associated with these challenges have either limited knowledge of the essence of evaluation or they do not plan and budget for it. The key recommendation is that before any project or program is implemented, the implementers and administrators concerned should undergo training in results-based monitoring and evaluation. This can also be a pre-requisite for anyone to receive funding or assistance for any development program or project.

References Amponsah, R. (2012). ICBE-RF Research Report No. 45/12:

in Public Management” Structure and Organization of Govern-

Investment Climate and Business Environment Research Fund

ment Research Committee, International Political Science Asso-

(ICBE-RF). Ghana Institute of Management and Public Adminis-

ciation. Dalhousie University/Canada School of Public Service.

tration (Gimpa) Accra, Ghana. www.trustafrica.org/icbe.

38

Bankov, K. (2000). Intellectual Effort and Linguistic Work: Semi-

Aucoin, P and Jarvis, M. (2004). Results-based Reporting:

otic and Hermeneutic Aspects of the Philosophy of Bergson.

Smart Practices for Improving Public Accountability. A Paper

Acta Semiotica Fennica IX, Helsingin yliopisto. International

Prepared for Conference: “Smart Practices toward Innovation

Semiotics Institute at Imatra.


eVALUation Matters First Quarter 2017 Harris, R. (1988). Language, Saussure and Wittgenstein: How to

Rodney, W. (1972). How Europe Underdeveloped Africa. Great

play games with words. London and New York. Routledge.

Britain. Bogle-L’Ouverture Publications.

Honadle, G. (1986). Development Management in Africa:

Scriven, M. (1991). Evaluation Thesaurus (4th edition). Newbury

Context and Strategy, a synthesis of lessons from six agricul-

Park: Sage Publications, Inc..

tural development projects. A.I.D Evaluation Special Study NO. 43. Development Alternatives Inc. US Agency for International Development.

Sithole, P.M (2014). Community-Based Development: A Study of Nhimbe in Zimbabwe. PhD thesis. Johannesburg: Witwatersrand University.

Imberti, P. (2007). Who resides behind the words? Exploring and understanding the language experience of the non-English speaking immigrant. Families in Society, 88(1), 67–73.

Sithole, P.M (2016). Use of indigenous knowledge systems in farming and the implication to the environment: A case study of Chimanimani District of Zimbabwe. Conference paper

Lauring, J. (2008). Rethinking social identity theory in inter-

presented at Chinhoyi University of Technology Conference in

national encounters: language use as a negotiated object

Zimbabwe held from 2 to 5 August 2016.

for identity making. International Journal of Cross Cultural Management, 8, 343.

Wynn, B. O; Dutta, A and Nelson, M. I. (2005). Challenges in Program Evaluation of Health Interventions in Developing Coun-

Mararike, C. G. (2016). Knowledge Production and African

tries. Centre for Domestic and International Health Security.

Universities: A struggle against social death. Harare: University of Zimbabwe Publications. Matunhu, J. (2011). A critique of Modernisation and Dependency Theories in Africa: Critical assessment. Africa Journal of

Author’s profile

History and Culture.

40

Dr. Pindai Sithole is a founding director of the Centre for Development in Research and Evaluation (CEDRE) International Africa based in Harare, Zimbabwe (www.cedreafrica.org) and co-founder and current President of the Zimbabwe Evaluation Society (www.zes.org). He is a development consultant and practitioner with 20 years of international experience in Results-Based Monitoring and Evaluation, Social Research, Results-Based Strategic Planning as well as teaching and training. His experience is drawn from public, civil society and private sectors in Africa, Asia and the USA. Pindai is also a part-time lecturer of graduate studies at Africa University in Zimbabwe in the Institute of Peace, Leadership and Governance (IPLG). He teaches Development Theory and Practice, Citizen Engagement, Peace and Conflict in Development, Public Policy Development, Governance and Results-based Leadership. He also supervises dissertations at Master’s degree and doctorate levels as well as mentoring undergraduate and graduate students in their research work. Pindai holds an Associate Degree in Accounting (Richland Community College, USA, 1993), BSC. in Finance with Management Information SyStems (Millikin University, USA, 1995), an MA. in Leadership & Management (Africa Leadership and Management Academy, Zimbabwe, 2007) and a PhD in Sustainable Development Studies with a special focus on Community-Based Development (Witwatersrand University, South Africa, 2014).


Rethinking Knowledge Management and Independence – Promoting the Use of Evaluation to Support Organizational Learning

This paper considers learning in the context of organizations, focussing on two distinct user groups: (1) senior managers and executive directors; and (2) operational staff. Whereas the AfDB’s current evaluation strategy does not clearly identify how different user groups are expected to consume and use evaluative information, the academic literature suggests that organizational learning is distinct from individual learning and is driven by a unique set of processes. In order to better leverage independent evaluation to support institutional learning, it is necessary to critically examine how organizations learn.


Erika Ismay MacLaughlin, African Development Bank

A

lthough development evaluation has traditionally been regarded as an accountability function, independent evaluation units are increasingly identifying “learning” as a major objective of their activities. For example, the (2013–2017) Independent Evaluation Strategy of the African Development Bank (AfDB) identifies three complementary objectives for independent evaluation: (1) learning; (2) accountability; and (3) promotion of an evaluation culture.1 However, beyond ensuring that evaluation reports are made available to potential users and that planned products are relevant to the needs of key stakeholders, the strategy does not clearly identify how learning is expected to take place among the different users of evaluative knowledge. If learning is to be among the core objectives of evaluation, it is necessary to not only identify the targeted user groups for evaluative information but also how they are expected to use this knowledge, and for what purpose.

Given that the AfDB’s independent evaluation function reports directly to the Board of Executive Directors, this paper considers learning in the context of organizations, focussing on two distinct user groups: (1) senior managers and executive directors; and (2) operational staff. Whereas the Bank’s current evaluation strategy does not clearly identify how different user groups are expected to consume and use evaluative information, the academic literature suggests

that organizational learning is distinct from individual learning and is driven by a unique set of processes. In order to better leverage independent evaluation to support institutional learning, it is necessary to critically examine how organizations learn.

How do organizations learn? Organizational learning is distinct from both individual learning and learning within groups. Crossan’s highly influential 4i model of organizational learning suggests that, within organizations, learning takes place at different levels, including among individuals, within groups and across the organization as a whole.2 Furthermore, at each of these levels, learning occurs through a distinct process. Individuals intuit relationships between events and the outcome of their work, reflecting a personal understanding about the factors which contribute to success or failure.3 The individual interprets this intuition in the context of his or her working environment and may discuss this interpretation with other individuals to reach a common understanding. Individuals or groups may then decide to integrate this new knowledge into their work. However, in order for learning to take place across the organization, the content of this learning must be institutionalized, meaning that new ideas are incorporated into standard practices and provide clear expectations for behaviour among individuals.4 The concept of institutionalization, a


43


process unique to organizations, illustrates that organizational learning not only requires that knowledge be shared beyond the individual, but that this knowledge must be used to influence behaviour on a broad scale.5 If we accept that evaluation can and should be used to support learning, Crossan’s model suggests that this learning will be expressed differently depending on the user group in question. An individual project manager may choose to apply a best practice to his or her own work, whereas an entire department may decide to adopt a new procedure in order to better manage risk. Finally, a senior management board may identify a new strategic direction for the entire organization which will impact the daily work of all its members.

“Through feedback processes, organizations monitor whether or not activities are achieving their expected results”. At all levels of an organization, learning is driven by the need to adapt to changes in the operating environment and ensure the achievement of results.6 The process of learning can also be multi-directional, such that the observations of individuals may eventually influence organizational policies or new organizational policies may change the way that individuals work. Through feedback processes, organizations monitor whether or not activities are achieving their expected results.7 A conclusion may then be reached about circumstances where activities must be modified in order to improve performance. However, organizational learning may also occur through feed-forward processes, whereby an organization changes its policies or practices to remain responsive to the operational environment and individuals must learn 44

to work differently in line with these new expectations.8 In a management context, feedback and feed-forward processes are implemented through feedback loops – systematic approaches to monitoring and analyzing the implementation of an organization’s activities and the achievement of expected results.9 These systems allow an organization to respond to cases where results have not been achieved and make adjustments to improve performance. A simple example of a feedback process is a thermostat. Thermostats measure the surrounding temperature of a room and activate heating or cooling systems as required to maintain a desired temperature.10 At their most simplistic level, feedback processes are most applicable to an assembly-line context, for which the consistency and quality of outputs are important operational goals. However, it is sometimes necessary to pose more fundamental questions about performance beyond identifying adjustments to existing activities. Returning to the example of an assembly line, a simple feedback loop may provide information about the extent to which a particular product is being produced efficiently, but a different type of inquiry is necessary to determine whether that product continues to be competitive.11 Whereas single feedback loops assess compliance with existing standards, double feedback loops question whether an organization’s policies and standards remain relevant and effective given the operating environment.12 Double feedback loop processes have the potential to alter an organization’s policies and expectations for behaviour among its individual members, thereby resulting in the institutionalization of knowledge. In the context of the development evaluation, the concept of single and double loop feedback mirrors the distinction between the self-evaluation and independent



evaluation. Furthermore, this distinction demonstrates the complementary value added by of these two tools with respect to learning. 13

The Self Evaluation Function and Organizational Learning As part of the self-evaluation function, project teams conduct evidence-based assessments to examine the performance of projects or programs and identify lessons to apply to future activities.14 The independent evaluation function implements a similar process; however, the design, management, implementation and reporting processes of the evaluation are carried out independently of project staff and management. By working outside of existing management structures, independent evaluation can consider a broader range of activities which cut across existing reporting lines and, as a result, is well-placed to consider an organization’s corporate policies and practices. In the context of a single development project, the self-evaluation function has high potential to contribute to individual and group learning due to a high level of individual familiarity with the context of a project as well as ownership of the evaluation process.15 Self-evaluation, conducted with appropriate rigour, may be better placed to identify the factors which contribute to success or failure for a specific project. Furthermore, it may be more likely that the project team, as owners of the evaluation process, will accept and adopt the recommendations.16 Conversely, the lessons from self-evaluation may only be relevant to specific circumstances. The extent to which these lessons lead to changes in behaviour is dependent upon the extent of their communication to like-minded peers as well as the extent to which these peers then have the opportunity and willingness to apply these lessons to their own work. 46

Recent reviews of the self-evaluation function at other development finance institutions (DFIs), including the International Monetary Fund (IMF) and World Bank, suggest that self-evaluation is often not leveraged to its full potential. An Independent Evaluation Office (IEO) review of the self-evaluation function at the IMF noted several challenges, including the lack of an organizational policy governing the timing, use, quality and content of self-evaluation reports.17 Furthermore, challenges were observed with respect to gaps in coverage and vulnerability of the self-evaluation function to budget restrictions.18 In its report on the self evaluation system of the World Bank Group, the Independent Evaluation Group (IEG) found that self-evaluation tends to emphasize accountability issues and performance ratings rather than opportunities for learning.19 Opportunities for genuine learning were further limited by a “compliance mindset” among staff as well as weak monitoring and optimistic reporting.20 Both of these reports identify limitations regarding the use of the self-evaluation function to support organizational learning. Whereas the IEO report found that self-evaluations had identified useful and relevant lessons for future operations, these lessons were often country-specific, with little communication of findings beyond the country in question.21 Furthermore, most interlocutors at the IMF felt the Board was not active in taking up lessons from the self-evaluation function and applying them to future decision-making.22 The IEG report similarly concluded that self-evaluation reports were not regularly mined for information outside of the independent evaluation function and that self-evaluation reports could be more valuable to inform implementation if there were incentives for management to acknowledge problems and raise “red flags”.



The Value of Independence for Organizational Learning Development evaluation, be it either self-evaluation or independent evaluation, has traditionally been used as an accountability tool.23 Evaluation naturally supports accountability by identifying: (i) what was achieved (i.e. what was implemented); and (ii) to what end (i.e. the achievement of results). Whereas the self evaluation function is capable of providing this information, it is not always well-positioned to do so strategically and independently. Ebrahim24 further argues that disproportionate emphasis on accountability may contribute to an “accountability myopia” wherein the need to justify activities and the use of resources overshadows the potential to examine the complex factors underlying performance, identify opportunities to innovate and learn from failure. One example is that of a non-profit institution which must provide an evaluation to their financial contributors in order secure additional resources for future projects. This context creates a series of perverse incentives which may limit the objectivity of the evaluation and also limit its value as a source of learning. In such an environment, implementers face pressure to: (i) design projects using conventional strategies, even if sub-optimal; (ii) focus on “low-hanging fruit” as opposed to more complex development problems; and (iii) under-report on project elements which did not work as anticipated. There is also a risk that project staff will turn monitoring into an administrative task rather than a management tool by seeking to report against a long list of compliance-related, output-level indicators. A review of results-based management within the United Nations development system found that pressure from donor countries to quantify the exact

contribution of agencies to development outcomes encourages the implementation of complex monitoring systems without enough consideration of the their relevance or value.25 These practices oversimplify the development process and encourage the identification of targets which are easily measured and achieved, but do not provide an indication of whether an intervention has contributed to a meaningful change in behaviour.26 The management literature provides similar examples which underscore the drawback of relying solely on self-evaluation and single-loop feedback. Argyris cites the example of a new product which is underperforming.27 Operational staff may understand that this product is not competitive and be cognizant of the serious challenges which remain before the product can be profitable. As this issue is communicated to different levels of management, however, there are increasing incentives to minimize the scale of the problem while overestimating the degree of “control over the situation”. As a result, operational staff and management face a double-bind due to the nature of their reporting relationships. Staff may face negative consequences for signaling that a process is not working but may also face negative consequences for failing to alert management to the problem.28 To be clear, the double-bind does not suggest that self-evaluation cannot be objective, but that there are often strong organizational incentives which influence its objectivity. By virtue of its impartiality, the independent evaluation function is able to avoid the downside of self-evaluation because it is not subject to the perverse incentives which discourage a frank discussion of challenges and failures. Aside from promoting accountability, part of the value of independent evaluation lies in “enhancing the credibility of


47


an organization”.29 The independent evaluation function is subject to the traditional reporting hierarchies. As such, the value of independence lies in the additional scope to challenge the status quo of an organization and identify challenges and failures without facing the pressures of the double-bind. For example, independent development evaluation is better placed to determine when a specific line

48

of programming which is not working as intended or identify an important contextual factor which has been consistently overlooked in the project appraisal process. Furthermore, independent evaluation can validate the results of self-evaluation30 and provide further justification to adopt a specific best practice. These two functions need not be duplicative or contradictory; they are complementary.



Creating Demand for Evaluative Knowledge – Promoting the use of Evaluation for Organizational Learning Whereas evaluation has an important role to play in organizational learning, the extent to which evaluative evidence is used to inform changes to organizational policies and practices has been difficult to identify concretely. Failure to leverage evaluative knowledge from both the self and independent evaluation functions is not only a missed opportunity, but raises concerns regarding the use of resources – one might question the utility of undertaking rigorous evaluations when a commitment to use this information in order to enhance the achievement of results is absent. A variety of tools have been implemented at the AfDB to promote the use of evaluation to support learning, including: (i) an Evaluation Results Database (EVRD), which provides both the Bank staff and the general public with access to lessons and recommendations from self and independent evaluations; (ii) publication of independent evaluation reports both in both hard and soft formats; (iii) launching of events and conferences to share lessons learned; and (iv) dissemination of evaluation reports and briefs via the IDEV website and electronic mailing lists. In addition to these learning tools, the AfDB has also developed of a tool to track the implementation of management actions and responses from evaluations (mars) and has instituted a formal requirement that lessons from past evaluations be identified and considered in the appraisal reports for new projects. However, these activities have not necessarily been effective in promoting the use of evaluation. The EVRD faced initial challenges regarding the quality of lessons from project completion reports which had to be validated before being included

in the database. Furthermore, although the database has been used by some external users, it has not been widely used by AfDB staff. This limited internal use suggests that the EVRD has not yet had a major influence on the design and implementation of the AfDB’s projects. Despite considerable communications initiatives, it is not possible to track the extent to which evaluative information is being used by targeted stakeholders or how they are using it, if at all. Finally, it is unclear how the MARS will contribute to assessing the impact of management actions on the Bank’s operations and performance. As established previously, organization learning is a deliberate activity through which new expectations for behaviour are codified within corporate policies and practices.31 Whereas learning can occur spontaneously among individuals, complex organizations lose the ability to learn spontaneously while remaining coherent. The need for deliberate action underscores the importance of demand for evaluative information and buy-in for the evaluative process. Deliberate use of evaluative knowledge can be seen through: (i) the incorporation of evaluative knowledge into the design of new policies, programs and projects; and (ii) changes to organizational policies and practices in response to evaluation recommendations. By contrast, the knowledge management activities tend to increase the supply of information without clearly addressing either the demand for the information or its ultimate use. Furthermore, formal requirements to use or consult evaluative information, including the requirement to identify lessons from evaluation in project appraisal reports, become symbolic rather than substantive if staff have no incentive to implement these mechanisms as they were originally intended. Expectations for the use of evaluative information must also consider an organization’s


49


operational context: In the ongoing push to increase approvals, it is unlikely and perhaps irrational that a proposed project not be approved at the highest levels simply because a review of evaluative evidence was less than thorough. The importance of demand for evaluative information to facilitate learning is well established in the available literature. John Mayne and other authors cite demand for evaluative information as a crucial component of an “evaluative culture” whereby an organization deliberately seeks out objective, evidence-based information about performance and uses this information to inform decision-making, adjust its practices and, ultimately, improve performance.32 The UNDP similarly emphasizes the importance of buy-in among senior management for results-based management, indicating that “senior management sets the tone and gives a general signal as to what is important and what is not”.33 Without such support, even formal requirements to review and cite evaluative knowledge are at risk of becoming a “check the box” exercise.

Use of Evaluation to inform Project Design – Independence, but not isolation With respect to use of evaluation to inform the design of projects and policies, Mayne envisions that senior management should serve as champions of evaluation and should be directly involved in challenging the logic and assumptions of projects as well as the evidence gathered on performance.34 In the context of a DFI, an independent evaluation group might provide senior management with regular and targeted briefings to support upcoming reviews of proposed projects. However, this approach underestimates the strong incentive to approve projects, expressed as an “approvals culture”. 50

The rejection of a proposal due to such scrutiny would undoubtedly encourage project teams to more thoroughly consider evidence from evaluation in the project appraisal process in order to avoid a similar outcome. However, the inefficiencies which would result from the rejection of a proposal after a lengthy appraisal process may ultimately reduce buy-in for the use of evaluative knowledge if these rejections are seen as too disruptive. There is a need for such scrutiny to take place earlier in the process of designing and approving projects. A better approach may be to encourage closer collaboration between operations and evaluation departments in the early stages of project design. A recent external review of the IEG notes that “the effectiveness and capacity of an evaluation unit to influence change requires strategic engagement and a close relationship with management and staff”.35 However, the need to maintain the independence of the evaluation function places limits on the degree to which these interactions can take place. Whereas consultation with senior management on the content of the evaluation work program is recognized as an appropriate means of ensuring that reports are relevant to the needs of the organization, there is concern that input from the evaluation function to inform the design of projects will create a conflict of interest when a project is evaluated. This concern has been overstated for two reasons: First, any role played by the evaluation function would be advisory. Accountability for project design choices would continue to rest with operations departments. This advice would be a means of achieving due diligence by ensuring that all relevant evidence is considered, without either dictating the details of the final design or guaranteeing the achievement of results. Secondly, independent development evaluation would



continue to be free of the double-bind facing operations staff. If an evaluation reveals that best practice at the time a project was designed failed to address one or more important factors for the achievement of results, there is no real incentive for the evaluators to either obscure this fact or provide an overly optimistic view of performance. Furthermore, independent development evaluators possess subject-specific knowledge which may benefit not only project design and implementation, but also self-evaluation. One common issue regarding the quality at entry of projects is the lack of a clear intervention logic and a realistic results measurement framework. Even without influencing the content of the project design itself, evaluators may have a role to play in ensuring that the project logic is coherent and that meaningful performance indicators are identified. Although there is already a process in place at the AfDB for ensuring that the design of each project meets minimum standards for quality at entry, the current process considers multiple elements of the project design aside from the monitoring of results and is similarly placed under considerable pressure by the “approval culture” of the Bank. There may be some value in receiving independent advice which exclusively targets project logic, performance measurement, evaluability and key lessons from the self and independent evaluation function. Moreover, greater use could also be made of formative and prospective evaluation processes to address issues of performance while a project or program is still being implemented. The use of prospective impact evaluation could be particularly useful in providing proof of concept for new lines of intervention. Such initiatives could identify relevant lessons and test our assumptions about how and why certain initiatives achieve their expected results.36 Furthermore, increased collaboration has

the potential to encourage buy-in and demand for evaluative information by enhancing ownership of evaluation among operations staff, while also leveraging the independence and expertise of the independent evaluation function.

Evaluating the Evaluators – Demonstrating the Credibility of Independent Evaluation The second means through which use of evaluation can contribute to organizational learning is where organizational policies or practices change as a result of evaluation recommendations. Although tools like the MARS can be used to assess self-reported implementation of recommendations, there are serious concerns about whether the implementation of evaluation recommendations result in an actual change in behaviour and whether new practices are actually enforced. Furthermore, the impact of evaluation recommendations is not always apparent. Although it is often assumed that the incorporation of lessons and recommendations into ongoing activities will improve the achievement of results, this assumption is rarely tested in an objective way. This lack of evidence raises the question of “who evaluates the evaluators?” One means of improving demand for evaluative knowledge would be to more clearly identify how the use of evaluation contributes to the achievement of results. Existing systems such as the MARS address this issue from accountability perspective by detailing the actions which have been taken. The system does not consider either the usefulness of the recommendations themselves or whether their implementation has had any tangible impact on performance. Furthermore, the MARS does not currently allow for analysis of persistent issues which are identified in


51


multiple recommendations over time. As the operational environment constantly changes, there is a need to continuously revisit recommendations and management actions to ensure that they continue to be relevant. The system could be better leveraged to identify and prioritize cross-cutting challenges that the AfDB is currently facing, and revisit the actions that management had implemented to address previously identified challenges, thereby promoting continuous organizational learning. As indicated by an external review of the IEG, the independent evaluation function “should develop a theory of change around users’ needs and the demand for its work”, and follow up on recommendations should entail, “not just a report of action plans, but also a discussion of outcomes”.37 In this regard, regular external reviews of the independent evaluation function would demonstrate that: (i) evaluation activities remain aligned to the needs of the organization; and (ii) evaluation reports provide relevant and credible guidance based on evidence and best practices. Most importantly, follow-up on evaluation recommendations may encourage more meaningful collaboration with management by clearly demonstrating how the use of evaluative knowledge has contributed to the achievement of results.

Conclusion By virtue of its independence from operational hierarchies, the independent evaluation function is in a unique position to ensure that policies and practices remain relevant to the operating environment and that they are promoting the achievement of results. Furthermore, independent evaluation can enhance the credibility of other evaluative functions within the Bank, most notably the self-evaluation function,

thereby encouraging the identification and adoption of best practices. Organizational learning is distinct in that it necessitates that behaviour is changed on a broad scale, thereby leveraging the knowledge of individuals, ensuring consistency and improving performance. Independent evaluation has the potential to contribute to organizational learning by influencing the design of new policies and projects and by changing existing practices and behaviours as a result of follow-up actions to evaluation recommendations. However, if this potential is to be leveraged, knowledge management practices must go beyond increasing the supply of information or tracking the implementation of evaluation recommendations for accountability purposes. These activities must be complemented by strategic initiatives to increase demand for evaluation among targeted user groups. Such initiatives should be rooted in a clear understanding of the needs of these user groups as well as how they will potentially use evaluative information. As a result, evaluation units may need to rethink what it means to be independent, collaborating more closely with operational staff and management to provide relevant inputs into the design and implementation of projects and programs. This approach may require the adoption of new tools, including formative evaluation and prospective impact evaluation. Finally, independent evaluation units should critically assess both the relevance and impact of the lessons and recommendations they produce. Such initiatives would promote the credibility of the independent evaluation function by demonstrating an understanding of the operational context. But most importantly, these initiatives could increase demand for evaluative knowledge, thereby promoting organizational learning and enhancing the achievement of results.


53


Endnotes 1

AfDB (African Development Bank Group) (2013) Independent Evaluation Strategy 2013–2017. Tunis: AfDB.

2

Crossan, M., and Berdrow, I. (2003) “Organizational Learning and Strategic Renewal”, Strategic Management Journal, 24, 1087–1108; see also Crossan, M., Lane, H. & White, R (1999) “An Organizational Learning Framework: From Intuition to Institution”, Academy of Management Review, 24 (3), 522–537).

3

Ibid.

4

Lengnick-Hall, C., Inocencio-Gray, J.L. (2013) “Institutional-

21 IEO (n. 14) 22 Ibid. 23 Ebrahim, A. (2005) “Accountability Myopia: Losing Sight of Organizational Learning”, Non-profit and Voluntary Sector Quarterly, 34(1), 56–87. 24 Ibid. 25 UNDP (United Nations Development Program) (2012)

Benefits and Liabilities of Prevailing Wisdom”, Journal of

“Results-Based Management in the United Nations

Leadership and Organizational Studies, 20(4), 420–435.

Development System: Progress and Challenges – A report

Ibid.

6

Crossan, M., and Berdrow, I. (2003) “Organizational Learning and Strategic Renewal”, Strategic Management Journal, 24, 1087–1108. See Crossan & Berdrow (n.ii); Crossan et al. (n.ii); Leng-

prepared for the United Nations Department of Economic and Social Affairs, for the Quadrennial Comprehensive Policy Review”. 26 Ibid. 27 Argyris (n. 9).

nick-Hall & Inocencio-Gray (n.iv).

28 Ibid.

8

Ibid.

29 IEO, 2015, citing Lamdany, R. & Edison, H. (2012) “Independ-

9

Argyris, C (1977) “Double Loop Learning Organizations”, Harvard Business Review (September/October), 115–125; Lengnick-Hall & Inocencio-Gray (n.4).

10 Ibid. 11 Ibid. 12 Ibid. 13 World Bank (2015) External Review of the Independent Evaluation Group of the World Bank Group: Report to CODE from the Independent Panel. (June) Washington, D.C.: World Bank. 14 IEO (Independent Evaluation Office of the International Monetary Fund) (2015) “Self-Evaluation at the IMF – An IEO Assessment” Washington, D.C.: IMF; see also IEG (Independent Evaluation Group) (2016) “Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group”, Washington, D.C.: World Bank. 15 IEO (n. 14). 16 Ibid. 17 Ibid. 18 Ibid.

54

20 Ibid.

ized Organizational Learning and Strategic Renewal: The

5

7

19 IEG (n. 14)

ent Evaluation at the IMF: The First Ten Years” 30 IEO (n. 14). 31 Lengnick-Hall & Inocencio-Gray (n. 4). 32 Mayne, J. (2008) “Building an Evaluative Culture for Effective Evaluation and Results Management”, Institutional Learning and Change (ILAC) Initiative Working Paper 8; see also Mayne, J. (2010) “Building an Evaluative Culture: The Key to Effective Evaluation and Results Management” The Canadian Journal of Program Evaluation, 24(2), 1–30; Nohria, N. and Beer, M. (2000) “Cracking the Code of Change”, Harvard Business Review, May-June 2000; Nyukorong, R. (2016) “The Strategic Building Blocks of a Learning Organization”, Journal of Resources Development and Management, 19. 33 Mayne (2010) (n. 32). 34 Mayne, (2008)(n.32) and Mayne (2010) (n. 32). 35 World Bank (n. 13). 36 Gertler, P.J, Martinez, S., Premard, P., Rawlings L.B. & Vermeersch, C.M.J. (2011) “Impact Evaluation in Practice”, The World Bank Group, Washington DC 20433. 37 World Bank (n. 13).



Author’s profile

Erika MacLaughlin has been a consultant and Evaluation Coordinator with IDEV since 2014. Before joining IDEV, she worked in research and evaluation with Foreign Affairs, Trade and Development Canada, the Canadian International Development Agency and the Public Service Commission of Canada. Erika received a BA (Honours) from the University of Western Ontario and a Juris Doctor from the University of Toronto.

Acknowledgement

The author would like to acknowledge the kind contribution of several colleagues from IDEV and other Development Finance Institutions, including Samer Hachem, Karen Rot-Munstermann, Marc Cohen, Souleymane Ben Daouda Dieye, Ruben Lamdany, Oscar Garcia and Keith Leonard.


55

Elaboration and running of a development evaluation policy

The development evaluation policy of a technical and financial partner (TFP) often has several objectives, which are sometimes difficult to reconcile because they require different organizational methods and evaluation methods for their achievement. The TFP will seek to resolve these problems through arbitrations and compromises that can evolve pragmatically over time, according to its priorities.


Bertrand Savoye, French Development Agency

M

a n y studies have addressed the methodological problems of evaluating development, from the information gathering stage to the interpretation of the results and the formulation of recommendations. In this article, we are interested in the often thorny questions that a technical and financial partner (TFP), in this case, the French Development Agency (AFD), is being confronted with while evaluating and implementing its evaluation policy. What are its expectations regarding evaluations in terms of learning and accountability and how can they be reconciled? What modalities of organization and methods of evaluation should be used to best meet its different expectations? In particular, we will focus on the discussion of scientific impact evaluations and the methodological issue of moving from the microeconomic to the macroeconomic scale. Finally, how can tasks and responsibilities be shared between countries and their technical and financial partners?

Expectations From theory to practice Let’s start by situating ourselves in an ideal evaluation world for a TFP. In this ideal world, the operational departments take into account the lessons learned from the evaluations and modify their intervention methods accordingly (learning Elaboration and running of a development evaluation policy

objective). The TFP informs the general public and its partners not only of the financed projects, but also of the results and even the development impacts that can be attributed to such funding (accountability objective). Finally, the undertaken scientific evaluation work or in-depth assessments contribute, according to different paths (Delarue, Naudet,Sauvat, 2009), to the improvement of the public policies of the partner countries and, more broadly, to the debate on the most relevant development policies (knowledge production objective). It is clear that at AFD and probably most TFP s, reality is quite far from this ideal. There are few examples of feedback to operations and of implementation of recommendations in evaluation reports. The accountability exercise to the general public2 is carried out mainly through monitoring of programs and projects implementation (i.e. the ability to inform and aggregate indicators of achievement and results) and not on the basis of retrospective evaluations. The conclusions reached by the scientific impact evaluations only have limited influence on the direction of development policies or TFP funding. For example, scientific impact evaluations financed by AFD highlight the mixed impacts of microfinance on poverty reduction. However, AFD will not give up on funding this sector. Even if it is assumed that scientific impact evaluations findings can be extrapolated to contexts other than those in which they were conducted, they are only 57


one of the many parameters for decision-making. Despite these gaps between theory and practice, the evaluation function plays a full part in the debate on sector strategies and the choice of financing instruments. But for this function to be alive, given the major risk of falling into the bureaucratic routine, compromises and trade-offs are necessary between objectives, which are often difficult to reconcile. Objectives that are difficult to reconcile The learning objective requires closer attention to operational service concerns and therefore: ❚❚

To target subjects presenting a real stake (hence at AFD a reasoned, systematic selection of the projects to be evaluated). For example, focus will be put on ex-post evaluations of projects belonging to sectors where AFD will continue to intervene, or ongoing projects, those that have not yet been evaluated or have not been recently. A policy of systematic evaluation of completed projects leads to certain evaluations which are no longer of great interest for operational services, either because they concern sectors in which AFD no longer wishes to intervene, or intervention modalities which are no longer applied.

❚❚

To closely associate these services with the conduct of evaluations in order to obtain judicious recommendations.

❚❚

To possibly use work that is partly internalized.

Pushed to the extreme, this logic can lead to self-assessment, although useful internally, but not credible in the context of external communication.

58

The accountability objective requires credible assessments and therefore to conform to different principles of neutrality and independence, hence the use of standardized and outsourced services and an unbiased selection of projects to be evaluated.3 Put to the extreme, this logic can lead to bureaucratic routine: one comes to evaluate just to evaluate, because this is part of the expected practices of a TFP or a financial institution, with the risk of seeing reports piling up, and their results not being really used, their recommendations being too general and disconnected from the operational teams’ concerns. Moreover, other lines of tension are added to this duality of objectives and to these modes of organization concerning: ❚❚

The level of interest: the risk of a routine process where evaluation would not be more interesting in itself than the respect of an ISO standard was mentioned. Conversely, if the interest in the subject is excessive, the exercise can be misled, with the temptation to use the expected results of evaluations as advocacy for the actions carried out, and the risk in this case of studying only “success stories” and of not evaluating projects of less satisfactory quality.

❚❚

The temporality of work: certain impacts (or sustainability of interventions) can only be correctly judged in the medium term (KfW development bank carries out some impact assessments 5 years after project completion, whereas at the AFD they take place in theory between 6 and 18 months). At the same time, however, the results may concern a generation of projects or methods of intervention that are already partly outdated and do not respond to operational concerns, such as whether to choose to continue a project and under what conditions.4



The TFP s have attempted, implicitly or explicitly, to seek ways of functioning which are certainly imperfect, but which make it possible to propose compromises between these different objectives. Multilateral institutions rely on internalized evaluations, but carried out by an independent structure within the Group (IEG for the World Bank Group, IDEV for the African Development Bank). In the absence of an independent structure within them, bilateral institutions use outsourced evaluations. At AFD, priorities have evolved over time. Until 2006, priority was given to learning. The evaluations were essentially internalized and the operational recommendations resulted in a formal validation process. From 2007 onwards, in order to respond to a growing demand for accountability and to harmonize practices with those of other


donors, priority was given to accountability, with the establishment of the systematic and outsourced evaluation framework of all completed projects of which AFD financing is traceable. Since 2013, the evaluation policy has again been changed in favor of learning (AFD, 2013). The use of outsourcing is maintained, but on the basis of a careful selection of the projects to be evaluated,5 however, covering the scope of all the interventions. In addition, a management response system for recommendations and follow-up of recommendations by operational departments is put in place. In the end, steering the evaluation function requires fine-tuned and not lasting adjustments between the objectives and requires the adoption of a rigorous and pragmatic posture. This observation also applies to the methodological choices and types of evaluation to be implemented.

59


What type of evaluation should be preferred?

❚❚

It is understood from the above that it is difficult for a single type of evaluation to achieve the different objectives sought, and on the contrary, it is necessary to mobilize a range of evaluations and methodological approaches.

The debate on scientific impact evaluations

Thus, at AFD, only two of the four objectives announced in the systematic and outsourced project evaluation system implemented since 2007 have been largely achieved. Namely the partnership dialogue – these evaluations are led by the local agencies in relation to the projects stakeholders; and the local capacity building in terms of evaluation – preferably conducted by local consulting firms. However, the accountability and learning objectives are not as fulfilled. The programming of the evaluations is done by selecting the most appropriate evaluation category for the objective (AFD, 2013): the scientific impact evaluation for knowledge production; the evaluation of projects clusters or the synthesis of project evaluations for learning, etc. While the undertaking of most types of evaluation leads to a relative consensus following the numerous methodological works carried out over the past years, particularly in the field of development by the OECD’s Development Assistance Committee, some methodological debates still play a leading role in the evaluation world. We are interested in two of them, namely: ❚❚

60

The choice to use rigorous quantitative approaches, such as those developed for scientific impact evaluations, an issue that is part of the wider controversy between qualitative and quantitative methods.

The choice between a microeconomic analysis and a macroeconomic analysis.

Beginning in the 2000s, the randomized scientific impact evaluation, or random controlled experimentation, was introduced in the areas of evaluation and development economics. It was presented as a scientifically sound approach with a rigor that largely superseded previous evaluations that had been unable to analyze and to attribute the effects of different types of interventions (Pamies-Sumner, 2014). This method has been widely used in academic work, and also in the evaluation standards of some TFPs (World Bank, USAID, DFID) or multilateral funds. Yet, while this approach does indeed make undeniable progress through its robust quantitative methods, it is often costly, complex to implement and constraining for the operational management of projects. Moreover, it deals better with certain sectors, in particular education or health, than with others. A new consensus seems to emerge today on a return to a greater plurality of methods and to favor mixed approaches of which the scientific impact evaluation is only the sub-component of a wider evaluation. This is particularly the case for the budget support evaluations implemented by different TFP s (EU, Dutch Cooperation, France, etc.). If one attempted a comparison with the police investigation work, looking for a series of clues through complementary investigations, the scientific impact evaluation could resemble DNA research. The latter provides irrefutable evidence of the perpetrator but does not say much or nothing about the motives for the crime, the circumstances that may be mitigating, Elaboration and running of a development evaluation policy


and so on. Similarly, the scientific impact evaluation requires at the very least to be complemented by qualitative analyses, or statistical analyses of framing. Thus, rather than focusing all the efforts on a single approach, it seems both more judicious and more challenging to adapt approaches to the covered themes and targets. In this sense, AFD seeks to rationalize the identification of the scientific impact evaluations to be carried out according to the following selection criteria: ❚❚

Strategic relevance: the evaluated intervention must be of significant importance for a strategic or interdisciplinary intervention framework.

❚❚

Scientific relevance: the evaluation must relate to an innovative intervention or be related to original and judicious questions from the point of view of the state of knowledge or other research in progress.

❚❚

Interest of partners.

❚❚

Feasibility: verifying the compatibility between the project’s and the evaluation’s time panel, the availability of data, the possibility of constituting a counterfactual, the ethical admissibility of the exercise, etc.

How do you go from the micro level to the macro level? Until the 1980s, macroeconomic evaluations were preferred, with the measurement of the multiplier effects of investments on the different branches of activity. Then, most of the time, microeconomic analyses and tools were used. However rigorous they may be, analyses that are exclusively at the microeconomic


level only provide partial answers to public policy questions. For example, it is not enough to know that the Senegal professional development program for businesses has a positive effect on the growth of the beneficiary firms, even if, from this point of view, its

“If one attempted a comparison with the police investigation work, looking for a series of clues through complementary investigations, the scientific impact evaluation could resemble DNA research”.

objective is reached. From a macroeconomic point of view, it is also necessary to determine whether a positive-sum or zero-sum game is observed globally, that is to say, if this effect does not occur to the expense of other local firms, without market share gain with respect to competition of imported products (Bastide, Savoye, 2015). Another example from a scientific impact evaluation carried out in France, is a reinforced support scheme for unemployed people set up by private placement operators to help them find lasting employment. The scheme has positive effects on the unemployed who benefit from it as they succeed in finding a job more quickly than the others (Crepon, Duflo, 2013). On the other hand, it has been shown that these effects are at the expense of those who do not benefit from this. In other words, there are movements in the line to find a job but no overall effect on the number of unemployed and the average duration of unemployment.

61


It therefore seems necessary to carry out investigations at the macroeconomic level where public policies are deployed and where the real cost/benefit of the measures adopted can be measured.

From the evaluation of projects by the TFPs to the evaluation of public policies by the countries In developing countries, particularly in Africa, although many ministries now have monitoring and evaluation services and sometimes engage in policy evaluation exercises, the TFPs take the initiative and still control a large part of the project or program evaluations. However, this device has certain disadvantages. First, while the level of an individual project may be of interest to the TFP for accountability and learning reasons, it is often less so for the country’s authorities, who focus on the broader level of a public policy (or one of its components).6 Secondly, evaluations focus mainly on the areas financed by the TFPs, at the risk of leaving large segments of public policies not fully covered, particularly in countries where the practice of public policy evaluations is still not well developed in the administration. However, the CLEAR report on nine African countries shows significant progress in this area in recent years (CLEAR, 2012).


Finally, there is some ambivalence in the evaluation of projects, as it relates both to the country’s strategies and operational practices and to those of the TFP. While most of them overlap, some may nevertheless stand out (for example, the TFP may aim to achieve its own outreach and influence objectives through the project). In line with the logic of mutual accountability advocated by the Busan Agreements, there should be increasing support for TFPs in public policy evaluation mechanisms led by national or local authorities, for accountability and learning purposes on the conduct of these policies. This system already prevails in a number of emerging countries, especially in Latin America (Mackay, 2007). However, this evolution implies that the TFPs are in a position to judge the reliability of the evaluation work carried out by the concerned authorities and can refer to the obtained results. Moreover, for TFPs, it must be supplemented by an evaluation of their sectoral or geographic strategies, rather than the individual projects which they finance. Finally, an evaluation of public policies based on quantitative approaches presupposes the ability to mobilize the statistical instruments which must themselves be able to produce the various indicators of public policy results throughout their implementation.

63


Endnotes 1

2

The analyses of this article are made under the responsibil-

to rely on the estimates of provisional accounts for year

ity of its author. They do not necessarily reflect the views of

n-1 or to wait for final accounts for year n-3? Is it necessary

the French Development Agency

to work for many months on the processing of survey data with the risk of publishing results too late, or to exploit data

For accountability to administrative supervison, the exer-

of imperfect quality?

cise is much more complete, with in particular the reporting of the evaluations carried out and the methods used. 3

5

interest, in particular the projects already evaluated in an

Either by exhaustive coverage or by random sampling of

earlier phase, a sector in which the AFD no longer inter-

the field to be evaluated 4

It can be noted that this dilemma between news and the quality of information also arises in the field of statistical production or that of the national accounts. Is it necessary

In order not to evaluate projects which would not be of

venes, etc. 6

The same is true in France. For example, local authorities will not evaluate the rehabilitation of a structure, but rather an entire urban policy displacement ...

Bibliography AFD (October 2013), “AFD’s evaluation policy”. Bastide N., Savoye B. (March 2015), “L’impact du programme de

Delarue J., Naudet JD., Sauvat V. (January 2009), “Are evaluations useful?”, Ex Methodological Post Notes n° 3, AFD.

mise à niveau des entreprises du Sénégal”, Papier de recherche

Mackay K (2007)., “Comment mettre en place des systèmes

2015–04, AFD.

de S&E pour améliorer les performances du secteur public”,

CLEAR (September 2012), “African monitoring and evaluation system, exploratory case studies”. Crepon B., Duflo E. et alii (April 2013), “Placement en emploi et

World Bank. Pamies-Sumner S. (2014), “Development Impact Evaluations”, A Savoir n° 27, June 2014, AFD.

Author’s profile

effets de déplacement”, La synthèse J-PAL.

64

Bertrand Savoye, is an economist working with the Research and Development Division of the French Development Agency (Agence française de développement, AFD). He formerly worked in the operational, training and retrospective evaluation services of AFD. Before joining the AFD, he worked at the National Institute of Statistics and Economics Research of France.


Evaluating for Development: Reshaping our evaluation criteria?

“… [G]rand overarching theses that encompass the whole of the development paradigm are unlikely to thrive, and development thinkers of tomorrow will need a new humility when faced with the growing complexity of the development terrain”. Curry-Alder et al, 2014. This paper argues that “development evaluation” does not attend sufficiently to “development’, especially considering the challenges facing the Global South. It is necessary to shift to a more dynamic approach that will enable us to evaluate FOR development. This, in turn, will compel us to make sure that the criteria that determine “what” and “how” we evaluate are refined in concept and in practice, cognisant of what defines an intervention that truly fosters development.


Zenda Ofir, Evaluation Specialist

Introduction

T

he development evaluation community is aware that our evaluation criteria, especially the ubiquitous OECD Development Assistance Committee (DAC) criteria, have to be updated. Thoughtful contributions have argued for refinements (e.g. Picciotto, 2014) or real shifts (e.g. Heider, 2017) in concept and practice. But what will shape them in future? On what basis do we decide which criteria to set aside, refine or add?

While “development” is now according to the 2030 Agenda the charge of all countries, the challenges remain the most severe among low-income countries. This paper argues that “development evaluation” does not attend sufficiently to “development’, especially considering the challenges facing the Global South. It is necessary to shift to a more dynamic approach that will enable us to evaluate FOR development. This, in turn, will compel us to make sure that the criteria that determine “what” and “how” we evaluate are refined in principle and in practice, cognisant of what defines an intervention1 that truly fosters development. The paper aims to stimulate debate on this issue by proposing several characteristics of “evaluation for development” (E4D) that should help us to reshape our evaluation criteria, and the way they are applied to Evaluating for Development: Reshaping our evaluation criteria?

interventions – while acknowledging that “interventions” are not the only focus for evaluation, but an important one. It also highlights the interesting similarity between the holistic approach to health interventions found in traditional oriental medicine, and the notion of evaluation for development viewed through a complex systems lens.

A new era for development evaluation? Over the past two decades the evaluation profession has become truly global, and development evaluation2 has been thriving. We have now entered an era where the challenges confronting evaluators working in a development context urgently demand new ways of thinking and working (Picciotto, 2015). Development blueprints, dominant ideologies and notions of “best practice” are being replaced by a colourful diversity of frameworks, models, goals and practices. At the same time, natural and man-made crises and disasters are destabilising large parts of the world, while countries and regions jostling for power are influencing development potential and strategies. The private sector is set to become a much more active and visible funder of development. And we are only now beginning to understand how development will be influenced by a hyper-connected world where the ambition of the 2030 Agenda with its Sustainable Development Goals (SDGs) is set to intersect with the Fourth Industrial 67


Revolution,3 and the digital economy with the “human economy”.4 The emphasis in the 2030 Agenda on the fact that development is the charge of all nations highlights the fact that development is a path, not a destination. This presents a good argument for merging “evaluation” in the Global North and “development evaluation” in the Global South.5 Yet development in Switzerland, Japan or Canada means something very different from development in Nepal, Yemen, Mali or Paraguay. Tweaking policies or sectors amidst robust institutions and systems while working from a strong foundation (based on development indicators) may be demanding. But it is clearly far less demanding than establishing a positive development trajectory from a weak foundation, over many fronts, for a prolonged period, while power, resource and capacity asymmetries interfere. This means that we can blend “development evaluation” completely with “evaluation” only if we are sure that our theories and practices truly enable us to serve those countries and societies that face major development challenges. After all, the foundational theories that shaped our profession originated in rich Western countries with little reference to the contexts that define t Re: Points for ammendment as per phonecall he Global South. We therefore have to make sure that our evaluation theories, practices, standards, criteria and questions help us to evaluate for development that genuinely improves the wellbeing of all our societies, and their ecosystems.

The problem: Development evaluation does not necessarily foster development The point of a development intervention is that it should contribute to development. Evaluating for Development: Reshaping our evaluation criteria?

But how often do we consider exactly what is meant by that? Have we been too laid-back in our engagement with the concept of development? Have we become overly complacent consumers of statements about development effectiveness and development evaluation? Development remains a contested concept; the differences between bottom-up/top-down, or micro/macro perspectives continue to lead to vastly different ideas and ideologies about how it can be achieved (Curry-Alder, 2014). As a result, it is very difficult for an evaluator to judge the merit of a specific development model during a certain phase in the evolution of a country or region. But we can be explicit about the ideology, values and model that we use when we evaluate the merit, worth or significance of a development intervention. This is displayed, in part, by the evaluation criteria that we choose to focus our interventions, and by how we apply them in practice. In doing so, we have to consider the following: ❚❚

Development progress is almost always considered and measured at country (i.e., societal, national) and regional level, as the proliferation of global indices and the reporting modalities for achievement of the SDGs illustrate.

❚❚

Definitions of development effectiveness confirm the need for collective, if not coordinated, action by a range of local and external actors. Development effectiveness is a measure of these actors’ aggregate impact, and accountability for results is shared by many.

❚❚ From a national perspective, we can get “development without development” (Chang, 2010). This happens for example when so-called development 69


interventions focus on enabling conditions such as poverty reduction, individual betterment or meeting basic needs without a vision of how the country can sustain a positive deve lopment trajectory in the long term. 70

❚❚

A country that shows development progress will need to have positive trajectories across multiple interconnected aspects of the life of its citizens. A narrowly focused intervention that does not at the right time unfold



in concert with the other interventions or goals, or that leads to localized or insignificant outcomes and impacts compared to what has to be achieved to ensure a positive development trajectory, cannot be assessed as a meaningful contribution to development. ❚❚

❚❚

This is especially problematic in what Chang calls “ersatz development” – whether supported by aid or philanthropic agencies, the private sector or a government – that relies on uncoordinated interventions that do not build on synergies between interventions, and do not enable system coherence that can facilitate change at a more macro level. If achieved outcomes or impacts do not sustain, or foster spontaneous action that leads to further outcomes and impact ripples, a so-called development intervention cannot be said to be successful. It can even lead to regression if people become disillusioned and disheartened as a result.

Thus, not every “development intervention” in a low-income country contri butes to development, even if it is part of a national or local development plan, meets “community” expectations, achieves desired outcomes or impacts, and/or shows a “silver bullet” effect. Or even if it successfully strengthens capacity, builds infrastructure, enhances access to health services, develops leadership or saves lives. Not only do we often fail to attend to the “higher levels” of theories of change that supposedly connect an intervention with a specific development context or model, but also our evaluation criteria do not necessarily compel us to attend in practice to all the critical issues that matter for development.


The solution: Evaluate for development Placing a focus on the notion of a more dynamic, holistic approach to evaluating for development might help us to refine our criteria. This approach to evaluation has as a main premise that we have to fully recognise the implications in practice of viewing development as a complex adaptive system (CAS) (Ramalingam, 2014). We then need to make sure that we embed this understanding in our evaluation criteria. From this perspective, evaluation for development (E4D) has at least five relevant characteristics6:

“Not only do we often fail to attend to the ‘higher levels’ of theories of change that supposedly connect an intervention with a specific development context or model, but also our evaluation criteria do not necessarily compel us to attend in practice to all the critical issues that matter for development”.

1. Development as CAS E4D views development through a complex adaptive “socio-ecological” systems lens. E4D acknowledges that development, and development interventions, are complex adaptive social-ecological 7 systems (Ramalingam, 2014; Orstrom, 2009). This means that evaluators have to grapple with the implications of dynamics and properties such as interdependence, non-linearity, coevolution, path-dependence, self-organisation and emergence that affect the relationships between 71


and within interventions, and the resulting patterns in society. We will need to sharpen our evaluation foci and criteria, bearing in mind that a complex adaptive systems approach has implications for evaluation practice far beyond evaluation-informed adaptive learning and management. 2. Attention to preconditions E4D shifts attention from overly simplistic theories of change and rigid notions of “results” to a stronger focus on precondi‑ tions for success. The current wave of “theories of change” based planning serves several important purposes. Yet they are often so simplistic compared to reality, and the underlying assumptions and results are sometimes so poorly conceptualised, that they become almost meaningless. Both development planners and evaluators need a stronger focus on identifying and understanding possible preconditions 8 for success. This is with a view to improving the likelihood that the positive development outcomes and impacts will be achieved, and sustained, within an evolving context. Preconditions can, to some extent, be identified through existing knowledge and insights (gleaned from the literature and experience), understanding the disposition of a society in a specific context, and evidence-informed learning.9

scales). For example, the path followed by an intervention to contribute to desired development impacts, which in turn relates to the development path of a society, country or region. An intervention might stall or fail before yielding results. It can suddenly reach a tipping point or fail due to a change in context that blocks an essential pathway. Thus, if evaluation is done at the wrong moment, the findings will not reflect the potential contribution of the intervention to development. This provides another key reason for not only tracking, but understanding progress and development trajectories through adaptive learning and management. 4. Positioning for development impact E4D focuses on assessing whether the intervention is well positioned for devel‑ opment impact. Under close scrutiny, the widely used criteria of relevance, effectiveness, and impact are not sufficient to foster a good understanding of how well an intervention has been, or is being positioned to enable it to contribute to development impact. In addition therefore, it is necessary to examine the intended and actual nature and scope of the intervention and of its outcomes and impacts. This will help determine the coverage and significance of the intervention in relation to a specific development context. Such analyses will focus on, for example:

3. Attention to trajectories ❚❚

The intervention design: How well are the intended outcomes and impacts connected to a given model of development? Targeting immediate, intermediate or root causes of a problem? Enabling or preventing change?

❚❚

Implementation: Initiated at the right time, given the society, country

E4D shifts attention from results as snapshots, to intervention and develop‑ ment trajectories. The current emphasis on adaptive management draws attention to the need to make assessments cognisant of trajectories (at different spatial and temporal 72



or region’s development priorities, needs and/or strategies? Signs of (potentially) catalytic actions that might, or have been shown to accelerate progress towards development impact? Of beneficial (or otherwise) reinforcing feedback loops, or of tipping points reached? ❚❚

Intended or actual changes: Incremental, developmental or transformative? Dealing with large societal or whole systems change, or modest directional, incremental nudges towards a certain result? Actions are causing, or have caused impact ripples beyond expectations? The size, scope and coverage of the (intended) changes, given what is required for development (For example, is “two percent improvement” enough to justify the intervention? Is there sufficient coverage of the population and its ecosystem? Have vulnerable or marginalised groups been sufficiently reached? Are resources adequate and appropriate to enable the enable the desired changes?).

5. Managing risk towards sustained (development) impact E4D focuses on managing the risk that desired outcomes and impacts will not be fully achieved, or might not sustain in support of development. A critical point is that if progress, impacts and development contributions are to be sustained, we have to bring much more nuance to how we evaluate “sustainability”. It is usually included as criteria in development evaluation, yet almost always only superficially addressed. It is inappropriate to judge an intervention as “successful” if the emerging or achieved positive outcomes and impacts do not sustain, or spontaneously allow other

74

desirable positive changes linked to development to emerge. This requires a much stronger focus on risk management to ensure the best chance of success, based on the dual strategy of (i) strengthening the potential positive (potential or emerging) development results, while (ii) attending to those factors that might prevent such results from being achieved or sustained. The following then become important evaluation foci: 1. Building strength through comple‑ mentarity: In low-income countries where development indicators are weak, development efforts have to proceed systematically, coherently and in synergy. In other words, activities in each intervention (and different interventions) should be executed in a certain order, in sequence or in parallel, building upon and enhancing one another for the most impact. In order to reflect an upward development trajectory in a society, this has to take place over decades, within and across ecosystems, integrating economic, sociocultural and environmental aspects. Evaluators therefore have to assess whether sufficient attention has been paid to the dynamics that result from the interdependence of interventions, actions, goals, etc. (Nilsson et al, 2016). This means attending to the extent to which an intervention or set of interventions has made use of the power of synergistic effects that make the whole more effective than the sum of the parts. 2. Neutralising any negative side effects. We cannot assess a development intervention as “successful” or having achieved “impact” unless we attend to unanticipated Evaluating for Development: Reshaping our evaluation criteria?


negative consequences, outcomes or impacts that might have significantly reduced or even nullified the positive outcomes and impacts (or that have potential to do so in future). | We have to understand whether enough has been done by the intervention designers and implementers to neutralise such negative side effects. We also have to search methodically for both positive and negative side effects, as well as for those influences that hinder or obstruct, or facilitate and enhance success, and use this information in our assessment. 3. Delivering and retaining impact: Making it “stick”. Implementation has to deliver development outcomes and impacts well, and in a manner that will help positive change to sustain, or foster other positive impacts. This justifies evaluative foci on effectiveness and efficiency – both well-known DAC evaluation criteria. It also justifies the increasing engagement with adaptive learning and management. This implies that we have to assess the extent to which adaptive learning and management have been properly applied in an intervention to ensure its adaptation where desirable, yet still within a system that is accountable for realistic, appropriate results. Furthermore, in order to ensure that positive changes “stick’, we also have to attend to what we can call – for lack of a better term – the legitimacy of an intervention. This will be determined by societal and stakeholder values and perspectives, for example considering whether stakeholder voices, gender or


vulnerable groups have been sufficiently attended to in the design and implementation of the intervention, or aspects of societal culture. | A critical aspect for consideration in E4D is the coevolution of societal culture and context (Figure 1). This shapes a society’s mental models and patterns of thinking and behaviour, i.e. its psyche, and its disposition and response towards a specific intervention (Ofir, 2016). In turn, an intervention can also affect the societal culture. This has significant implications for development, yet is seldom taken into account when we evaluate. If development impacts are to sustain, this aspect deserves to be at the centre of attention of the evaluation community.

Towards an E4D framework There are interesting similarities between the growing interest in a more holistic, complex systems based approach to development, and the growing interest in the potential of integrative medicine to address chronic disease.10 This latter trend has sparked renewed interest in traditional oriental medicine (TOM), which is a holistic, systems-based approach to health from the East that, in turn, has been based on an integrated view of a traditional system of governance. There is growing evidence that the complicated herbal mixtures and other treatments used in TOM might be more effective and suitable for managing health and preventing or addressing chronic disease than has been previously thought (Kim et al, 2015). The characteristics of a holistic “evaluation for development” approach

75


Figure 1: Elements of the coevolution of societal culture and context that shapes the psyche of a society and its patterns of behaviour

Expression in Symbols Rituals “Heroes” Memes & demes

Global power constellations Global relationship dynamics Global ecosystems change

Physical environment Economic comfort Political climate Social conditions

Shared values

Underlying assumptions & beliefs Norms Interpersonal relations Laws Institutions

Citizens’ cognition, ways of perceiving

Societal culture

Context

History & Time

Source: Ofir, 2016.

quite readily map onto key aspects of TOM. An effort has been made to show the similarities in diagrams in figures 2A and 2B. While concepts or frameworks from one discipline or practice should be applied with caution to another, there are reasons not to dismiss outright the apparent synergies between the two fields of work – the one dealing with an intervention in the health status of a person; the other, with an intervention in the development status of a societal group. TOM considers the human body as a miniature version of the larger universe, in line with a fractal understanding of life. It is fully aligned with the principles of systems

76

biology, which has emerged as central to the study of biology and medicine. Recent developments in mind-body medicine and epigenetics have also shown that our genetic make-up and biochemical reactions are greatly influenced by complex interactions within our bodies and minds, and with our behaviour as well as social and physical surrounds. Further work is ongoing to clarify the synergies and differences between the two systems, and to consider the implications for development and its application.



Figure 2A: Components of an emerging evaluation for development (E4D) framework for interventions

N D IT IO

TO EC

EC O

AJ

PR

Major intervention

Reducing side effects Neutralizing (negative consequences & influences)

Neutralising components

Complementary components

C

O

N

TE

XT

Delivery, Retention

RE N U IO LT IT U S C PO IS D

SUSTAINING DEVELOPMENT

ES RI

Strengthening Impact Coherence Synergy (synergistic effect)

POSITIONING FOR (DEVELOPMENT) IMPACT

TR

N S

Development impact Relevance Coverage Equity Effectiveness Significance

MANAGING RISK

Making it “stick’: Efficiency Legitimacy Sustainability Responsiveness (to culture and context)

Figure 2B: Elements of the holistic systems approach of traditional oriental medicine interventions

O TI U IT

Neutralising component for reducing side effects

N

IO

T SI

O

N D

P IS

D

Delivery, Retention Ambassador Shi

S

Neutralising Assistant/Official Zuo

Complementary Minister Chen

RR O U

Complementary component for enhancing efficacy of major herb

SU

ST C

O

N

Major King/President Jin

H Y LT OR EA T H JEC A

TR

N

Major herb conveying major drug efficacy

Delivery/retaining component for facilitating delivery of major herb to target site, & retaining it for prolonged availability in cells


77


Revisiting our evaluation criteria

and revise the criteria to bring renewal and nuance to what and how we evaluate.

In spite of the current dynamism in the profession, evaluation has to evolve quickly to become a more powerful force in support of sustainable development. We need innovation, in particular also by those who are embedded in the Global South and are faced daily by intense development challenges.

This paper suggests one approach to do so, highlighting the implications of E4D in our search for a refined set of evaluation criteria.11 Aspects for consideration are captured in figure 2a. Some of what should be done is likely going to be very hard to put into practice. But if we want to embark on Doing Evaluation Differently12, we will require the most powerful actors in the profession – the funders and commissioners of evaluation in the Global South and Global North – to consider how best to work in concert to help ensure that our profession truly serves development. Additionally, we will require that this is fully reflected in our evaluation criteria as well as in our evaluation theories, practices and standards.

Evaluation criteria, including the DAC criteria, have to reflect the values that we hold about what is important if evaluation is to foster development. How the criteria are conceptualised and applied can have a profound influence on our work, and on development. It is therefore important and urgent that we examine

Endnotes 1

2

Policies, strategies, portfolios, themes, programs, projects,

7

regularly interact in a resilient, sustained manner; is

and executed actions that result in perturbations within

defined at several spatial, temporal, and organisational

or across systems.

scales; a set of critical resources (natural, socioeconomic, and cultural) whose flow and use is regulated by a combi-

Defined in the OECD DAC Glossary as the “systematic and

nation of ecological and social systems; and a perpetually

objective assessment of an on-going or completed develop-

dynamic, complex system with continuous adaptation.

ment intervention, its design, implementation and results”. 3

It seems increasingly likely that the Fourth Industrial

8

and those that emerge as the intervention unfolds. They

and leadership, and disrupt social, technical and ethical

might include, but are not limited to, what is often called

aspects of development. Cloud computing, smart grids,

the “enabling environment”.

mobile web services and social media are blending with artificial intelligence, advanced robotics, and a fusion of technologies that blurs the lines between the physical, digital, and biological spheres. Power asymmetries will inevitably affect the “haves” and “have-nots”. These forces can facilitate, obstruct or destabilise development progress. A more holistic, less selfish conception of societies’ needs and interests, and addressing humanity as a whole within planetary boundaries. 5

It remains essential to make the (admittedly rough) distinction between the Global North and Global South. The differ-

A precondition is a necessary but not sufficient condition for change to happen. They include starting conditions

Revolution will reshape the nature of work, enterprises

4

A coherent system of biophysical and social factors that

products, events, processes, etc. – systematically planned

9

This is another rationale for adopting adaptive learning and management in development and evaluation.

10 Integrative medicine combines the reductionist approach of conventional medicine with the holistic approaches of “alternative” medicine for health management. 11 Potential adjustments to evaluation criteria are just briefly touched upon here, and will be expanded in a forthcoming paper. 12 In line with the “Doing Development Differently” initiative.

ences in development indicators between these two parts of the world remain too stark to ignore. 6

This is a preliminary list, aimed at stimulating further thought. It will be followed by a more detailed forthcoming publication.

78



References Chang, Ha-Joon (2009). Hamlet without the Prince of Denmark:

International Council for Science. Available at: http://www.icsu.

How development has disappeared from today’s “development”

org/publications/reports-and-reviews/working-paper-frame-

discourse. Available at: http://hajoonchang.net/wp-content/

work-for-understanding-sdg-interactions-2016/SDG-interac-

uploads/2012/01/HamletwithoutthePrinceofDenmark-revised.

tions-working-paper.pdf. [Accessed 12 Sept. 2016]

pdf [Accessed 15 Dec. 2016].

Ostrom, E. (2009). A General Framework for Analyzing Sustain-

Curry-Alder, B., Kanbur R., Malone D.M., Medhora R. (eds.). (2014). International Development. Ideas, Experience & Prospects. Oxford University Press, 943.

ability of Social-Ecological Systems. Science, 325, pp. 419–322. Ofir, Z. (2016). Evaluation and the nation state: where culture, context and development meet. Available at https://www.

Heider, C. (2017). Rethinking Evaluation – Have we had enough

europeanevaluation.org/sites/default/files/ees_newsletter/

of R/E/E/I/S? Blog post, January 10. Available at: https://

ees-newsletter-2016-09-september-r10-web.pdf

ieg.worldbankgroup.org/blog/rethinking-evaluation#comment-form. [Accessed 15 Feb. 2017].

Picciotto, R. (2015). The Fifth Wave. Social impact evaluation. Published by The Rockefeller Foundation. Available

approach to traditional oriental medicine. Nature Biotechnol-

uploads/20160628150404/The-5th-Wave-Social-Impact-Evalu-

ogy, 33, pp. 264–268.

ation.pdf [Accessed 10 Oct 2016].

Nilsson, M., Dave Griggs, D., Visbeck, M. and Ringler, C. (2016).

Ramalingam, P. (2013). Aid on the Edge of Chaos. Oxford Univer-

A draft framework for understanding SDG interactions.

sity Press. 440.

Author’s profile

Kim, H.U., Ryu J.Y., Lee J.O. and Lee S.Y. (2015). A systems

at:

https://assets.rockefellerfoundation.org/app/

Zenda Ofir is a South African evaluation specialist. With a PhD in Chemistry, she was Senior programme manager at a South African national science council and Director Research at University of Pretoria before turning to evaluation. Since 2000 she has worked from local to global level on evaluation assignments, focusing on Africa and Asia, and advised organisations such as GAVI, UNDP, CLEAR-AA, IFAD, AGRA and the Rockefeller Foundation. She is a former President of AfrEA, former Vice-President of IOCE, and former Board Member of AEA. She serves on the editorial boards of Evaluation and Program Planning and the African Evaluation Journal. She has been a visiting professor at the University of Hiroshima, has taught at the United Nations University in Tokyo, and has been designated Honorary Professor at Stellenbosch University, South Africa, an appointment that recognises professional eminence in a career outside the university.


79

1

New evaluations

The Burundi Country Strategy and Program Evaluation by IDEV evaluates the impact of US $739 million of AfDB support for development in Burundi during the 2004–2015 period. Evaluators manoeuvred within a difficult context marked by five years of political and economic turmoil. Rakesh Nangia, Evaluator General: “The context of fragility in Burundi presented numerous challenges for the evaluation team. IDEV is committed to presenting reliable and comprehensive reports on development progress and is mandated to deploy innovations necessary to achieve its goals”.

The Comprehensive Evaluation of the Development Results (CEDR) of the African Development Bank is a report of unprecedented scope, which measures the relevance, effectiveness, sustainability and efficiency of AfDB’s operations across the African continent over the past decade. It was published in November 2016 and presented at the ADF-14 replenishment meeting in Luxembourg.

Independent Development Evaluation African Development Bank

Impact Evaluation of the Rural Water Supply & Sanitation Program in Tanzania 2006–2015 Summary Report

September 2016

An IDEV Impact Evaluation

News in pictures

From experience to knowledge... From knowledge to action... From action to impact

Rakesh Nangia, Evaluator General: “Assessment of data to produce the CEDR exceeds that of any previous report. A rigorous quality assurance process was established to ensure robust conclusions and consistency of evaluations across countries and projects”.


South Africa: Evaluation of the Bank’s Country Strategy and Program 2004–2015 Summary Report – Redacted version

An IDEV Country Strategy Evaluation

Independent Development Evaluation

South Africa: Evaluation of the Bank’s Country Strategy and Program 2004–2015 Summary Report


Redacted version

January 2017

Rakesh Nangia, Evaluator General: “Across its work in finance and infrastructure, stakeholders in South Africa saw the AfDB as a financier, rather than adding value as a knowledge provider or supporter of their capacity. The AfDB must think carefully about its comparative advantage and innovatively about funding instruments for the future”.

Below: Joint monitoring visit to Medupi Coal Power Plant, South Africa in 2015.

Other new evaluations Independent Development Evaluation


Zambia: Evaluation of the Bank's Country Strategy and Program 2002–2015

Rural Water Supply and Sanitation Programme: Federal Republic of Ethiopia Main Report

Summary Report

October 2016

An IDEV Impact Evaluation

From experience to knowledge... From knowledge to action... From action to impact


Design & layout: CRÉON • www.creondesign.net

ability of projects. However, the evaluation points out that scale up private sector initiatives and mainstream gender

Zambia: Evaluation of the Bank’s Country Strategy and Program 2004–2013 – Summary Report

erformance of the African Development Bank’s assistance 015 period and its contribution to the development of the uding agriculture, governance, infrastructure development, pment, access to water supply and sanitation, promotion of cted for two purposes: (i) to inform the development of the for 2016–2020; and (ii) to support IDEV's Comprehensive opment Results. The assessment reveals, among others, ategy Papers have been well-aligned with both national nk's comparative advantage, also its portfolio has become tegrated approach to development challenges. Moreover,


Evaluation

bidjan 01, Côte d’Ivoire

What do we get for US $ 5 billion of investment? And is the Bank managing itself to maximize performance? The evaluation of the AfDB’s country strategy and program in South Africa aims to find out. The report covers more than a decade of the Bank’s engagement (2004–2015), principally in the finance and energy sectors.

October 2016

17/03/2017 17:18

News in pictures

81


2

New capacity building initiatives

The Twende Mbele M&E Partnership Programme was officially launched at the African Evaluation Association Conference in Kampala, Uganda on 28 March 2017. Twende Mbele, a Swahili term meaning “moving forward together” focuses on using peer learning amongst African countries to build M&E systems and strengthen government performance and accountability. IDEV is one of the partners of the program.

Above: IDEV Division Manager Karen Rot-Munstermann attended the ceremony and is pictured to the left of Twende Mbele Chairman Dr Ian Goldman, who is shaking hands with Ugandan Prime Minister Dr Ruhakana Rugunda.

http://idev.afdb.org/en/event/launchtwende-mbele-afrea-conference-2017

The Executive Committee of the African Parliamentarians’ Network on Development Evaluation (apnode) met in Abidjan on the 11th and 12th November 2016 to discuss the follow-up of the Network’s 2nd Annual General Meeting and its Strategic Plan. The members of the Evaluation Platform for Regional African Development Finance Institutions (EPRADI) also met at that time, and reaffirmed their commitment to harmonize evaluation methods and practices. From left to right (bottom): Ms. Karen Rot-Münstermann, Sen. Roger Mbassa Ndine (APNODE Chairperson), Hon. Dr. Susan Musyoka. From left to right (top): Hon. Jean Assou Apezouke, Hon. Evelyn Mpagi-Kaabule, Hon. Imbassou Ouattara Abbas and Sen. André Richard Moussounda Mikala.

82

Left: African Parliamentarians’ Network on Development Evaluation (APNODE) members sign the Declaration in support of the EvalAgenda 2020.

News in pictures


3

Knowledge sharing IDEV and the Norwegian Agency for Development Cooperation (Norad) hosted a seminar on Private Sector Assistance entitled Current Evidence on Additionality, on Monday, October 24 2016 in Oslo.

Above: IDEV Division Manager Rafika Amira, and Evaluator General Rakesh Nangia are among the panellists in Oslo.

Highlights of idev’s participation at the AfrEA Conference in Kampala IDEV hosted three knowledge sharing and evaluation capacity development sessions during the African Evaluation Association (AfrEA) International Conference, held in Kampala from 27th to 31st March 2017.

Thursday 30 March The key findings of IDEV’s Comprehensive Evaluation of the Development Results (cedr) of the AfDB Group were presented by Division Manager Samer Hachem. Friday 31 March IDEV presented its evaluation methodology for conducting very large-scale evaluations, on the premise that evaluations of the SDGs will be undertakings of similar scope and complexity. IDEV, as hosts of the African Parliamentarians’ Network on Development Evaluation (apnode) secretariat, led the discussion on how parliamentarians can be the change agents in implementing the SDGs.

News in pictures

The seminar, which drew on the findings of the synthesis report Towards Private Sector-Led Growth: Lessons of Experience, brought together over 80 participants including Norwegian policymakers, private sector, academia, civil society and the media, in addition to African diplomatic missions present in Oslo.

Above: Mr. Samer Hachem speaking at the 2017 AFREA conference.

83

Past issues

Fourth Quarter 2016 Evaluation Matters: Evaluation Week 2016


Fourth Quarter 2016: Evaluation Week Special Achieving transformation requires not only investments and policies, but also a change in mindset. With this in mind, we at IDEV organized the AfDB Development Evaluation Week 2016 on the theme of Driving Africa’s Transformation. This edition of Evaluation Matters captures the images, the debates and the words of wisdom from experts at the event.

Evaluation Week 2016

http://idev.afdb.org/en/document/evaluation-matters-fourth-quarter-2016-evaluation-week-special

Third Quarter 2016

The concept of value for money (VfM) in development work is the subject of much debate and confusion. In VfM analysis by development agencies, four key terms known as the 4 Es are often used. These are Economy – minimizing costs; Efficiency – getting more results for the costs; Effectiveness – successfully achieving the intended outcomes; and Equity – reaching different groups. How are these terms interpreted and applied by development practitioners? Are we getting value for money in development work – and that includes evaluation of development work?

Evaluation Matters: Value for money in development work

Third Quarter 2016: Value for Money in Development


Value for money in development work

http://idev.afdb.org/en/document/evaluation-matters-third-quarter-2016-value-money-development

Second Quarter 2016 Evaluation Matters: Evaluation of Private Sector Development Assistance: Trends, Challenges, and Opportunities


Second Quarter 2016: Evaluation of Private Sector Development Assistance This issue of eVALUation Matters examines the challenges and opportunities as well as the emerging trends of private sector development.

Evaluation of Private Sector Development Assistance: Trends, Challenges, and Opportunities

http://idev.afdb.org/en/document/evaluation-matters-second-quarter-2016-evaluation-private-sector-development-assistance

First Quarter 2016

AfDB commitments, whose content varies considerably, collectively affect all facets of the Bank’s work. Three independent evaluations were conducted to determine whether the AfDB is delivering on its commitments under GCI VI and ADF 12 and 13 Commitments. These evaluations informed the ADF-13 Mid-Term Review Meeting held in Abidjan, Côte d’Ivoire, from 11–13 November, 2015.Evaluation Matters zooms in on key findings, lessons and recommendations of these evaluations.

Evaluation Matters: The Evaluation Year in Review 2015

First Quarter 2016: Is the AfDB delivering on its commitments?


Is the AfDB delivering on its commitments? Independent evaluation provided some answers THE EVALUATION YEAR IN REVIEW

2015

http://idev.afdb.org/en/document/evaluation-matters-first-quarter-2016-afdb-delivering-its-commitments

84


idev.afdb.org

African Development Bank Group Avenue Joseph Anoma, 01 BP 1387, Abidjan 01, Côte d’Ivoire Phone: +225 20 26 20 41 E-mail: [email protected]