Evaluation - Overview

Introduction
Agents
Functions
Tasks
Policy Recommendations

Introduction

Evaluation is the process of determining the merit, value and worth of projects and programs. How merit, value and worth are measured depends on who drives the evaluation and to what purpose. These web pages demystify the current practice of conflict management evaluation.

Until recently, evaluations of conflict management projects and programs were almost always carried out by funders. The purpose of these evaluations was generally to hold projects and programs accountable for their spending. However, during the past decade or so, funders have become more aware of the many different purposes evaluation can serve. At the same time, new actors have begun to initiate and drive evaluations of conflict management projects and programs.

The practice of conflict management evaluation is often described as being in a state of methodological anarchy. This is because there are few evaluation models, theories or even guidelines available specific to the field of conflict management. Evaluation models and theories therefore tend to be imported from other fields on an ad hoc basis. Unfortunately, these imported models and theories are often ill-suited to the nature of conflict management interventions and the conflict context.

This does not mean the field needs a single model or theory of conflict management evaluation. Rather, a range of evaluative tools need to be developed to cope with the challenges of evaluating conflict management interventions in the conflict context. With such a toolkit available both new and more refined approaches to the practice of conflict management evaluation will result.

Agents

Lederdach illustrates the make-up of a society as a three leveled triangle with: a small elite or leadership community at the top; a slightly larger community of middle-range leaders; and an even larger grass-roots community at the bottom. The practice of conflict management evaluation can be illustrated as another three-leveled triangle with a small elite community of those who fund conflict management interventions at the top; a slightly larger middle-range community of those who implement and administer conflict management interventions; and an even larger grass roots community of those targeted by interventions at the bottom.

The Elite Community

The elite actors who engage in conflict management evaluation are funders. This elite community includes donor institutions, private foundations, trusts, specialized agencies and NGO’s. For the purpose of these web pages, it is assumed that these actors do not directly administer or implement conflict management projects and programs.

  • Donor Institutions provide the most funding for conflict management interventions. International organizations and governments are the largest Donor Institutions.
  • Private foundations, trusts and specialized agencies also provide funding for conflict management projects and programs on a somewhat smaller scale.
  • NGO’s often act as intermediary funders. That is, they filter funding down from Donor Institutions, Private Foundations, Trusts and Specialized Agencies to other NGO’s or local groups that then implement and administer projects and programs.

As funders, these elite actors are most likely to initiate and drive ‘top-down’ evaluations. The purpose of these evaluations is generally to hold projects and programs accountable for their spending.

The Middle-Range Community

The middle-range actors who engage in evaluation are those who implement and administer projects and programs. This middle-range community includes NGO’s, local groups and the practitioners employed by these organizations.

  • NGO’s often implement and administer projects or programs directly. They may do so with funds provided by Donor Institutions, Private Foundations, Trusts, Specialized Agencies or even other NGO’s. NGO’s are most likely to participate in evaluation at the request of an elite actor who funds a project or program they administer. However, they may also initiate and drive ‘bottom-up’ evaluations. The purpose of these evaluations is generally to improve the performance of the project or program.
  • Local Groups receive funds from Donor Institutions, Private Foundations, Trusts, Specialized Agencies or NGO’s to carry out projects and programs. Like middle-range NGO’s, these groups are most likely to participate in evaluation at the request of an elite actor who funds the project or program they administer. However, they may also initiate and drive ‘bottom-up’ evaluations, most often to improve the performance of the project or program.
  • Practitioners are individuals employed to implement and administer conflict management projects and programs. They are a key source of information during the evaluation. Practitioners have a special interest in evaluation as a way of improving the performance of the project or program. Practitioners may therefore initiate and drive ‘bottom-up’ evaluations.

The Grass Roots Community

The grass roots actors who engage in evaluation are those targeted by conflict management projects and programs. During an evaluation, the evaluator interviews or otherwise engages these target groups in order to gauge the impact of the project or program being evaluated.

Functions

Evaluations of conflict management projects or programs may serve one or more of the following functions:

  1. Accountability. Evaluations are most often carried out to hold conflict management projects and programs accountable for their spending.
  2. Judgment. An evaluation may deliver judgment on the project or program’s overall performance. For example, an evaluation may deliver judgment on whether the intervention achieved its stated goals and objectives.
  3. Improved Performance of Projects and Programs. As an abundant source of ‘lessons learned’ and as a means to the end of ‘best practices’, evaluation improves the performance of conflict management projects and programs.
  4. Improved Performance of Funders. By evaluating the interventions they fund, Donor Institutions, Private Foundations, Trusts, Specialized Agencies and NGO’s can improve their own practice and analysis.
  5. Evaluative Capacity. A number of approaches to evaluation teach middle-range and grass roots actors to reflect on the performance of interventions and generate recommendations for change.
  6. Change Management. Evaluations often generate recommendations for change. Evaluations bring these recommendations to the attention of those capable of implementing change, for example, funders and practitioners.
  7. Responsibility. In the conflict context, evaluation matters in pragmatic terms because poor conflict management interventions cost lives. Evaluation also matters in ethical terms because it helps weed out poor interventions before they can exact such a cost. Evaluation is therefore a way for those working in the field to fulfil their sense of ethical responsibility to all the communities affected by the conflict.

Tasks

There are three tasks essential to the practice of conflict management evaluation:

  1. Tracking Change. All conflict management interventions aim to make something different. In order to judge the ‘value’ of conflict management projects and programs, evaluators therefore have to track change. They do so by: observing change; plotting its moving path; and searching for change by following evidence.
  2. Attributing Change. Attributing change is crucial to the practice of conflict management evaluation. In general, attributing change means explaining change by indicating cause. In terms of conflict management evaluation, it means determining a causal relationship between tracked changes and the project or program being evaluated.
  3. Engaging Stakeholders. The honest engagement of actors at all three levels of Lederdach’s society is essential to the effective practice of conflict management evaluation. This facilitates the communication of information vertically and horizontally between and within groups or communities.
    The nature of conflict management interventions and the nature of the conflict context makes completing these three tasks difficult.

Policy Recommendations

For ease of reference, the various policy recommendations made throughout these web pages are listed below:

Evaluators of Conflict Management Interventions

  • A code of ethics should be developed for evaluators
  • Training should be provided for evaluators.
  • Evaluators should have greater input into the terms of reference set for evaluations by funders.
  • Evaluators hired by funders need to be given the freedom to deliver honest evaluations. Funders should reassure evaluators that the findings of the evaluation will not endanger the evaluator’s professional livelihood.
  • External evaluators should be given the necessary time and resources to overcome obstacles presented by their ‘outsider’ status.
  • Greater use should be made of internal evaluators.

Tracking and Attributing Change

  • Conflict management interventions need to set clear and appropriate goals and objectives. Funders should encourage this by encouraging openness and honesty in their grant application procedures.
  • Evaluators should be given the time and resources to develop tools capable of scanning for qualitative as well as quantitative; positive as well as negative; unintended as well as intended; direct as well as indirect; and macro-level as well as micro-level impacts.
  • Local people should be used as a source for tracking and attributing change

Timing

  • Evaluation should be repeated at various points throughout the life-cycle of an intervention and following completion of its implementation.
  • Evaluation should be integrated into the development and implementation of an intervention

Indicators

  • Evaluators, middle-range and grass roots actors should begin to articulate indicators appropriate to the evaluation of conflict management interventions.

Engaging Stakeholders

  • Funders’ grant application procedures should encourage projects to plan for well-designed and repeated evaluation.
  • Funders need to provide evaluators, middle-range and grass roots actors with the freedom to acknowledge failure. Funders should recognise the concept of ‘good enough’ rather than the absolute concepts of success and failure.
  • The interests of all communities should be taken into account when setting the terms of reference for an evaluation.
  • Funder-driven evaluations should focus less on accountability and more on learning, managing change and improving the performance of interventions.
  • Evaluations should become more 'participatory’ and include stakeholders from all communities. Special attention should be devoted to building the evaluative capacity of middle-range and grass roots communities.

Ownership of Evaluation

  • Evaluations should be shared vertically and horizontally between and within groups or communities.

Ethical Dimension

  • Consideration must be given to the impact of evaluation on those living in conflict situations during evaluation design, the evaluation process and when responding to the findings of an evaluation.
  • The integrity of evaluations should not be subject to the desire for some good PR.