 | ||||
 |  |  |  |  |
 |  |  |  | |
|
| Evaluation - Evaluating Change | ||||
Introduction Introduction All conflict management interventions aim to make a difference. In order to judge the ‘value’ of conflict management projects and programs, evaluators therefore have to track and attribute change. Tracking change means: observing change; plotting its moving path; and searching for change by following evidence. In general, attributing change means explaining change by indicating cause. In terms of conflict management evaluation, it means determining a causal relationship between tracked changes and the project or program being evaluated. The nature of conflict management interventions and the nature of the conflict context makes tracking and attributing change especially challenging. Tracking Change: Goals and Objectives The evaluator normally begins the task of tracking change by looking at the stated goals and objectives of the project or program being evaluated. These tell the evaluator what the intervention aims to make different. The stated goals and objectives of interventions therefore direct the evaluator to where they should search for change. They also provide the evaluator with a yardstick against which they can measure change. However, the goals and objectives of conflict management interventions are often unclear or inflated. This relates to the scarcity of resources in the field of conflict management. In the bid for these scarce resources, middle-range and grass roots agents are likely to tell funders what they want to hear as regards the purpose and potential impact of proposed interventions. If funding is secured in this way, there may then be significant discrepancy between the stated goals and objectives of a program and what it is actually doing in practice. Further, if the evaluator uses these stated goals and objectives to direct them in their search for change they are likely to be unsuccessful. To avoid this, evaluators often try and guess the ‘real’ objectives of conflict management projects and programs. They may also disaggregate a goal such as ‘to prevent war’ into more manageable components that can be evaluated and that can act as useful yardsticks for measuring change. While such methods help evaluators to better track change, they rely heavily on individual evaluator’s discretion and judgement. This may ultimately undermine the validity of the evaluation. Indicators are essential tools for tracking change. Indicators are symptoms of change as well as clues and pointers towards it. They are pieces of information in the form of qualitative or quantitative statements that summarize the characteristics of systems or highlight what is happening in a system. An evaluator tracks change by selecting indicators and then examining the same indicators before, during and after the implementation of an intervention.
Funders are less interested in qualitative indicators because they cannot be counted but must be described, analysed and perceived. This can be done through, for example, interviews, role plays, art and drama. However, measuring qualitative indicators is therefore complex and demanding of stakeholders’ time and resources. Moreover, the qualitative impacts of conflict management interventions in, for example, relationships, attitudes and behaviours, rarely fit into the short evaluation time-frame preferred by funders. In a paper reviewing the state-of-the-art in evaluation by focusing on peace and conflict impact assessment (PCIA) methodology, Mark Hoffman argues that the field of conflict management needs to articulate indicators appropriate to its needs. However, standardising a set of indicators to for the practice of conflict management evaluation is controversial. In 1999, The Reflecting on Peace Practice Project was launched by the Collaborative for Development Action to pool the experiences of a number of international agencies working on conflict. The project has recently determined a number of criteria generic across a range of contexts that appear to make conflict management interventions more effective. Eventually, it may be possible to derive indicators from these criteria that are also generic across a range of conflict contexts. However, given the varied and volatile nature of conflicts, one should be careful not to write such indicators in stone.
Conflict management interventions may unintentionally make something different. The evaluator must therefore track both intended and unintended change. This task is important because unintended changes in the conflict context may have dramatic consequences. For example, a project or program may fulfil its goal by providing support to a community. However, this support may in effect privilege one group over another and thereby fuel the conflict. The field of conflict management has yet to develop tools capable of scanning for unintended change. Further, even if such tools were available evaluators would then have to somehow weigh the unintended changes brought about by the intervention against the intended changes. This could prove difficult. For example, what is the ‘value’ of a project that fails to fulfil its stated goals and objectives and bring about intended change; but does bring about positive and unintended change?
Tracking change also means coming to terms with the concept of ‘transfer.’ Herbert Kelman introduces the concept of transfer when he describes his work with interactive problem solving and explores the processes that can be used to evaluate such work. A working definition of ‘transfer’ for the purposes of conflict management evaluation is the ‘ripple’ or ‘multiplier’ effects of interventions beyond their immediate scope of action. Transfer therefore describes the widening and deepening of an intervention’s impact. For example, if an individual’s attitude changes as a result of their participation in a community relations programme, this change may ‘transfer’ to the individual’s family and the local community with whom the individual interacts. Importantly, transfer can be negative as well as positive; unintended as well as intended. The evaluator therefore faces the challenge of tracking the intended and unintended changes the project or program brings about directly as well as indirectly. The field has yet to develop a range of tools capable of doing so. It seems unlikely that standard social science data-collection methods will prove adequate to this task. Possible alternative methods include: interviewing local people who may not have been directly involved in the intervention, but who may have an impression of its effects; or hypothesising about the types of indicators likely to be seen if the intervention did have a multiplier or ripple effect, then looking to see whether these indicators arise in focus groups, open-ended interviews etc. Church and Shouldice argue in their recent study of conflict resolution evaluation that there is a broad agreement amongst stakeholders that evaluation is of greatest use when it takes place during the intervention’s life-cycle. An evaluation may, for instance, show certain aspects of a project or program to be working particularly well or badly. If presented with the results of the evaluation at an early enough stage, practitioners can adapt their strategies accordingly and improve the likelihood of the intervention achieving or superseding its goals. Evaluating during the course of a project or program facilitates improvement of the intervention’s performance before it is too late. For most conflict management projects and programs there are significant differences between their short, medium and long term impacts. Thus, in order to track all the changes that an intervention brings about, evaluation needs to occur more than once during the intervention’s life-cycle. Obviously, an evaluation carried out during the early implementation phase of an intervention can only judge the immediate short-term impact of an intervention. However, as the evaluation is carried out later in the intervention’s life-cycle, so it can become more exploratory and better determine the ‘ripple effects’ of an intervention beyond its immediate scope of action. It is therefore important to repeat evaluation at various points throughout a project or program’s life-cycle, as well as following the completion of implementation. In this way, all the changes generated by the intervention will be tracked by the evaluator.
It is possible to conclude that repeated evaluation throughout the life-cycle of an intervention is invaluable to the good practice of conflict management evaluation. However, because conflict management projects and programs operate in highly pressured environments it is difficult to ensure that adequate time and resources are devoted to evaluation. Indeed, the current reality in the field is that evaluations are carried out when funders demand it. Most often, this means evaluations are carried out only after projects and programs have been completed. However, there is evidence to suggest that this situation is slowly changing as funders’ interests in evaluation proliferate. For example, a number of funders now suggest in their grant application procedures that projects with plans for well-designed and repeated evaluation are more likely to receive support. A final aspect that needs to be taken into account when considering the timing of evaluation is local customs and contexts. The timing of an evaluation must be suited to each context in order to follow the principle of ‘do no harm’ as described by Mary Anderson in Do No Harm: How Aid Can Support Peace or War. For example, when carrying out an evaluation in a deeply traumatized society, an evaluator may upset participants in the project or program by asking them questions that bring back disturbing memories to the participants. Attributing change is crucial to the practice of conflict management evaluation. In general, attributing change means explaining change by indicating cause. In terms of conflict management evaluation, it means determining a causal relationship between tracked changes and the project or program being evaluated. The nature of conflict management interventions and the nature of the conflict context makes attributing change especially challenging. Attributing Change: Complexity Today’s conflicts involve many different actors - civilian and military, in both direct and indirect ways. They are also driven by many different forces ranging from the social to the political to the economic. Finally, there are a myriad of international responses to these conflicts ranging from humanitarian assistance to Track I diplomacy efforts. In the conflict context it is therefore difficult to know which actions have brought about which outcomes. An evaluator may, for example, be able to track changes in people’s attitudes through a survey. However, then attributing these changes to a youth exchange program in which respondents to the survey participated, is far more difficult. The concept of transfer and its implication that interventions bring about indirect as well as direct changes, further complicates the task of attributing change. The Collaborative for Development Action recently launched a Local Capacities for Peace Project. The aim of the project is to reach a better understanding of how the efforts and actions of humanitarian and development agencies interact with and, in some cases, reinforce conflicts. An important ‘lesson learned’ drawn from the Local Capacities for Peace Project is that people know. That is, local people in conflict zones frequently attribute changes to specific projects and programs operating in the field. They are also often able to justify what they ‘know’ citing evidence of cause and effect. Even if the impact of an intervention is visible only in people’s opinions and not in ‘reality,’ these opinions may subsequently shape behaviours and thereby become reality. The opinions of local people are an easily overlooked but important source of attribution available to evaluators.
|