 | ||||
 |  |  |  |  |
 |  |  |  | |
|
Evaluation - Approaches | |||||
Introduction There are a number of innovative approaches to the evaluation of conflict management projects and programs. This section examines five of these approaches: Participatory Evaluation; Utilisation Focused Evaluation; Impact Evaluation; Action Evaluation and Macro-evaluation. In different ways each of these approaches takes special account of the conflict context and/or the nature of conflict management interventions. Participatory Evaluation overcomes some of the difficulties involved in evaluating conflict management interventions in the conflict context. In particular, Participatory Evaluation engages a broad range of stakeholders in the evaluation and facilitates the management of change by these stakeholders. However, in contrast to more conventional one-off ‘snap-shot’ evaluations, Participatory Evaluation is extremely demanding of stakeholders’ time and resources. Role of the Evaluator Unlike more conventional approaches to evaluation, Participatory Evaluation does not depend on the role of an ‘objective’ external evaluator. The role of the evaluator in Participatory Evaluation is that of a facilitator, who supports the engagement of middle-range and grass roots stakeholders in the evaluation. Evaluators are therefore often ‘internal’ rather than ‘external’. For example, in a 1997 Participatory Evaluation of 15 Humanitarian Fund Projects in Haiti, 28 participants were drawn directly from the projects and trained to act as Participatory Evaluation facilitators. This helped promote the honest engagement of middle-range and grass roots communities in the evaluation process. Management of Change The guiding principle of Participatory Evaluation is to include as many stakeholders in the practice of evaluation as possible. It is therefore one in a series of ‘bottom-up’ or ‘people-centred’ approaches to evaluation that have recently been developed. These include Empowerment Evaluation, Self-evaluation and Internal Evaluation. A common feature of these approaches is that the evaluations are process oriented. That is, how change occurs is considered more important than the results of that change. Participatory Evaluation has therefore become especially popular in the field of development as interventions have moved away from ‘blueprint projects’ towards more flexibly planned projects designed to evolve and adapt to local contexts. The process oriented nature of Participatory Evaluation shifts the focus of the evaluation away from tracking and attributing change to facilitating the management of change by middle-range and grass roots stakeholders.
Engaging Stakeholders In order to ensure that middle-range and grass roots actors ‘buy in’ to Participatory Evaluation, the evaluation is designed to provide useful feedback and recommendations to the intervention being evaluated. The evaluative capacity of middle-range and grass roots communities is also built up during their participation in the evaluation process. For instance, stakeholders learn to collect and analyse data, as well as to generate recommendations for change. This develops middle-range and grass roots communities’ sense of ownership over the evaluation. Moreover, by including middle-range and grass roots actors in the evaluation process, Participatory Evaluation provides a range of perspectives for reflection and learning. Participatory Evaluation therefore promotes collaborative problem solving.
Utilisation Focused Evaluation Similarly to Participatory Evaluation, Utilisation Focused Evaluation promotes the honest engagement of all stakeholders in the practice of evaluation. However, while Participatory Evaluation achieves this by including as many stakeholders in the evaluation as possible; Utilisation Focused Evaluation achieves this by ensuring the utility of evaluations. Process The utility of an evaluation concerns both how people experience the evaluation and how people apply its findings. The focus of a Utilisation Focused Evaluation is therefore on intended use by intended users. Michael Quinn Patton has written extensively about this focus and the practice of Utilisation Focused Evaluation more generally in his book Utilization-Focused Evaluation. He has also delivered an important series of Utilisation Focused Evaluation Training lectures. In these Evaluation Training Lectures, Quinn Patton notes that the first step of a Utilisation Focused Evaluation involves identifying a group of intended users for the evaluation. Normally these intended users are those who are capable of changing the intervention to some degree. These intended users then determine the intended use or uses for the evaluation. These intended uses could include, for example, influencing Donor Institutions, improving the performance of the intervention and generating knowledge. As Michael Quinn Patton suggests, a psychology of use is crucial to the success of Utilisation Focused Evaluation. That is, it is only through active involvement in the design and process of an evaluation, that the intended users of an evaluation will develop the necessary willingness to implement the evaluation’s findings and recommendations. Importantly therefore, Utilisation Focused Evaluation does not advocate any particular model, method or use. Role of the Evaluator The mandate of the evaluator in a Utilisation Focused Evaluation is to be "active-reactive-adapative." The evaluator needs to be active in terms of focusing the evaluation on use; reactive in terms of taking the interests of the intended users into account; and adaptive in terms of reconciling the interests of the intended users with the evaluator’s interest in ensuring the utility of the evaluation. Utilisation Focused Evaluation is therefore best described as a decision making process played out between the evaluator and the intended users of an evaluation. At every point in this decision making process, it is the responsibility of the evaluator to ask the intended users, ‘how would that affect your use of the evaluation?’ Building Evaluative Capacity By focusing the attention of intended users on the intended use or uses of an evaluation, Utilisation Focused Evaluation builds the capacity of stakeholders to think and act evaluatively. This ‘building capacity’ aspect of both Participatory Evaluation and Utilisation Focused Evaluation is important in justifying the cost of evaluation itself and in encouraging middle-range and grass roots stakeholders to devote their time and resources to evaluation. Impact Evaluation is a relatively new approach to evaluation that, as its name suggests, aims to assess the impact of an intervention. Simple enough in theory, however, the nature of conflict management interventions and the nature of the conflict context makes assessing impact difficult. Defining Impact In their recent study of conflict resolution evaluation, Church and Shouldice differentiate between the outputs, outcomes and impacts of projects and programs. They argue that conventional approaches to evaluation are relatively successful in assessing the outputs of evaluations, that is, the immediate tangible results of an intervention necessary to achieving the intervention’s objective. They are also relatively successful in assessing the outcomes of evaluations, that is, the short-term results of interventions, partially generated by the intervention’s outputs. For example, an output of a Track II Diplomacy intervention might be the number of leaders who attended a meeting; an outcome of the same intervention might be a set of recommendations endorsed by all participants in that meeting. However, conventional approaches to evaluation are less successful in assessing impacts, that is the overall or longer-term effects and consequences of interventions. Impact Evaluation focuses on this task and therefore faces all the challenges of tracking and attributing change in the conflict context. Tools Impact Evaluation uses a number of tools in order to assess the impact of conflict management interventions in the conflict context. Conflict management evaluators are often faced with a lack of reference or baseline data to compare with data collected during the evaluation. Therefore, as DANIDA’s (Danish International Development Assistance) Evaluation Guidelines suggest, the evaluator must often judge the impact of an intervention on the basis of general experience. While these judgements may be accurate, this method of evaluation is sullied by its lack of validity. Hence, Impact Evaluation makes use of triangulation as a research method. DANIDA’s Evaluation Guidelines describes the application of this research method to the practice of evaluation in detail. Here, it is enough to note that triangulation involves using several information sources and methods simultaneously in order to generate knowledge about the same topics. For example, a triangulation effect would result from supplementing information drawn from a survey, with general experience data from similar interventions, and interviews with key informants to provide contextual information.
The construction of counterfactuals is another tool evaluators use to understand and measure the impacts of interventions. Counterfactuals enable evaluators to ascertain what the situation would be like if the intervention had not taken place. For example, the welfare of participants in an intervention might be compared to the welfare of a control group who have not taken part in the intervention. This comparison helps depict what the welfare of individuals would have been like in the absence of the intervention. Other methods of simulating counterfactuals include using computable general equilibrium models (CGE) or comparing countries with the intervention against countries without. Through the use of techniques such as triangulation and counterfactual construction, Impact Evaluation addresses some of the challenges faced when tracking and attributing change in the conflict context. Action Evaluation copes with the challenge of judging success and failure in the conflict context by radically redefining these concepts. Jay Rothman is a pioneer of Action Evaluation who has written extensively about its theory and application. Rothman describes Action Evaluation as a process designed to support stakeholders as they collaboratively define and redefine success. It is an approach to evaluation founded on the principle that the more consciously ‘success’ is defined and monitored by stakeholders; the more likely it will be achieved by them. As Marc Howard Ross suggests in a review of Peace and Conflict Impact Assessment (PCIA) as a Peacebuilding Tool, Action Evaluation is founded on the premise that ‘failure’ is a beginning rather than an end. Role of the Evaluator Action Evaluators are therefore trained to collect, analyse and share data with middle-range and grass roots communities. Unlike more conventional approaches to evaluation, the evaluator is not expected to act as an ‘objective’ third party. Indeed action evaluators are encouraged to engage in close relationships with middle-range and grass roots actors. Yet at the same time, action evaluators are expected to maintain a degree of detachment from the intervention. Rothman describes this detachment as an important element of ‘disciplined bias.’ However, in order that this element of ‘disciplined bias’ is not perceived by other stakeholders to be evidence of the evaluator’s ‘outsider’ status, the evaluation process is designed to be transparent and open to scrutiny by all. Process Action Evaluation copes with some of the challenges of tracking change in the conflict context. In particular, Action Evaluation clarifies the goals and objectives of projects and programs. As Rothman suggests, the opening ‘base-line goal setting’ phase of an Action Evaluation spells out the purpose and functions of an intervention. This is achieved through the collection of information about the various goals elite, middle-range and grass roots actors have for the intervention. The action evaluator then analyses this information and feeds it back to these stakeholders who then define their shared goals. This goal setting phase ensures that the stated goals and objectives of an interventions are clear to everyone concerned. Moreover, the collaborative nature of this goal-setting phase helps ensure that the goals set for an intervention are realistic and appropriate. It is subsequently easier to evaluate the extent to which interventions achieve their goals and objectives.
Following the definition of shared goals, the action evaluator facilitates the continuous monitoring and assessment of these goals. In fact, a computer assisted and web-based instrument and database system have been designed to enable on-going data analysis and program monitoring. Assessment is thereby fully integrated into the development and implementation of an intervention. Hence, Action Evaluation is able to take account of the sudden, ongoing changes that characterize the conflict context. Moreover, the goal-setting phase of an Action Evaluation is repeated at various points throughout the intervention’s life-cycle. This provides space for the shared goals of an evaluation to evolve over time. Action Evaluation is therefore especially suited to the evaluation of conflict management interventions in the conflict context. Ultimately, conflict management interventions are assumed to impact on the dynamics of peace or conflict in a given context. Evaluating such impact is extremely difficult. Broadly speaking however, macro-evaluation aims to do just this. That is, macro-evaluation determines how grassroots, micro-level interventions, ‘ripple up’ to the country or region-wide macro-level. Macro-evaluation therefore focuses upon the long-term impacts of interventions. Assessing Macro-Level Impact The field of conflict management has yet to agree upon a precise definition of macro-evaluation. According to some, macro-evaluation means assessing how interventions in various fields and multiple policy instruments impact on the dynamics of peace and conflict. This type of macro-evaluation is perceived to inform the development of more integrated and ‘joined up’ approaches to working in areas of violent conflict. Recently, a number of donor countries including Sweden, Canada, Norway and the UK, together with a variety of NGO’s including CARE, Oxfam and Save the Children, have begun to develop holistic approaches that mainstream peacebuilding with, for example, traditional sustainable development and humanitarian assistance. Mark Hoffman, who has written extensively about the practice of evaluation, argues that this interest in developing holistic approaches is partly a reaction to new awareness of connections between the fields of development, aid, humanitarian assistance and conflict management. It is also a reaction to critical evaluations of international responses in Rwanda, Sudan (Operational Lifeline Sudan) and Afghanistan (Review of the Strategic Framework for Afghanistan, 2001).
Adding Up Conflict Management Interventions Macro-evaluation is also defined as a process of ‘adding up’ or assessing the impact of only conflict management interventions on the dynamics of peace and conflict. This type of macro-evaluation takes the evaluation of an intervention’s impact a step further by seeking to evaluate the extent to which the impact of an intervention synergizes with other conflict management interventions. This type of macro-evaluation shows whether the conflict management interventions in a given context add up to more than or less than the sum of their parts. It is therefore said to deliver judgement on the success or failure of the field of conflict management as a whole.
|