Marco Kokic/Internal Federation, India

Chapter 6 - summary
Measuring the impact of humanitarian aid

Knowing and being transparent about the effects of one’s actions is part of being an accountable organization. Yet the measurements of success too often focus on ‘outputs’ only – how many tonnes of food aid or blankets delivered, how many cubic metres of clean water provided, how much cash spent per capita. These crude measures fail to analyse the actual results of such aid – whether lives have been saved, health and nutrition have improved, money was well spent – and whether these results were because of the aid effort or for other reasons. Many challenges – both technical and ethical – face those aiming to assess humanitarian aid’s results.

Impact is defined as a significant or lasting change in people’s lives brought about, at least in part, by a given intervention. These changes may be positive or negative, expected or unexpected. Change may result directly from a given intervention, e.g. saving lives by providing food; or indirectly, e.g. lobbying to protect a group of people. The key questions are: what has changed, is it significant or lasting, and to what degree can it be attributed to a given set of actions? And who decides? There are three different approaches:

• 'Project-out' approach: defines outcomes in advance and assesses whether they have been achieved. This can lead to 'ego-centric' results which exaggerate the importance of interventions and diminish the role of survivors.
• Assessing broader impacts: starts with a given intervention, but assesses broader, unexpected changes and analyses different stakeholders' opinions.
• 'Context-in approach': assesses significant changes in people's lives, then explores the sources of those changes. This approach compares changes brought about by different actors and assesses how different actions combine to promote change.

The unpredictability of emergencies requires a continual process of 'tracking' impacts and adapting indicators to the context. Some key indicators include:

• Mortality and morbidity rates – important both in designing and adapting appropriate responses.
• Coverage and differential impact – the proportion of men, women and children covered by a particular intervention.
• Protection and security – the degree to which human rights have been protected over and above material needs.
• 'Connectedness' – how far emergency responses support longer term recovery and sustainability.
• Coherence and coordination – the degree to which different agencies' agenda are consistent and their actions coordinated.

In addition, there are some broader frameworks which emphasize particular elements of performance, including Sphere's minimum standards and the Red Cross Red Crescent’s Code of Conduct (227 signatories by March 2003). The use of the Code's ten principles in evaluating responses to the Gujarat earthquake enabled comparative assessments between different agencies.

However, critics argue that universal standards undermine the importance of diversity, participation and an understanding of context in humanitarian work. Another approach is the Quality Project of French NGO Groupe URD, which measures impact through surveys, interviews and research.

Involving local actors in assessing impact brings its own challenges. Participatory research often ignores power inequalities and fails to capture the views of the most marginalized. The space and time available for participatory research will vary according to context, so methods must be adapted to specific situations. And attention must be paid to ensure the confidentiality of interviewees for security reasons.

In the Gujarat evaluation, a survey of 2,300 local people improved the evaluation's objectivity. The vast majority felt satisfied with the quality of food and nutrition services provided. But many also felt that agencies arrived with pre-determined projects and sought only minor modifications or simply free labour from local people. If the findings of participatory research are going to be acted upon, then organizations must be able to cope with dissonant feedback and modify plans accordingly.

One of the key challenges is how to attribute impact to individual interventions or agencies. Demands to improve accountability have led to many donors assessing only those outcomes for which individual agencies can be held accountable. This creates perverse incentives:

• Agencies will take fewer risks, sticking only to interventions which they can report on most easily – even if other less easily reportable actions could have greater impact.
• Undermining role of professional judgement. Performance indicators are selected for ease of measurement and control, rather than because they accurately reflect the quality of aid. This restricts initiative to adapt to changing circumstances.
• Pressure to show attributable results will limit cooperation with other agencies, where it's difficult to distinguish between different contributions – even if cooperation could have greater impact.

Greater emphasis is needed on assessing how different actors combine to promote change. This requires fundamental shifts in how donors relate to aid agencies.

Some specific recommendations have been made regarding the ethics of assessment methods: avoid raising expectations that can't be met; promote evaluations that are meaningful for locals rather than aiming purely to extract information; respect informants' personal and time constraints; recognize that evaluations can increase tension and put people at risk; ensure that the most marginalized are listened to; strive to put evaluations into the public domain and make lessons available to others.

Analyst Hugo Slim has proposed a framework for ethical analysis which is useful for prospective and retrospective impact assessment. Questions to consider include whether an agency:

• has a duty-bound (life-saving) or consequentalist (life-enhancing) ethic;
• acts on the basis of sound motives or other considerations;
• gathers enough information to inform its decision-making;
• has the capacity to do anything differently;
• is committed to debating the consequences of difficult decisions; and minimising the likelihood of negative impacts; and
• promotes the ethical skills and knowledge of its staff.

In 2003, international NGOs used this framework to debate whether to take money from 'belligerent' governments for relief operations in Iraq. Oxfam, which subscribes to a 'consequentialist' ethic, was concerned that operations using belligerents' money could have negative longer term consequences for the Iraqi people.

For progress to be made in assessing impacts, it is important to test different approaches in real situations. But if organizations do not improve their capacities to learn from such assessments, then little real change will occur. Avoiding this entails developing methods – such as ‘real-time’ evaluations or ‘impact tracking’ – that provide timely feedback to decision-making. It also requires profound changes in organizational cultures so that agencies can cope better with criticism. In conclusion, agencies must address the following:

• Make the question of impact central to all programme design, monitoring, reporting and evaluation.
• Ensure the right stakeholders are involved in judging impact - including the disaster-affected themselves.
• Examine agency culture, behaviour and incentives. Is learning rewarded? Are errors buried? What disincentives to improving impact and learning exist?
• Explore how to work with others, to pool information and learning. Analyse how change is fashioned together, rather than individually.
• Develop ethical frameworks to help navigate through the dilemmas of an era when geo-politics dominate humanitarian affairs.

This chapter and box were contributed by Chris Roche of Oxfam Community Aid Abroad.