THE APPROACH

This guide presents an evaluation framework that assesses the contribution to recovery associated with post-disaster interventions. It is a method focused on assessing positive and negative changes to the lives of affected people and other local stakeholders, in the medium term following a disaster event. The output of the approach is a report that presents detailed findings, an in-depth analysis drawing on the findings, and a concluding section that discusses the contribution to change generated by post-disaster interventions – including a series of summary statements.

The approach assumes that changes in people's well-being and livelihoods can be most clearly identified at a household level. Assessing change necessarily involves identifying what the situation was like for households before and after the disaster occurred, as well as the situation following a period of post-disaster recovery and intervention. We present a technique for collecting retrospective data to cover these changes, combining quantitative and qualitative data collection and analysis.

The approach requires similar levels of staffing and resources as many other types of evaluation. However, although the methodology is structured, it requires key skills in design, field data collection, and interpretative analysis that may not always be available in agency field teams. It may often be advisable to commission a specialist third party to undertake the evaluation: this approach is well suited to a group of agencies acting together to commission the work. Note that the Contribution to Change methodology is designed to complement other evaluation tools; it is not a substitute for process evaluations or audits aimed at examining whether specific project inputs and outputs have been met.

Emphasis on evaluating 'contribution'

This guide uses the concept of Contribution to Change to describe the relative importance of post-disaster interventions in aiding people's recovery. This is distinct from a focus on 'attribution', which seeks to establish what specific changes have resulted from an agency's intervention.

This is because the activities of an individual agency, and the effects of those activities, will not normally occur in isolation but rather as part of a multilayered, complex response by both local and external actors (see 'Example from the field 1'). Social, economic, and political contexts, including the effects of international markets, access to communications, past or present conflicts, and environmental factors, also have a bearing on the final outcomes for a given population following an emergency. Thus it is more realistic in such settings to consider 'contribution' to outcomes or, as some authors prefer, 'contributory impact'.

The Contribution to Change approach views recovery as depicted in Figure 1. The idea is to try to understand the relative importance of the intervention activities depicted by the bottom arrow (normally this refers to the range of interventions in a given site, though in some instances the method may be applied to a single intervention).

Another important aspect of this approach is the recognition that it is not sufficient to look only at outcomes that have been achieved. An intervention may have contributed in a major way to the change that has occurred, but if that level of change has itself been only limited, there is a danger that the value of the contribution will be exaggerated.

We therefore also need to look at the overall progress of recovery in order to calibrate the contribution that interventions have made to the extent of recovery that has actually been achieved. Both are important aspects of assessing contribution to change.

Concentrating on just the 'intervention' part of the previous diagram, we can depict this distinction as shown in Figure 2.

Assessing contribution is not an easy task. Although we present a systematic methodology for undertaking this type of evaluation, the end results of the analysis require careful interpretation of the evidence collected. We need to look at the effects of the intervention, consider other responses and contributions, and be aware of the effect of external factors and issues in shaping recovery. All these elements need to be included in the analysis and its conclusions presented in the report.

The most important output of the Contribution to Change methodology is a detailed discussion of how these elements have shaped key aspects of people's lives. This will be reflected in what we call the narrative analysis (see Part Three, Step 9) and the conclusions (Step 10), which are set out in the final report (Step 11).

However, one way to provide a shorthand interpretation of contribution is to create Contribution to Change statements (the tool for which is provided in Step 10). This tool generates simple statements about level of recovery (the overall progress of recovery) and contribution to recovery achieved (the changes that can be linked to interventions), and compares these to create a Contribution to Change statement (the contribution of the intervention towards meeting the recovery that is required).

Other defining elements of the methodology

Focus on livelihood changes at household level

The methodology is rooted in the idea of undertaking analysis at the grass-roots scale in order to reveal the most significant changes in people's lives associated with a disaster and the subsequent 'recovery' period. Recent reviews of evaluation in post-disaster settings argue that it is imperative to ensure that assessing the effects of interventions on the 'lives of affected populations' is at the heart of evaluation (Proudlock et al., 2009).

The intention of the approach is to take a holistic or multi-dimensional view of people's lives – one that looks broadly at different aspects of people's well-being. It is felt that a more holistic evaluation of the changes in people's lives is needed as all too often agency evaluations focus on the components of their own intervention, for example the provision of health services or clean drinking water. The data therefore give a partial picture of the changes in people's lives and may miss crucial information needed in order to assess how well communities have recovered.

It is important not to view disaster impact and recovery narrowly in terms of only income-generating activities. The 'livelihoods' concept is one useful way of thinking about this (see Scoones, 1998). It refers to the range of material and non-material assets and resources to which people have (or do not have) access in order to achieve well-being. This enables recognition of the role that different types of assets have in supporting recovery from disaster situations.

We can consider this level of approach as both normative (in terms of a focus on people's needs/recovery) and instrumental (in terms of agencies' operational objectives – by putting interventions into their social context).

Another important consideration in developing this approach concerns the level at which change is expected to be observed. We have decided to focus on households as the most important location at which changes can be observed. Most core humanitarian and post-disaster recovery interventions – education, health, WASH, shelter, food distribution, nutrition programmes, seed distribution, income generation, cash-based programming, infrastructure redevelopment, etc. – ultimately have impacts that are most likely to be recognized by affected populations if they result in changes in the daily activities and livelihoods of households.

Although the focus is on a household level of analysis, it is usually also important in the evaluation to build in attention to other key social dimensions such as differences in disaster impact and recovery associated with gender, age, ethnicity, caste, and income group. This may require targeting within the data collection design. For example, short- and long-term impacts can play out differently for female-headed households, as well as for women and men at an intra-household level (see Box 3 for a summary of how gender dimensions are approached in this guide).

The timeframe of the approach

The measurement of change necessarily means that data or information about a given change (e.g. to a household's or community's livelihoods) needs to correspond to at least two points of time. In the case of post-disaster interventions, it makes sense to look at three or four points in time, because we need to understand both how the disaster impacted people's livelihoods and how the recovery progressed.

To describe the different points of time that are of interest, we have used the following terminology, which is represented graphically in Figure 3:

• T-1 – before the disaster (before the onset of a hazard);

• T0 – the disaster event (the onset of a hazard – see also Box 4);

• T+1 – early post-disaster (after an initial emergency period);

• T+2 – late post-disaster (after a recovery phase period).

Because of the interest in livelihood trajectories and because the impacts of a disaster on these may not be immediately assessable, T+1 is an important time point in the analysis. Conceptually at least, this is the point when livelihood impacts are considered to be at their greatest, but also the point when recovery actions are seen to commence, including the implementation of post-relief interventions. We recognize that this is a simplification of what happens in reality on the ground, but it serves as a basic conceptual framing for the methodology.

The precise timing of T-1, T+1 and T+2 will depend on the unique context and considerations for evaluation design, but Part Two, Step 2, provides some guideline pointers to help with these decisions. The data collection tools described in Part Two relate in different ways to these four points in time, focusing most on the differences between T-1 and T+1 (what were the impacts?); T+1 and T+2 (what have been the responses?); and T-1 and T+2 (has there been an effective recovery?).

Retrospective data collection

The method presented in this guide assumes that primary data collection for the analysis will actually take place at T+2. As discussed in the Introduction, it is seldom the case that detailed baseline data can be collected prior to this point that relates to all aspects of livelihoods and to a basket of interventions.

Given the inherently unpredictable nature of rapid-onset disasters, the chaos that usually ensues, and the need, for humanitarian reasons, to respond urgently, evaluators rarely have the opportunity to identify a clear baseline at the time of the disaster. That means that while data corresponding to T+2 can be collected in real time, in most cases data corresponding to T-1 or T+1 will have to be collected retrospectively, based on the participants' recollection and backed up by any written records that may be available.

Piloting of the surveys has shown that retrospective data can be collected from households, with reasonable confidence in the reliability of those data. Triangulation between sources should reveal discrepancies and unreliable sources can be discarded from the analysis. Nevertheless, to minimize bias, the retrospective data collection ideally should take place within 6 to 12 months, although a balance has to be struck between issues of data recall (see Box 5) and the need to allow enough time for livelihood changes to become manifest.

In some cases, it may be possible to follow a two-phase data-collection approach, where initial and follow-up data can be collected separately. This will be used when a first visit to the site(s) affected by the disaster can be done one to two months after the event (for the collection of data covering the period T-1 to T+1). A follow-up visit can then be done 12 to 15 months after the disaster (for the collection of data covering the period T+1 to T+2). Both visits will collect similar data, although questions will need to be modified to reflect the different timings of data collection.

Mixed methods for data collection

The methods and tools described in Part Two and Part Three aim to provide a relatively simple and effective way of identifying the changes at the household level for agencies interested in identifying and documenting credible evidence of their contributions to change.

Any evaluation requires important consideration of the most appropriate methods of gathering data or evidence about livelihoods at the household and community levels. Methods used need to be robust enough to collect credible evidence but simple enough for field application in a range of contexts (see Box 6). The approach proposed in this guide is designed around the use of qualitative and quantitative methods that should be quite familiar to most researchers and evaluators.

Quantitative methods use larger samples and are good for determining the direction and amount of change across a population. Typically, the quantitative methods likely to be useful include:

• household questionnaire surveys;

• community-level questionnaires.

Qualitative approaches usually involve in-depth work with smaller numbers of people. They can bring depth of understanding to the context and processes of change, and can be useful for understanding perspectives on the reasons why a change has or has not occurred. They are also more participatory in terms of providing more opportunities for the 'voice' of disaster-affected populations to be expressed. Typically, the qualitative methods likely to be useful include:

• semi-structured interviews (key informant, household and group interviews);

• participatory techniques (group exercises such as participatory mapping and event timelines).

Recognizing the different advantages and limitations of both qualitative and quantitative research tools, the approach taken in this guide is to recommend the joint use of both qualitative and quantitative methods to develop a more detailed narrative of change.

Where possible, the evaluation should utilize other sources of data such as government records and statistics, agencies' own monitoring and evaluation reporting, and spatial imaging. These may explain, supplement, or provide valuable checks for household-level data (see Part Two, Step 1).

Where possible, during the process of analysis, different sources of data should be compared using the principle of triangulation to verify findings, reveal inconsistencies, and allow consideration of possible sources of bias or inaccuracy. This means that some limited repetition of question themes within different data collection tools and sources is useful.

Planning and management

As noted in the Introduction, the Contribution to Change methodology is particularly suited to a multi-agency approach, although it can also be conducted for single agencies. In either case, the management team that is commissioning the work has to make decisions on the scope and scale of the evaluation, and on the roles, responsibilities, and composition of the evaluation team. It is also important to consider at this stage how the outputs of the evaluation are expected to link with other evaluation processes.

Planning and resources

The first considerations relate to the scope and scale of the evaluation. The characteristics of the disaster and of subsequent interventions will largely determine how large-scale and complex the evaluation might be, but the final decision is likely to rest also on cost and staffing requirements. It is critical to understand fully the methodology presented in Part Two and Part Three before coming to these decisions, but, in terms of resource implications, the key factors are the time, staffing, and logistics required for:

• preliminary work – initial investigations, data collection design and field preparation (see Part Two, Steps 1–5); in most cases, this will require a small team of senior-level personnel (approximately two to five people, depending on the number of field sites) over a period of four to six weeks;