CHAPTER 1
Introduction
This book is the result of an action-research exercise jointly undertaken by a number of international and local non-government organisations (NGOs) based in four continents. The research was initiated by Oxfam from Great Britain and Novib from the Netherlands (hereafter referred to as Oxfam and Novib). The book is an attempt to share with others the lessons learned from that experience, as well as some of the challenges that arise from thinking about and practising impact assessment.
It is aimed particularly at practitioners, and evaluation specialists may find that I skim over some of the more theoretical issues. This is not because these are unimportant, but because our research showed that there is a need to demystify the subject. The aim is to make impact assessment accessible without being simplistic about it. The book also explores those elements of impact assessment which look beyond the project level at organisational processes. I hope therefore that those involved in the development of monitoring and evaluation systems at an organisational level will find this book helpful. However, it is essentially about the insights gained from the case studies, rather than an attempt to cover the abundant literature on the subject.
Impact assessment — making the case for development aid
Despite the statistics in recent UNDP Human Development Reports and in the World Bank's annual reports, which record a marked improvement in a number of indicators of human well-being, the scale of world poverty remains a scandal which shames us all. In many parts of the world inequality, insecurity, and conflict are growing at alarming rates. Bilateral development aid has had its critics for many years, but during the past decade we have also seen a growing number of critiques of NGOs (Smillie 1995, de Waal 1996, Sogge 1996). These critiques together describe a vicious circle which the NGO sector, particularly in the North, faces and which it has helped to create. This circle has five main elements (see Figure 1.1): there is increasing pressure on NGOs to demonstrate results and the impact of their work. Moreover, there is increased competition between NGOs, and a growing need for a high profile and press coverage in order to raise funds and to facilitate advocacy work. Poor institutional learning and weak accountability mechanisms are characteristic of many NGOs, which both leads to and is the result of the absence of professional norms and standards.
These elements combine to produce a growing gap between the rhetoric of agencies and the reality of what they achieve. They also fuel growing scepticism about the value of aid, and lessen trust between agencies — some argue that agreements and partnerships based on shared values have been replaced by bureaucratic trust based on plans, budgets, and accounts. Moreover, the elements of this vicious circle to perpetuate the tired old image of aid going from donor to 'victim', and a view of development as something that is done to other people, far away. This analysis, although based on little empirical evidence, holds a certain truth which NGOs ignore at their peril. One recent study of NGO impact which had a stronger empirical base concluded that the true impact of NGO development work remained unclear and that there was little consensus on which tools and methods were the most appropriate to find out. This study, undertaken by Riddell et al. (1997) for the OECD/DAC Expert Group on Evaluation, reviewed 60 reports covering 240 projects in 26 countries, and undertook 13 country case studies.
Both the critique outlined above (mainly of Northern-based NGOs) and the findings of this OECD study point to the inadequacy of most current attempts to promote institutional learning, impact assessment, and greater accountability in the NGO sector — accountability to those who NGOs seek to support, as well as to those who fund this work. In a climate of increased competition, individual organisations and the sector as a whole tend to exaggerate the case for support just as their opponents tend to exaggerate the case against. This can have two consequences, the dangers of which have been pointed out for some time (Cassen 1986, Riddell 1987). First, support for development aid depends on the public's belief in its effectiveness. The moral case for providing support rests upon its achieving its objectives. However, a reluctance to admit that the effectiveness of much of that is done is unpredictable and difficult to assess, makes not just NGOs, but also international co-operation programmes, vulnerable to public criticism and the odd polemic attack.
Second, those making the case for co-operation must not create the belief that aid flows constitute the sole, or even principal, means available to donors and governments of improving the welfare of people living in poverty. Often, changes in policy and practice, for example improved terms of trade or greater debt relief, may be more beneficial.
In the long term, the case for aid can only be sustained by more effective assessment and demonstration of its impact, by laying open the mistakes and uncertainties that are inherent in development work, and by an honest assessment of the comparative effectiveness of aid vis-à-vis changes in policy and practice. The research project that forms the basis of this book was designed to make a contribution to this process, as well as to the institutional learning of the agencies involved. A broad range of intermediary and grassroots organisations in nine countries participated alongside the Northern NGOs Oxfam and Novib. More specifically, we had the following aims:
• to develop greater clarity about the key elements of impact assessment;
• to explore how unequal levels of power and participation of the various stakeholders can affect an impact assessment process;
• to test a range of approaches to undertaking impact assessment;
• to look at the organisational context in which impact assessments take place.
These issues are discussed in more depth in Chapters 2 and 3, and I will return to them in the final chapter.
The structure of this book
The second half of this Introduction describes the case studies and the organisations involved in preparing them. The case studies form the foundation of the book, and I have attempted to retain their richness and diversity. Chapter 2 discusses Oxfam and Novib's overall approach to impact assessment. It explores issues relating to ethics and participation, and emphasises the importance of ensuring that gender and other aspects of difference are embedded into all processes of impact assessment. Chapter 2 also sets out some of the assumptions and hypotheses which emerged from a literature review at the beginning of the research and which were subsequently tested in the case studies. For example, we confirmed the hypothesis that impact assessment must be seen as an integral part of development work throughout the project- or programme-cycle. Finally, this chapter summarises some of the key dilemmas regarding how to attribute any observed change to an intervention, and how to synthesise diverse and sometimes contradictory findings.
Although Oxfam and Novib feel that the distinction between emergency, development, and advocacy work is increasingly redundant, these terms do provide a convenient short-hand for different aspects of NGO work. So although Chapters 3 to 6 are structured along these lines, there is much overlap between them. Chapter 3 explores how the case studies were designed for assessments in single communities, across several communities, and across a range of projects. It also gives some guidance about basic operational matters which must be addressed in designing an impact assessment, such as how to go about sampling, what to do if there is no baseline data, and how to cross-check results. It also looks at how some of the most difficult issues such as setting indicators and measuring attribution were tackled in our case studies. In Chapter 4,1 look at the wide range of different tools, methods, and approaches used as well as the problems encountered in the case studies. The reader will find summaries of the lessons learned about various kinds of tools and methods such as surveys, interviews and workshops, direct observation, participatory tools, and case studies. Chapters 3 and 4 make up the core of the book. Many of the chapters that follow refer back to some of the basic lessons described in these chapters.
Chapter 5 on impact assessment and emergencies refers mainly to one of the case studies as well as to some recent material. It explores some of the specific difficulties and challenges of undertaking impact assessment in crisis situations and compares them with the lessons described in previous chapters.
Because of the limited number of case studies directly dealing with emergency situations, this chapter should be seen as a preliminary exploration of the issue; it mainly focuses on relief interventions. The chapters on advocacy and organisations and impact assessment are also relevant to work in emergency situations. Recent events such as Hurricane Mitch and the Kosovo crisis may offer opportunities to work on more longitudinal and holistic approaches to emergency situations and their assessment.
In Chapter 6, which focuses on impact assessment and advocacy, I outline current approaches to this growing field of NGO activity. Drawing on the case studies and other experiences, I discuss both how to assess the impact of advocacy and how to use impact assessment in advocacy work.
Chapter 7 considers how we can assess the impact of projects and programmes on development organisations, and at how these organisations manage impact assessment processes. It also considers the impact that development organisations have on people's lives.
I draw some conclusions from this research in Chapter 8 and indicate areas which NGOs need to invest in if they are serious about impact assessment. I end by noting key policy implications that emerge from the case-study findings, and thinking about what these might mean for the future of NGOs. The appendices include a bibliography, further details of the organisations and individuals involved in the case studies as well as a list of acronyms.
The case studies
When Oxfam and Novib embarked on this study in 1994/5, we undertook an initial literature review as well as a rapid survey of what sort of work on impact assessment was happening in a number of countries where we supported projects and organisations. A number of reports resulted (Hopkins 1995 a and b, Dawson 1995 a, b and c) which gave a useful overview of existing concepts and tools as well as suggesting a number of principles and hypotheses for impact assessment, which are discussed in Chapter 2.
However, the literature review concluded that a more 'hands-on' approach was needed to test a range of approaches in diverse contexts with different types of organisations.
The most important challenge is to promote, through case studies, a systematic application of the methodologies on impact assessment. There has been in the past few years a significant production of tools of project evaluation ... but what is really lacking is both a vigorous and systematic application of techniques and methodologies and to learn from experience. (Hopkins 1995a)
Oxfam and Novib then approached a number of partner organisations and their own field staff to explore the potential for undertaking action-research on impact assessment and developing a number of case studies. As is common with this sort of exercise, willingness to participate became the most important criterion. So although we also wanted to ensure a good geographical balance, to include emergency and advocacy as well as long-term 'development' work, and to balance forward-looking with retrospective studies, these criteria were inevitably secondary.
In the end, case studies were carried out in Africa (Ghana, Kenya, Zimbabwe and Uganda), in South Asia (Pakistan, Bangladesh and India), in Latin America (El Salvador) and in Europe (the United Kingdom). They represent a balance of prospective work, mid-term assessments of ongoing work, and retrospective reviews — although it became clear during the exercise that these distinctions are in reality often blurred. Unfortunately, only two of the studies can be described as assessing an emergency programme. However, some of the others deal with work that emerged from responses to emergency or crisis situations. The case studies do not include rescue or acute emergency operations, but I refer to some material on assessments in these circumstances in Chapter 5.
Given the nature of the research and the diversity of programmes and partners involved, the case studies did not follow a set methodology. We adopted an open and flexible approach which was designed to encourage innovation, creativity, and 'learning by doing'. We hoped that this might lead to greater variation in the approaches, tools, and methods used and maximise the potential for learning across the case studies. A few of the original participants had to drop out due to unforeseen circumstances and were be replaced as other opportunities emerged: a fixed method would not have permitted such adaptation. On the other hand, this flexibility has made the job of synthesising the experiences all the more challenging!
Who was involved and why?
Table 1.1 gives an overview of the organisations involved in the case studies. These range from BRAC in Bangladesh, which has a budget of some $200 million, 20,000 full-time staff and works with nearly 2 million people, to the Matson Neighbourhood Project in Gloucester, UK, which has a budget of £.300,000,16 staff (eight of whom are part-time), and covers a single housing estate of 6,000 people. Information is provided for Oxfam GB and Novib as a whole as well as for specific country programmes. The difference in size and scope of the participating organisations is reflected in the size and scope of the impact assessment processes they have undertaken, as well as in the human and financial investments they have been able to make over and above the financial support of Oxfam or Novib. In all but one case (Matson), either Oxfam or Novib are — or have been — funding these organisations, in some cases for many years.
Given this relationship, it is of course possible that participants felt somewhat 'obliged' to take part in the research or hoped that it would have a favourable effect on their relationship with Oxfam and Novib. Some of the tensions and difficulties that emerged during the case studies can probably be attributed to this. Nevertheless the main reasons of agencies for participating can be summarised as follows:
• to learn more about impact assessment;
• to assess and improve performance, and plan for the future;
• to integrate learning from the research into management, monitoring and evaluation, and reporting systems;
• to share experiences of impact assessment with others;
• to improve the relationship with both communities and donors;
• to motivate staff and the communities supported;
• Three areas of consensus can therefore be defined as explicit reasons for involvement: learning, performance, and accountability.
Overview of the case studies
This publication cannot look in great depth at the background to the case studies or the context in which these projects and programmes developed. Below is a short summary of each of the projects, which is intended as a quick reference guide.
Institutional capacity-building in Northern Ghana (supported by Oxfam GB, Integrated Social Development Centre, Northern Ghana Development Network)
This programme was set up in 1995/6 to strengthen the capacity of more than 45 NGOs and community-based organisations (CBOs) which make up the Northern Ghana Development Network (NGDN). The project, undertaken by Oxfam in collaboration with the Integrated Social Development Centre (ISODEC) and the Northern Ghana Development Network, seeks to assess the impact of the institutional capacity-building programme on poverty in this region by establishing a baseline against which future changes in poverty levels and in organisational development can be measured. In doing so, past trends are also being identified. Changes will be tracked at regular intervals as the programme progresses, at the individual, community, and organisational level. This approach to impact assessment (often referred to as the longitudinal approach) helps us understand the relationship between these levels. It is hoped that the member organisations of NGDN will use the methods developed in this research for their own impact assessment needs.
