CHAPTER 1
Introduction
Rosalind Eyben and Irene Guijt
Abstract
This chapter introduces the concerns that have guided the Big Push Forward and its culminating conference about the politics of evidence from which originated the case studies in the present book. The book's principal themes that emerge from the case studies are identified and the chapter concludes with a summary outline of the contributing chapters.
Keywords: Big Push Forward, results and evidence, power practices, accountability, transformational development
In June 2014, the board of the Global Partnership for Education (GPE) adopted a new funding model 'to have a positive leveraging effect on the development of national sector-wide policies, strategies and systems' (GPE, 2014: 4). Thirty per cent of the funds were to be used for ex-post payments against predetermined results. The United Kingdom, Australia, and the United States had pushed for a greater percentage of the allocation to be through payment by results (PBR), but the majority of the board members were reluctant to risk a higher proportion of funding to a bold experiment they did not fully understand – and of which the pilot phase was cancelled. One of those objecting to PBR in principle (a civil society representative) emailed Rosalind Eyben:
I don't think the Board really understood the implications of this when it approved this new model – and [recipient] countries certainly were not aware of what this means in practice.
Despite some discomfort about where the PBR model was taking them, most board members found it difficult to challenge the proposal. Everyone wants results! Eyben's informant, on the other hand, was keenly aware of how PBR risks instrumentalizing education, apart from its possible perverse effects (see Chapter 2). He feared that judging children's performance against measurable learning outcomes would become the sole yardstick for value, crowding out a transformational approach to education as a process of empowerment.
PBR is one of the mechanisms of a 'results and evidence agenda' that seeks to improve and manage development aid through protocols, procedures, and mechanisms for reporting, tracking, disbursing, appraising, and evaluating its effectiveness and impact. As detailed inChapter 2, the agenda became influentialin the first decade of the present century, with, by around 2010, an increasing number of development practitioners concerned about what they saw to be the agenda's pernicious effects. As in the GPE case, they worried that the results and evidence agenda undermined the potential for development aid to support transformational development and risked reinforcing power relations and structures that reproduce rather than diminish inequality, injustice, and the non-fulfilment of human rights. Concerned practitioners mentioned the time and money wasted in negotiating with funders over the utility and feasibility of imposed protocols and complained of the accountability pressure that forced the generation of 'sausage numbers' (Chapter 3), leaving limited time and energy for adaptive and responsive programming in support of complex change processes. One worried senior official from a United Nations agency rang the authors of this chapter to explain how his agency had negotiated for several months with a government funder:
They themselves knew it was ridiculous what they were asking for but they said it was political. In the end it comes down to money and for X millions of dollars we had to agree. But we rely on you academics working on the flanks to start a conversation about this.
Where could these and the growing number of other concerned voices discuss the trends, consequences, and options?
The initial impetus for what became the Big Push Forward (BPF) was a conference in May 2010 in the Netherlands about evaluation in complex contexts (Guijt et al., 2011). Although it was borne out of a deep disquiet about how the term 'rigour' had been captured by advocates of certain impact approaches, most participants discussed methodological innovations and challenges with little reference to the shrinking political space within which the methods discussed could be used. Knowing that many practitioners were interested in the politics of how impact and results are defined and assessed, the authors organized a meeting in Britain where more than 70 participants shared their experiences and reaction to the results and evidence agenda. Six months later (April 2011) we launched the BPF to make the results and evidence agenda a legitimate subject for public debate. Our first blog laid out the challenge:
Hard evidence, rigorous data, conclusive proof, value for money, evidence-based policy are tantalizing terms promising clarity about what works and what should be funded in international development. Yet behind these terms lie definitional tussles, vested interests and contested worldviews. For those who hold the purse strings, certain ways of knowing and assessing impact are considered more legitimate than others. Yet increasingly people are recognizing the need for multiple and mixed methods and approaches to better understand complex change and that these compared to imposed standards are more likely to lead to fair assessments helping us learn how to support a fairer world.
In the two years that followed, development practitioners commented on our blogs and debated the issues at meetings and workshops with us, including in Germany, the Netherlands, Nigeria, Sweden, Switzerland, the United States, and the United Kingdom. In Australia, for example, we contributed to a workshop on evaluation methods for staff of AusAID (now part of the Department of Foreign Affairs and Trade), organized by the Development Leadership Programme. Participants discussed the practical challenges staff faced in juggling methodological and political demands in the messiness of development programming in complex contexts (Roche and Kelly, 2012). In these forums, practitioners spoke of managing these demands in ways that would enable them to pursue approaches more aligned with transformational development. We found that the politically alert were subtly playing the game and changing the rules, with some seeing the results agenda as creating opportunities to support transformational development.
The BPF conveners (joined in mid-2011 by Chris Roche, Cathy Shutt, and Brendan Whitty) were discovering how diverse and context-specific the effects of the results and evidence agenda were. Findings from a BPF survey (Chapter 3), comments on our website and the 'From Poverty to Power' debate (Green, 2013) revealed that the agenda was not necessarily harmful; it could trigger improvements in design and learning, as well as in accountability. Yet, whether on balance the effects were judged as positive or negative for transformational development depended on the perspective of the judge. That, in turn, begged a political question: 'Whose perspective counts?' We realized that such a fundamental question, along with others about the values and ideology of the results and evidence agenda, risked being ignored in a context of 'highly constrained resources, crazy time pressure, and the need to deliver some (any!) results to feed the MEL [monitoring, evaluation, learning] machine' (ibid.).
Furthermore, while accounts about the more negative power dynamics of the agenda were circulating over cups of coffee and glasses of beer, detailed case studies were rare. People were frightened of going public about their experiences of distortions and problems. They feared exposing international aid to an often-sceptical press, or being subjected to ridicule – or worse, putting their jobs or organizations at risk. The BPF conveners, therefore, decided to create an opportunity for practitioners and researchers to reflect critically on these issues in a safe space through a conference held in April 2013. We organized it so as to protect participants' identities, with a strict insistence on 'Chatham House rules'. Even so, some participants were reluctant to present case studies (even verbally) and others unwilling to go public in this book. Power was influencing people to opt out of sharing their experience of the reality of the situation. Even the contributors here have exercised care in their decisions about what to include and what to leave out of their chapters – and about how to portray choices about results and evidence.
To our chagrin, budget constraints prevented the financing of participants' travel and other costs, limiting the range of perspectives present at the conference. Apart from a video link with Ola Abu Alghaib in Ramallah, experiences of grantee organizations in aid-recipient countries were largely absent (as they are in this book). Participants were either development agency staff (bilateral, multilateral, and international non-governmental organization (NGO)) or managers or technical advisers in projects these agencies funded. The others were consultants and researchers, as were the BPF conveners.
The conference was organized around participants' own case studies of the artefacts – the processes, mechanisms, and tools employed by them and their organizations to assess results and to generate and use evidence (Chapter 2). This book contains a selection of these cases, as well as an assessment of the organizational and power dynamics associated with their use. Conference participants examined whether and how one can reconcile messy, unpredictable, and risky pathways of societal transformation with bureaucracy-driven protocols. Participants stressed the importance of conscious engagement in the politics of evidence (Chapter 11), providing positive examples of how they either successfully resisted the unhelpful demands of the results and evidence agenda or used these to good effect to challenge myths, promote methodological advances, and gain theoretical insights.
We chose the conference title 'The Politics of Evidence' from Denzin and Giardina's (2008) volume about why and how academic institutions and policymaking bodies regard evidence from qualitative research as being less robust and rigorous than quantitative research. We broadened the 'politics of evidence' term to encompass both how power works to define what is or is not accepted as robust evidence or determined as a result (in both programming and evaluation), and the resistance and contestation that such power generates.
Power and the politics of evidence
We analyse power both as an asset that individuals and organizations control and also as a process to which we are all subject. When conceptualizing 'power' as an asset, the politics concerns who controls the definition of a result or of evidence and what is acceptable to whom (Morse, 2006). This kind of power operates through the formal institutional arrangements for policy-making and implementation, such as the civil service, the legislature, and local government; certain policy actors, such as ministers and parliamentarians, are visibly powerful. Janet Vähämäki (Chapter 8) uses this perspective on power to examine 30 years of recurrent struggle over results-based management between Swedish ministers and the staff of the Swedish International Development Agency (Sida). Power as an asset can also be less formal. Private individuals and organizations, such as philanthropic foundations or academic think tanks, influence meaning and value in development. The financial or intellectual capital they deploy gives them legitimacy in articulating and promoting (or interrogating) the discourse. In 2012, the BPF (itself exercising some modest power in this manner) undertook a preliminary power analysis of the results and evidence agenda from this perspective, as summarized in Chapter 2.
Power as an asset can be related to the idea of 'power over', when A (with more power) controls how B (with less power) behaves. The international aid system is often portrayed as a power chain – official aid agencies at the top, international NGOs and recipient governments in the middle, and local community organizations at the bottom. Yet many rules and procedures cannot be clearly attributed to specific decisions made by identifiable actors. Much of the results and evidence agenda is an effect of more diffuse power dynamics. Even when someone 'in charge' makes a decision, they may be unaware of – or indeed unable to prevent – what actually happens, since this is an outcome of multiple interactions by myriads of interconnected actors. Those in authority – chief executives and government ministers – are as much subjects of power as the most junior staff member; and multilateral aid agencies as much as grassroots organizations.
This kind of invisible power is at work in all our relationships – each time we walk into a room, make a suggestion, or participate in a workshop. It is the process of socializing and embedding that shapes what we think, say, and do. An example from the present book is how societal norms render children marginalized and their voices disregarded (Chapter 9). Participatory and visual evaluation methodologies designed to elicit children's perspectives and knowledge fail to influence adult decisions, unless there are complementary efforts to change the social norms. Because it shapes habitual patterns of relations, this kind of power is rarely noticed – and therefore remains unchallenged. 'Power relations were never discussed during the term of the project: nobody brought it up,' observe Bernward Causemann and Eberhard Gohl (Chapter 10). Only by becoming aware of such power can anyone seek to change inequities in the relationship, an act of conscious agency.
Both of these approaches to power are drawn upon in Power and Organizations, which identifies 'significant and subtle power practices' (Clegg et al., 2006: 176) at work in institutional life. We find six approaches particularly relevant to the international development sector. Awareness of these can help people understand the nature of their experience with the results and evidence agenda:
• locking members inside and keeping outsiders out and systematically misrepresenting other realities;
• the division of labour into complex chains of power, enabling those at the top to maintain a distance from the effects of power;
• staff staying obedient through a ceaseless round of activities with little time for reflection;
• delegation to intermediaries, obliging them to implement decisions that have been made higher up the system;
• making those who are the subjects of power complicit in its exercise;
• applying instrumental and value-free science.
Reference is made to these power practices in the thematic analysis that follows.
Principal themes
The case studies in this book take up two substantive issues of concern to those interested in transformational development, namely the impact of the evidence and results agenda on the ability to pursue rights-based approaches and whether the growing emphasis on upward accountability is trumping mutual learning. Perspectives on these issues are informed by personal values and professional formation, and influenced by the position – or positionality – of individuals in an organization or of an organization within the aid system. These interact to affect people's judgement as to whether, on balance, the effects of the results and evidence agenda are positive or negative and to influence how people respond to the changes induced by that agenda. The dynamics of perspective and of push and response are thus examined first, followed by a discussion of how these play out with respect to the substantive issues of rights/results and learning/accountability.
Position and perspective
We launched the BPF to 'push back' against the bureaucracy-driven protocols we believed were shrinking the possibilities of development organizations supporting messy, unpredictable, and risky pathways of societal transformation. Not everyone shared our initial views! The survey of practitioners conducted prior to the conference (Chapter 3) suggests that whether the agenda's effects are experienced as largely positive or negative depends on a person's organizational role and on their organization's location in the aid nexus. BPF conveners fell into that category of development professionals, mainly academics and consultants, with the most negative views of the results and evidence agenda. In contrast, people with monitoring and evaluation (M&E) responsibilities had more positive experiences than, for example, programme officers, and senior staff were more positive than technical advisers or those in middle management. The importance of position and perspective is also evident in Chapter 5, where Chris Roche looks at the arguments between managers responsible for finance and fundraising, who want standardized, organization-wide performance measures, and those in programming and advocacy, who seek a variety of forms of assessment and reporting depending on the context and activity.
