impact assessment

'... impact assessment studies keep donors happy... we don't use them very much' (director of a large Asian microfinance institution that has received substantial amounts of aid-financed IA consultancy and internal IA-capacity building quoted Hulme 2000)

'So we find out that such and such a percent of people have such and such a percent increase in income. So what. It doesn't tell us what to do about it.' (participant at Micro credit summit meeting New Delhi 2001)

Impact Assessment: Current Challenges

Impact assessment methodologies are currently at a crossroads.

On the one hand the underlying agendas of pro-poor development and 'improving practice' necessarily require participation by poor women and men in deciding priorities and identifying strategies. This new agenda means that simple 'rigorous' measurement of before and after situations for random samples with control group is now rarely sufficient. The focus on pro-poor development and improving practice implies new questions, methods and processes to address local priorities, differences between poor people, causality and attribution and to identify and bring about pro-poor change - ie the underlying and explicit justification for the large amounts of money spent on impact assessment.

On the other hand the sheer numbers of people involved, the potential conflicts of interest and consequently difficulties of decision-making require rigorous quantification and analysis in order minimise domination by vocal vested interests. 'One-off snapshot' impact assessments are likely to be extremely expensive and insufficient. Meeting this challenge requires multistakeholder collaboration on an ongoing basis in order to cost-effectively and reliably collect the large amounts of information necessary. This includes networks for community research as part of a sustainable learning system.

For more see MAYOUX, L. (2002) From Impact Assessment to Sustainable Strategic Learning, EDIAIS

Reversing the Paradigm

In recent years there has been increasing emphasis on 'integrated impact assessment', using varying combinations of survey, qualitative and participatory methods in order to meet these competing demands. Complementarities have been recognised particularly between the depth and detail contributed by qualitative research and the objectivity and statistical robustness contributed by survey research. However in the current impact assessment paradigm , dominated by quantitative surveys, participatory methods have generally continued to be seen as a fashionable and 'politically correct' frill to the more serious task of 'expert' survey and (more rarely) qualitative research. As a result participatory methods have received insufficient investment of training, time and resources to be done well.

The various papers on this website argue the case for 'reversing the paradigm' to bring systematic and rigorous use of participatory methods to the centre of most monitoring, evaluation and impact assessment. Since the early 1990s experiences of quantification using participatory methods have repeatedly shown how when used well, participatory methods generate not only qualitative insights and but also quantitative data which are generally more accurate than those from conventional survey approaches and methods. Participatory methods can also be cost-effective by providing a better basis for targeting and by focusing more expensive forms of quantitative and qualitative investigation on issues and situations which need further investigation.

The challenges for participatory approaches and methods are not so much assuring rigour and reliability as ensuring that their mainstream use achieves their potential for enabling very poor women and men to have an equal voice in priorities and policies for pro-poor development. Much will depend on how they are used and by whom and the levels of political will not only to 'hear the voices of the poor' but also to 'listen' and take action.

For more see MAYOUX, L. and R. CHAMBERS ( 2005). " Reversing the Paradigm: Quantification, Participatory Methods and Pro-Poor Growth." Journal of International Development, March.

Participatory Evaluation

Evidence indicates that participatory evaluation has potentially significant contributions to make to increasing the relevance and reliability of evaluations, and to the pro-poor development process.

At the same time participation also has potential costs as well as benefits for all concerned. Participatory processes do not substitute for institutional commitment to wider goals of pro-poor development which may or may not be key concerns of many participants. On the contrary, these goals need to determine the underlying and non-negotiable principles on which the participatory process is based.

A clear and strategic commitment is needed to:

•  ensuring inclusion and informed participation of the most vulnerable stakeholders who are commonly excluded from many evaluations which claim to be participatory.

•  including these stakeholders in those stages in evaluation where participation can be most directly empowering to them. Participation may be more important at the design, analysis and dissemination stages than the actual collection of information itself.

Beyond this, the most important issue in design of participatory evaluations is to ensure collection of reliable information so that the participatory and inclusive process of analysis and dissemination leads to implementation of effective pro-poor policies. Although any one single evaluation cannot resolve all the tensions and trade-offs inherent in pro-poor development, it can make a contribution as part of an ongoing multistakeholder learning process to:

•  building up capacities and structures for ongoing representation of poor women and men and other vulnerable people in the policy making process.

•  facilitating direct interaction between powerful stakeholders and poor people in order to break down the barriers of complacency, misinformation and prejudice which are in themselves key causes of poverty.

For more see Mayoux, L. (2005)
Between Tyranny and Utopia: Participatory Evaluation for Pro-Poor Development.
PARC

Empowering Enquiry

All impact assessment methodologies, including statistical surveys, informal interviews as well as participatory methods, can be more empowering for those giving their valuable time to answering questions.

Empowering Enquiry provides simple guidelines which can underpin any methodology. Questions can be sequenced to:

• start by clarifying the vision people have
• celebrate what they have already achieved
• identify challenges to further progress
• identify clear concrete strategies for moving further along the road to their vision.

Questionnaires, interviews and participatory meetings can all be designed to contribute to increasing people's understanding of their situation and ways forward as well extracting information without necessarily increasing their length.

For more see: Mayoux, L. (2003b). Empowering Enquiry: A New Approach to Investigation, EDIAIS.

Return to top

Area Networking and Lateral Learning

One element of an empowering strategic learning process is area networking or lateral learning which brings peoples' representatives together to discuss experiences, programme policies and ways forward.

ANANDI (Area Networking and Development Initiatives) is an Indian NGO working with the poorest women of the tribal and other marginalised communities in Gujarat to organize for collective action and gender advocacy. Since 1999 ANANDI and other partner organizations have been facilitating lateral learning through holding of area networking ‘mela’ events or fairs. These events have brought together anything from 300 to 700 women’s representatives from the local groups to discuss many different issues and government policies: panchayat reform, cultural, ethnic and caste issues, basic needs and food security.

The melas provide an ongoing focus for reflection and action at local level. Since 2003 ANANDI has been adapting some of the PALS tools for more focused group-level planning and assessment to feed into the mela process. Contact ANANDI.

For more see Mayoux, L. and ANANDI (2005) Participatory Action Learning in Practice: Experience of Anandi, India.
final version: Journal of International Development March 2005.

Return to top

Further information: For more details in innovations in participatory methods see Participatory Action Learning System pages on this website or details in Annotated Bibliography at the top of this page.

For detailed Case Studies of impact assessment in practice see pages on Micro-finance and Ethical Enterprise or details in Annotated Bibliography at the top of this page.

Return to top