M&E managers should be aware of a basic toolbox of data collection methods. This part of the module contains an initial session on the issue of sampling, relevant to all methods, and then sessions on each of key methods: desk review, observation, interviews and focus groups, surveys, information systems, population counts, Rapid Appraisal Process (RAP), the "in-and-out' field visit, i.e. in emergencies where access to a location is only possible for a few hours, the monitoring field visit, and case studies. The sessions are intended to focus on suitability of each method for different needs and contexts, limitations, and key issues in ensuring quality of data collection. Attention is given to applications in crisis and unstable contexts for all methods.
Sound sampling is key for quantitative and qualitative data collection methods. This session covers basic concepts and their practical application with particular emphasis on non-statistical sampling.
The purpose of this session is to help workshop participants understand why the study and use of existing data is so critical and must be an early step in developing any data collection plan for monitoring or evaluation activities. It also leads participants to identify the range of data sources to consider in a desktop review, as well as some basic questions that help to prepare the review. This session can be facilitated by drawing heavily on participants’ experience and common sense.
LINKS TO OTHER SESSIONS SESSION 2.X – CO-OPERATION AND CO-ORDINATION FOR INFORMATION EXCHANGE SESSION 4.X – ToR
This session covers a critical means of data collection that is inevitably part of every evaluation activity and should be part of most monitoring systems. This session helps strengthen observation as a more reliable and valid means of data collection.
The purpose of this session is to introduce and review the most common interview methods. It focuses on the range of interview methods, leaving aside the household questionnaire-based interview that is conducted in structured surveys. The session emphasises the particular value of interviews — as a means of obtaining information on the reasons behind behaviours, on different points of view, and on contextual and process issues. It compares structured and unstructured interviews, as well as other types: individual and key informant interviews, community interviews and focus group interviews. The session covers practical considerations on organising and conducting these interviews, and includes a role play so participants can practice individual interviewing techniques. Special facilitator notes and content sheets are provided to focus on different kinds of interviews (key informant, community interview and focus groups).
Surveys are widely used in UNICEF assessment, monitoring and evaluation activities. UNICEF’s primary reference for looking at them in greater detail is the End-decade Multiple Indicator Survey (MICS) manual, which was issued in February 2000 (see http://www.childinfo.org). However, for the most part, this session is more general in nature, providing a broad overview of key concepts and considerations so participants can understand what is involved in using and managing surveys. This session is divided into components that can be run independently. The key content is presented in the first part, which introduces the fundamental steps in managing a survey. It also discusses the types of questions that surveys can help answer, and how surveys can be used in both monitoring and evaluation activities. Detailed information is provided on several survey methodologies well-known in UNICEF and among many UNICEF partners: the EPI-like univariate survey, the multiple indicator survey (MICS), and USAID’s Demographic and Health Survey (DHS) and sentinel community surveillance (SCS). The second component explores the preparation and use of questionnaires through a practical exercise. Finally, a discussion on the advantages and disadvantages of using questionnaires in crisis and unstable contexts closes the session.
This session discusses setting up and maintaining information systems. Information systems can be used for a wide range of monitoring needs. If properly maintained, they can provide information on the evolution of the situation and trends. They are effective, but they can only present data related to a limited sample, that is to say the users/direct beneficiaries. Particular forms of information systems are those based on sentinel surveillance. Here only a limited number of representative sites are chosen and studied through a combination of techniques, including household surveys and qualitative techniques.
How many? Quantitative information is required in the early stages of a crisis to get at least a rough estimate of the size of the affected population. This session presents some rapid techniques for counting population in crisis and unstable contexts and their limitations: counting population, breakdown by age and gender, moving population and difficult situations.
The Rapid Assessment Process (RAP) is a combination of informal methodologies, such as semi-structured and unstructured observations, key informant interviews, focus groups and informal surveys. The main virtues of RAPs lie in how quickly they can be done and adapted to almost any situation. The main disadvantage is that those methodologies that imply interpretation of qualitative data and generally purposive sampling can be biased. The first part of this session introduces RAP, the core principles guiding them and their pros and cons. The second part illustrates in detail and through practical applications the range of techniques involved. The third part discusses how different techniques should be selected and can be combined in an assessment: to validate results and improve their accuracy and to prevent bias from creeping into the results.
The rapid field visit is a particular type of assessment aimed at quickly gathering basic information about areas affected by severe security or access constraints. It can also be used for monitoring. This sub-module indicates the steps to follow in a field visit and provides a checklist to guide information collection.
Should be dealt with here of in session 6??? (data analysis frameworks)
Sampling Nichols P 1991 Social Survey Methods: a field guide for development workers, Oxfam Development Guideline no. 6 50-72
Desk review Observation USAID Centre for Development Information and Evaluation, 1996. Performance Monitoring and Evaluation TIPS. Number 4. Using Direct Observation Techniques. Preparing and conducting interviews Patton, M. Q., (1982). Practical Evaluations. Sage Publications. Thousand Oaks, CA. pp.160-179. (14 pg.) This text is a summary of a more detailed 1980 publication by the same other “Qualitative Evaluation Methods”. While written from an evaluation perspective it is a good read for anyone managing or doing open interviews. It briefly restates the purpose of this type of interviewing, looks at three variations – informal conversational interview, interview guide approach and standardised open-ended interview – including their strengths and weakness, and then looks at the practical challenges of wording questions -- keeping questions open, clarity, neutrality, sensitivity, probing as well as how to give supportive response and maintain control of the interview. Roche, C. (1999) Impact Assessment for Development Agencies: Learning to Value Change, Oxford: Oxfam/Novib. Pg. 112-115 (4 pages) Texts. USAID Centre for Development Information and Evaluation, 1996. Performance Monitoring and Evaluation TIPS. No. 2. Conducting Key Informant Interviews USAID Centre for Development Information and Evaluation, 1996. Performance Monitoring and Evaluation TIPS. No. 10. Conducting Focus Group Interviews Surveys UNICEF (2000) Monitoring Progress Towards the Goals of the World Summit for Children: End-Decade Multi-indicator Cluster Survey Manual. UNICEF: New York. The full MICS Manual is available at www.childinfo.org. This is an excellent and detailed exposition of MICS surveys. Andersson (1996). Evidence based planning – The philosophy and methods of sentinel community surveillance. Ciet International and The Economic Development Institute of the World Bank Gosling, L. and Edwards, M. (2003) Toolkits: A practical guide to planning, monitoring, evaluation and impact assessment. Pp. 217-221 Nichols P 1991 Social Survey Methods: a field guide for development workers, Oxfam Development Guideline no. 6 pp. 33-49
Information systems Gosling, L. and Edwards, M. (2003) Toolkits: A practical guide to planning, monitoring, evaluation and impact assessment. pp 92-107
Population counts Brown V, Jacquier G, Coulombier D, Balandine S, Belanger F, Legros D (2001) “Rapid assessment of population size by area sampling in disaster situations”, Disasters, 25:2, p 164-171 MSF (1989). Mission exploratoire, mission d'evaluation. Situation sans deplacement de populations. MSF, Paris, France. Telford J (1997). Counting and Identification of Beneficiary Populations in Emergency operations: registration and its alternatives. ODI, London. Rapid Appraisal Process (RAP) Gosling, L. and Edwards, M. (2003) Toolkits: A practical guide to planning, monitoring, evaluation and impact assessment. Pp. 193-216 PRATT B, LOIZOS P (1992). Choosing Research Methods. Oxfam, Oxford. SLIM H (1992). The application of RAP and RRA techniques in emergency relief programmes. In: Scrimshaw N and Gleason G (eds), Rapid Assessment Procedures - Qualitative Methodologies for Planning and Evaluation of Health Related Programmes. INFDC Boston. Available on: http://www.unu.edu/unupress/food2/UIN08E/uin08e0r.htm Millelsen, Britha (1995). Methods for Development Work and Research. A guide for practitioners. Sage Publication New Delhi. PRA - chambers
http://www.ids.ac.uk/ids/particip/research/pra/rcwkshpjun99.pdf
RELAXED AND PARTICIPATORY APPRAISAL NOTES ON PRACTICAL APPROACHES AND METHODS
Chambers, R., 1985. "Short-Cut Methods in Social Information Gathering for Rural Development Projects," in Putting People First?Sociological Variables in Rural Development, New York, New York, Oxford University Press, p. 399-414. USAID Centre for Development Information and Evaluation, 1996. Performance Monitoring and Evaluation TIPS. Number 5. Using Rapid Appraisal Methods The in-and-out field visit Monitoring field visit
Case study
General
UNICEF, New York, 1991. A UNICEF Guide for Monitoring and Evaluation: Making a Difference? Evaluation Office, |