OBJECTIVES

• Understand traditional criteria (reliability, validity, objectivity) and alternative criteria for judging data collection and analysis methods, and situate values underlying each.
• Identify factors introducing bias in data collection activities, design, implementation and analysis.
• Practice determining the suitability of data collection methods in different contexts.

OUTLINE

This part covers a range of criteria used in judging research methods, an initial analysis of the advantages and disadvantages in different data collection methods and then turns to the wider range of considerations that will come into play in actually making such choices.

5.1.1

CRITERIA FOR JUDGING METHODS

This session seeks to clarify and contrast different criteria - traditional 'scientific' and alternatives oriented to qualitative and participatory data collection - used in judging data collection methods.

FN	Criteria for judging methods This session uses a mix of PowerPoint slides, discussion, participants' examples, and exercises to clarify the more commonly used criteria for judging research methods - reliability and validity, objectivity vs. subjectivity, and bias - and the alternative criteria for qualitative data collection and analysis and for participatory research.
CCS	Reliability and validity Objectivity and subjectivity Bias Criteria for judging methods
OH	TO DO: Reliability and validity TO DO: Objectivity and subjectivity TO DO: Criteria for judging methods
EX	Quiz on reliability and validity Quotes on objectivity and subjectivity

REFERENCE MATERIALS

Millelsen, Britha (1995). Methods for Development Work and Research. A guide for practitioners. Sage Publication New Delhi.

Patton, M., 1977. Utilisation-Focused Evaluation, 3rd ed., Thousand Oaks: Sage Publications. p. 241-264
This excerpt makes the case for involving primary users in method selection, demystifying judgements on methodological design, including concepts of validity and reliability, in the context of practical constraints and tradeoffs. The reading is focused on evaluation but equally valid for major assessments and operational research.

PRATT B, LOIZOS P (1992). Choosing Research Methods. Oxfam, Oxford.

Trochim, William M. The Research Methods Knowledge Base, 2nd Edition. http://trochim.human.cornell.edu/kb/intreval.htm . (version current as of 2001)

5.1.2

CHOOSING DATA COLLECTION METHODS

This session illustrates how, when selecting the most appropriate data collection method(s) for M&E, it is important to achieve the most suitable trade-off among often contrasting factors. A possible solution could be to chose not "the best method", but "the best combination of methods".

FN	Choosing data collection methods This session begins with a group work and plenary discussion on the advantages and disadvantages of different data collection methods, including consideration of context and use. In a subsequent group exercise, participants discuss information needs and possible ways to gather data for a given case study. The results of the group work will be fed as examples in a presentation on factors guiding the choice of data collection methods and on the advantages of combining different data collection methods.
CCS	Choosing data collection methods Overview of data gathering methods Quantitative and qualitative methods: compared and mixed
OH	-
EX	Exercise: choosing methods - evaluation of school dropout reduction project Exercise: choosing methods - evaluation of project for street children Exercise: choosing methods - assessment of population affected by natural disaster Exercise: choosing methods - assessment of internally displaced population