Guide to Data Collection Methods

Method
Archival Data

Appropriate Uses
Longitudinal observations, questions looking at trends

Strengths
Easy, cost-effective, credible Provides quantitative data Historical context of the program

Weaknesses
Difficulty in collecting multi-site data Can become expensive Poorly maintained databases can cause problems

Modified Archival Data

See archival data, with added benefit of having evaluation-specific data collected Natural setting Capture detailed descriptions of day-to-day Capture program-context behavior Capture unintended consequences Interaction between participants Greater honesty in responses/less influence of social desirability Obtain input from a large sample on Allows for more respondents to participate attitudes, goals or priorities Relatively efficient data collection Easy aggregation of data Provides data relatively quickly and Poll group as a way of collecting information inexpensively (faster results) on feelings on issues Exploration of unanticipated issues Provide insights about the effects, Insight into the respondents' ways of advantages, and limitations of a program thinking after implementation Allows for reaction and interaction between Initial step in survey research to generate participants topics, questions, and response options Corroborate & interpret survey findings Exploration of unanticipated issues Allows for obtaining real examples and stories Exploration of unanticipated issues Gather in-depth perspectives from individuals Insight into the respondents' ways of thinking Can address sensitive topics

Observations

Qualitative data can be difficult/timeconsuming to aggregate Hard to develop a sound, reliable instrument Data may be difficult to interpret if questions were not clear/straight-forward Issues with primacy effect Risk moderator bias - might give cues towards desirable responses Requires a trained moderator Issues with social desirability Limited generalizability of the results Can not address sensitive topics Qualitative data can be difficult/timeconsuming to aggregate Time-consuming and expensive Qualitative data can be difficult/timeconsuming to aggregate Issues with social desirability Limited generalizability of the results Issues with recency effect

Surveys/Questionnaires

Focus Groups

Interviews

QUICK TIP #1: If you design a survey with many open-ended questions, consider using interviews or a focus group instead - Select between interviews and focus groups depending on your timeline for data collection and analysis, the nature of questions/sensitivity of topics, and the need for in-depth case studies or consensus opinions QUICK TIP #2: When possible, utilize multiple data collection methods (mixed method studies) or longitudinal data to triangulate your results across data points Lisa O'Leary Shuler Teaching & Learning Laboratory, MIT 7/15/2009