Research Methods & Design in Psychology

Lecture 2

Survey Design 2
Lecturer: James Neill

Overview
Survey construction - nuts & nolts Sampling Ethics Levels of measurement Measurement error

What is a Survey?
A standardised stimulus A measuring instrument A way of converting fuzzy psychological stuff into hard data for analysis

Survey Construction Nuts & Bolts

Constructing questions Modes of response Response formats <-> LOM Measurement error Survey formatting

Constructing questions
Define target constructs Check related research & questionnaires Draft items (aim to have multiple indicators) Pre-test & revise

When drafting questions aim to:

Focus directly on topic/issue Be clear Be brief Avoid big words Use simple and correct grammar

Bias in questions
Inapplicable Over-demanding Ambiguous Double negatives Double-barrelled Leading Loaded

Bias in responding
Social desirability Acquiescence or Yea- and Nay-saying Self-serving bias Order effects

Modes of Survey Administration

Interview
high demand characteristics can elicit more information

Questionnaire
lower demand characteristics information may be less rich

Objective vs. subjective

Objective: How times during 2000 did you visit a G.P.? Subjective: Think about the visits you made to a G.P. during 2000. How well did you understand the medical advice you received?
perfectly very well reasonably poorly not at all

Open-ended vs. close-ended

Open-ended
rich information can be gathered useful for descriptive, exploratory work difficult and subjective to analyse, time consuming important information may be lost forever useful for hypothesis testing easy and objective to analyse time efficient

Close-ended

Open-ended questions Examples

What are the main issues you are currently facing in your life? How many hours did you spend studying this week? _________

Close-ended questions Example 1

What are the main issues you are currently facing in your life? (please all that apply)
financial physical/health academic employment/unemployment intimate relations social relations other (please specify) ________________________________

Close-ended questions Example 2

How many hours did you spend studying this week?
than 5 hours less 5 to 10 hours > 10 to 20 hours > more than 20 hours

Close-ended rating scales

Likert scale Graphic rating scale Semantic differential scale Non-verbal scale Frequency scale

Likert Scale
Pick a number from the scale to show how much you agree or disagree with each statement:
1
strongly disagree

2
disagree

3
neutral

4
agree

5
strongly agree

1
strongly agree

2
agree

3
neutral

4
disagree

5
strongly disagree

Graphic Rating Scale

How would you rate your enjoyment of the movie you just saw? Mark with a cross (X)
not enjoyable very enjoyable

Semantic Differential Scale

What is your view of smoking? Tick to show your opinion.
Bad ___:___:___:___:___:___:___ Strong ___:___:___:___:___:___:___ Masculine ___:___:___:___:___:___:___ Unattractive ___:___:___:___:___:___:___ Passive ___:___:___:___:___:___:___ Good Weak Feminine Attractive Active

Non-verbal Scale
Point to the face that shows how you feel about what happened to the toy.

Verbal Frequency Scale

Over the past month, how often have you argued with your intimate partner?
1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesnt apply to me at the moment

Sensitivity & Reliability

Scale should be sensitive yet reliable. Watch out for too few or too many options

Scale of measurement guidelines

General aim: Maximise sensitivity Maximise reliability

(i.e. more options) (i.e. less options)

How many measurement options? Minimum =2 Average = 3 to 7 Maximum = 10?

FEELING ABOUT SOMETHING

EXTREMELY POSITIVE
GOOD

EXTREMELY NEGATIVE 2-Categories

NOT GOOD

3-Categories
GOOD FAIR POOR

4-Categories
VERY GOOD GOOD FAIR POOR

5-Categories
EXCELLENT VERY GOOD GOOD FAIR POOR

Watch out for too many or too few responses

Capital punishment should be reintroduced for serious crimes

1 = Agree

2 = Disagree

1 = Very, Very Strongly Agree 7 = Slightly Disagree 2 = Very Strongly Agree 8 = Disagree 3 = Strongly Agree 9 = Strongly Disagree 4 = Agree 10 = V. Strongly Disagree 5 = Slightly Agree 11 = V, V Strongly Disagree 6 = Neutral

Sampling
Sampling Terminology What is Sampling? Sampling Techniques Example: Shere Hites Sex Survey Summary of Sampling Strategy

Sampling Terminology
Population Sampling Frame Sample Representativeness

What is sampling?
Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen. - Trochim, 2002

Sampling Techniques
Probability sampling
Random Systematic Cluster
Multi-Stage Cluster

Non-probability sampling
Quota Convenience Snowball

Representativeness of sample depends on:

adequacy of sampling frame selection strategy adequacy of sample size response rate both the % & representativeness of people in sample who actually complete survey Note: It is better to have a small, good sample than a large, poor sample.

Sampling Example:
Shere Hite American Sexology

Male-Female Relations
Shere Hite doyenne of sex polls Media furors & worldwide attention 127-item questionnaire about marriage & relations between sexes 4500 USA women, 14 to 85 years Society and men need to change to improve lives of women

Some of Hites findings....

70% married for 5 years having affairs...
(usually more for emotional closeness than sex)

76% did not feel guilty 87% had a closer female friend than husband 98% wanted basic changes to love relationships only 13% married for 2+years were still in love 84% were emotionally unsatisfied 95% reported emotional & psychological harassment from their men

Some of the critical comments....

She goes in with prejudice & comes out with a statistic. The survey often seems merely to provide an occasion for the authors own male-bashing diatribes. Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men.

Response rate & Selection bias - 1

100,000 questionnaires
Sent to a variety of womens groups - feminist organisations, church groups, garden clubs, etc.

4,500 replied
(4.5% return rate)

Response rate & Selection bias - 2

We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire. - Regina Herzog, University of Michigan Institute for Social
Research

Summary of sampling strategy

Identify target population and sampling frame Selection sampling method Calculate power and required sample size Maximise return rate

Survey Format Checklist

Introduction/covering letter or verbal introducation
e.g. Who are you? Are you bona fide? Purpose of survey? Ethical approval? How results will be used? Confidentiality? Further info? Complaints?

Instructions
Sets the mind frame, but be aware few people will read it without good prompting and being easy-to-read

Group like questions together Consider order effects, habituation, fatigue, switching between response formats

Survey Format
Font type / size, number of pages, margins, double vs. single-siding, colour, etc. Demographics - single section, usually at beginning or end of questionnaire, only use relevant questions Space for comments? Ending the questionnaire say thanks! Pre-test the questionnaire & revise/refine

Pre-test & Revise

Pre-test items and ask for feedback Revise:

items which dont apply to everybody redundancy skewed response items misinterpreted items non-completed items

Reconsider ordering & layout

Ethical issues: How to treat respondents

Minimise risk/harm to respondents Informed consent Confidentiality / anonymity No coercion Minimal deceit Fully debrief

Other ethical issues

Honour promises to provide respondents with research reports Be aware of potential sources of bias/ conflicts of interest Represent research literature fairly Dont search data for pleasing findings Acknowledge all sources Dont fake (or unfairly manipulate) data Honestly report research findings

Levels of measurement = type of data

Levels of Measurement = Type of Data

4 levels of measurement
Nominal/Category Ordinal Interval Ratio

Levels of measurement discrete vs. continuous

Categorical / Nominal Ordinal / Rank Interval Ratio

(Discrete) (Discrete) (Discrete?) (Continuous)

Each level has the properties of the preceeding levels, plus something more!

Categorical / Nomimal
Arbitrary assignment of #s to categories e.g. male = 1, female = 2 No useful information, except as labels

Ordinal /Ranked Scales

#s convey order, but not distance e.g. in a race, 1st, 2nd, 3rd, etc. Often must be treated as categorical

Interval Scales
#s convey order & distance, 0 is arbitrary e.g. temperature (degrees C) Usually treat as continuous for >5 intervals

Ratio Scales
#s convey order & distance, meaningful 0 e.g. height, age ratios - e.g. 2 x old, 3 x high

Why do levels of measurement matter? different analytical procedures are used for different levels of data
More powerful statistics can be applied to higher levels

Measurement scales -> Analysis

categorical & nominal -> non-parametric

interval & ratio -> parametric

What are parametric stats?

= procedures which estimate PARAMETERS of a population, usually based on the normal distribution any procedure which uses M, SD

e.g. t-tests, ANOVAs

any procedure which uses r

e.g. bivariate correlation, linear regression

Practice Exam Question

Level
Nomimal /Categorical Ordinal / Rank Interval Ratio

Properties Examples Descriptive Graphs Statistics

What are non-parametric stats?

(Distribution-free Tests)

= procedures which do not rely on estimates

of population PARAMETERS any procedure which uses frequency

e.g. sign test, chi-squared

any procedure which uses rank order

e.g. Mann-Whitney U test, Wilcoxon matchedpairs signed-ranks test

Parametric vs. non-parametric stats?

parametric statistics are more powerful but are also more sensitive to violations of assumptions

Measurement error
Observed score = true score + measurement error = true score + systematic error + random error Measurement error is any deviation from the true value.

Sources of Error

Non-sampling

Sampling

Paradigm

Personal

Sources of measurement error

Paradigm Personal researcher bias Sampling Non-sampling

To minimise measurement error

Use well designed measures Reduce demand effects Maximise response rate Ensure administrative accuracy

Summary
Survey construction - nuts & bolts Sampling Ethics Levels of measurement Measurement error

Respond to UCs student survey and win an iPod Video valued at $380
Features: 30GB, music videos, home movies and video podcasts, new iPod games, audiobooks, photo albums and, of course, an entire library of up to 20,000 songs.

Logon to OSIS before 14 March, fill in the student survey & youll enter the draw!