Professional Documents
Culture Documents
Foddy Cambridge University Press, 1993 written by Walter Wood, M.Eng. Telecommunications Technology Management 2000
Chap 1 An initial statement of the problem .............................................................................. 3 principal causes of error in the gathering of data through survey procedures: ........................ 3 10 examples of problems illustrating the inadequacy of many of the questions that have been used in social research in the past: ........................................................................................... 3 Chap 2 A theoretical framework............................................................................................... 3 two broad approaches to collecting verbal data: ...................................................................... 3 principal assumptions that have defined the general orientation adopted by survey researchers in the past: ............................................................................................................. 3 4 steps in a successful Q-A sequence (The Q-A Comm Cycle):.......................................... 4 5 sources of response variability:............................................................................................. 4 Chap 3 Defining topics properly............................................................................................... 5 The researcher has clearly defined the required information (Assumption 1)......................... 5 Respondents have the required information (Assumption 2)................................................... 5 Respondents are capable of verbalizing the information the researcher wants under the conditions of the research situation (Assumption 3)................................................................ 5 Chap 4 Formulating intelligible requests for information......................................................... 5 The meanings of individual words........................................................................................... 5 Structural complexities and the requirement that questions should be interpreted as intended ................................................................................................................................................. 6 Chap 5 Contextual influences on respondents interpretations of questions ............................ 6 Clues afforded by either the question itself or its components ................................................ 6 Clues contained in accompanying sets of response options..................................................... 6 The impact of preceding questions (& answers)...................................................................... 6 The impact of previous answers on the respondent ................................................................. 7 The impact of the definition of the overall situation................................................................ 7 Chap 6 The need to provide response frameworks ................................................................... 7 Descriptive accounts ................................................................................................................ 7 Levels of social generality of responses .................................................................................. 7 Explanations............................................................................................................................. 8 Evaluations............................................................................................................................... 8 Chap 7 The limitations of human memory ............................................................................... 9 Problems associated with the recall of information in long-term memory .............................. 9
Problems associated with short-term memory ....................................................................... 10 Chap 8 Filters: establishing the relevance of questions to respondents .................................. 10 What kind of no opinion filter should be used? .................................................................. 10 The positioning of filters........................................................................................................ 11 Should a middle category be offered at all?........................................................................... 11 Chap 9 Reducing question threat ............................................................................................ 11 The definition of question threat............................................................................................ 11 Chap 10 The open vs closed questions debate........................................................................ 12 The principal advantages and disadvantages associated with the two formats...................... 12 An evaluation of the assumptions underlying the use of open questions .............................. 13 An evaluation of the assumptions underlying the use of closed questions ............................ 14 Problems associated with developing response options for closed questions........................ 14 Problems associated with recording responses to closed questions....................................... 14 Problems associated with interpreting responses to closed questions ................................... 14 An evaluation of three additional uses claimed for open questions....................................... 14 Chap 11 Measuring attitudes .................................................................................................. 15 Single rating scales ................................................................................................................ 15 Problems associated with batteries of rating scales (i.e. summated scales)........................... 16 A possible alternative to summated scales............................................................................. 17 Chap 12 Checks to ensure that questions work as intended.................................................... 17 Editting rules.......................................................................................................................... 17 Piloting questions................................................................................................................... 18 Question testing ..................................................................................................................... 18 Chap 13 Concluding comments .............................................................................................. 18 The TAP paradigm for constructing questions .......................................................................... 18 Tap ......................................................................................................................................... 18 Applicability .......................................................................................................................... 18 Perspective ............................................................................................................................. 18
Chap 1 An initial statement of the problem principal causes of error in the gathering of data through survey procedures:
respondents failure to understand questions as intended a lack of effort, or interest, on the part of respondents respondents unwillingness to admit to certain attitudes or behaviours the failure of respondents memory or comprehension processes in the stressed conditions of the interview interviewer failures of various kinds (e.g. the tendency to change wording, failures in presentation procedures and the adoption of faulty recording procedures)
10 examples of problems illustrating the inadequacy of many of the questions that have been used in social research in the past:
1) factual questions sometimes elicit invalid answers 2) the relationship between what respondents say they do and what the actually do is not always very strong 3) respondents attitudes, beliefs, opinions, habits, interests often seem to be extraordinarily unstable 4) small changes in wording sometimes produce major changes in the distribution of responses 5) respondents commonly misinterpret questions 6) answers to earlier questions can affect respondents answers to later questions 7) changing the order in which response options are presented sometimes affects respondents answers 8) respondents answers are sometimes affected by the question format per se 9) respondents often answer questions even when it appears that they know very little about the topic 10) the cultural context in which a question is presented often has an impact on the way respondents interpret and answer questions
principal assumptions that have defined the general orientation adopted by survey researchers in the past:
1) researcher has clearly defined the topic about which information is required 2) respondents have the information that the researcher requires 3) respondents are able to access the required information under the conditions of the research situation 4) respondents can understand each question as the researcher intends it to be understood 5) respondents are willing (or at least can be motivated) to give the required information to the researcher 6) the answers that respondents give to a particular question are more valid if they have not been told why the researcher is asking the question (** refuted below **)
7) the answers that respondents give to a particular question are more valid if the researcher has not suggested them to the respondents 8) the research situation per se does not influence the nature of the answers given by respondents 9) the process of answering questions per se does not change the respondents beliefs, opinions, habits, etc 10) the answers that different repondents give to a particular question can be meaningfully compared with one another
2) because any topic is multidimensional, any topic can be defined in terms of either a single dimension or a combination of dimensions 3) utterances (including Qs and As) can be made on different levels of generality 4) utterances about a topic can also be made on a number of theoretical levels 5) utterances are always framed within a descriptive, explanatory or evaluative frame of reference
Chap 3 Defining topics properly The researcher has clearly defined the required information (Assumption 1)
1) the researcher has clearly defined the topic at least 3 ways the multidimensional nature of any topic bears on Q-A behaviour: a) respondents answers in terms of specific dimensions will not always be congruent with their global judgements b) when respondents are required to answer in terms of specific dimensions, the dimensions that they are to focus on must be properly defined c) how a topic is defined can dramatically affect the way the responses are distributed 2) the researcher has a clear idea of the kind of information that is required about the topic 4 rules to ensure that the info collected satisfies the reason for conducting the research a) do not formulate specific questions until you have thought through the research question b) write the research question down and keep it handy while formulating specific questions c) keep asking why do I want to know this? d) prepare dummy tables showing the relationships that are anticipated write the avowed purpose beside each question in the first drafts of questionnaires or interview schedules (i.e. to write beside each question what information they think it will elict and what they will do with that information)
Respondents are capable of verbalizing the information the researcher wants under the conditions of the research situation (Assumption 3)
common pitfalls: proceeding too fast; failure to stimulate recall; questions too complex; questions too long
Chap 4 Formulating intelligible requests for information The meanings of individual words
4 factors in failure of respondents to give the same meaning to individual words: 1) the evolution of context-specific nuances of meaning cultural context, use of slang, choice of words to stress 2) their relative difficulty (e.g. number of syllables, frequency in common use) keep words/questions simple 3) lack of clear empirical referents 5
abstract words lacking clear empirical referents: frequency terms regularly, usually, often, frequently, seldom, rarely evaluative terms good, bad, approve, disapprove, agree, disagree, like, dislike the words used by the researcher when formulating questions should be as specific and as concrete as possible defined in empirical terms: frequency questions should be worded in terms of numeric referents per specified time period clarify concepts with short vignettes (word pictures) 4) the operation of related nuances of apparently similar words e.g. negative response options are perceived as being stronger than the countervailing positive response options
Structural complexities and the requirement that questions should be interpreted as intended
sheer number of words used more words = greater likelihood that the question will be interpreted incorrectly grammatical complexities avoid asking two or more questions at once avoid adding qualifying clauses, phrases, and instructions avoid the use of negatives
quintamensional plan: i. a general question to establish whether respondents have the required information concerning the topic ii. an open question to get at respondents general perceptions or feelings abut the topic iii. a dichotomous question to elicit perceptions or feelings about a specific aspect of the topic iv. an open question to get at reasons for responses toward the aspect of the topic specified in step 3 v. a rating question to allow respondents to indicate the strength of their responses towatd the aspect of the topic specified in step 3 the funnel sequence motivating full communication
b) priming / anti-priming (or discounting) sequence 2) the psychological need to be consistent convergence of attitude concerning a particular topic 3) even-handedness
Descriptive accounts
respondents can look at any situation and describe it in many different ways if respondents adopt different perspectives when answering a question, it can be argued that they are not answering the same question; if this happens, the answers that they give cannot be meaningfully compared the researcher can assume the responsibility of clearly specifying the perspective that respondents should have in mind
the word you is prone to being interpreted collectively when used in conjunction with behaviour which the respondent does with others other problematic/ambiguous pronouns: one & who the researcher must be clear/specific about the level of social generality
Explanations
respondents can give any number of explanations for the same behaviour 5 different kinds of answers to the question: why did you do X? (a non-exhaustive list) 1) causal antecedents what caused the respondent to do X? 2) goal antecedents the respondents purpose for doing X? 3) enablement factors how it was possible for the respondent to do X? 4) causal consequences what happened after the respondent had done X? 5) the researchers expectations why the respondent had done X when the researcher had expected the respondent to do Y?
Evaluations
evaluative judgements are always relative; they are never absolute statements; evaluations always involve comparisons with standards
1) evaluative standards that are external to the question personal answers are likely to be affected by reference group effects when: they have been evoked by previous questions the researcher has neglected to clearly specify the standards that should be used reference group effect dead give-away words (all, always, each, every, never, nobody, only, none, etc) the problem with dead give-away words is that people learn that there are exceptions to most rules and consequently avoid selecting answers that include them 2) evaluative standards that are internal to the question usually either implicit or poorly defined the common rating scale epitomises the failure to use properly defined standards of comparison in social research lack consensual meaning simply because they lack empirical referents the implied evaluative standards of many rating tasks are often biased or imbalanced
researchers should get as familiar as possible with the events that they are investigating so that they can refer to key events in the same terms that the respoondents would have used when they were encoded get respondents to work in chronological order either forwards or backwards (usually better to have them work backwards because of recency effects) the researcher should ask a series of cross-cutting questions to provide multiple frames of reference or recall cues 3 ways to encourage more accurate recall: 1) respondents explicitly instructed to recall accurately and told to take their time in answering 2) respondents given encouragement 3) get an explicit agreement to respond accurately and completely
10
1) question threat generated by factors that are idiosyncratic to particular respondents might generate purely personal fears or feelings of guilt 2) characteristics of the questions per se that threaten respondents difficult vocabulary excessive question length grammatical complexity difficult instructions key words that have negative stereotypic or cultural connotations 3) threat associated with the nature of the relationship which exists between the researcher and the respondent a) fears of negative sanctions being either: socially rejected or thought less of by the researcher respondents will be under psychological pressure to give answers that they think are normatively acceptable to the interviewer when they perceive the interviewer to be socially similar to themselves strategies to reduce fears of social rejection: increase the psychological distance between the interviewers and respondents ensuring the respondents answers are anonymous ensuring that the questions do not contain clues about what the researcher thinks is normal b) materially sanctioned or physically punished in some way
11
inteviewers who have higher social status than their respondents can pose either a political or an economic threat for the respondents strategies: procedures that maximise trust on the part of the respondents anything that researchers can do to indicate a lack of naivety on their part may help to override the effects of the respondents fears stressing the fact that the respondents views are too important to be incorrectly recorded
4) a complicating fourth factor: the respondents definition of the situation any topic might generate two or even all three types of threat the researcher can decide whether to concentrate on the problem of minimising the threat that is most likely to be dominant in the situation or employ a number of threat-reducing strategies at the same time Types of questions Questions that concern normative issues, e.g.: matters of hygiene, social morality, social responsibility, aesthetic judgement, group loyalty Questions that concern either political or economic interests Types of threat generated Fear of being socially rejected by the interviewer Appropriate threat-reducing strategies allow respondents to answer anonymously decrease the likelihood that respondents will see the interviewer as a social peer use inteviewing procedures that increase either the physical or psychological distance between the respondents and the inteviewer increase respondents level of trust stress confidentiality establish lack of inteviewer gullibility emphasize social significance of respondents answers
Fear that the interviewer will impose either economic or political sanctions on the respondent
Chap 10 The open vs closed questions debate The principal advantages and disadvantages associated with the two formats
Open questions a) allow respondents to express themselves in their own words b) do not suggest answers indicate respondents level of information indicate what is salient in the respondents mind indicate strength of respondents feelings c) avoid format effects d) allow complex motivational influences and frames of reference to be identified Closed questions a) allow respondents to answer the same question so that answers can be meaningfully compared b) produce less variable answers c) present a recognition, as opposed to a recall, task to respondents and for this reason respondents find them much easier to answer d) produce answers that are much easier to computerize and analyze
12
e) are a necessary prerequisite for the proper development of sets of response options for closed questions f) aid in the interpretation of deviant responses to closed questions * Note: there are problems associated with each of these as pointed out in the following sections
4) open questions avoid the format effects that have been associated with closed questions little basis for this assertion 5) answers to open questions allow complex motivational influences and frames of reference to be identified an answer to an open question indicates the way in which a respondent has intepreted it good reasons for rejecting this hypothesis an answer to an open question indicates the motivation(s) that have influenced the respondents orientation to the topic good reasons for rejecting this hypothesis the answer to an open question indicates the frame of reference used by the respondent theoretically nonsensical 6) problems associated with probing inadequate answers to open questions a) interviewers should clearly understand the researchersgoals so that they can probe respondents minds to clarify obscure answers ( crucial that probes be non-directive) b) interviewers should know the required sorts of answers c) interviewers should attempt to get adequate answers 13
7) problems associated with coding responses to open questions a question should be put to respondents in such a way that it elicits complete answers of the kind required in the absence of clear guidelines about what sorts of answers are required, respondents can, and will, focus upon such a variety of aspects of the topic that their answers are neither comparable nor codable
14
2) open questions are useful when a set of meaningful alternatives is too large and complex to present to respondents 3) it may be necessary to use open questions when the researcher has reason to suspect that rapidly shifting external events will affect answers
15
respondents are less influenced by intermediate or middle labels than they are by the labels in the end positions labelling of all categories produces data of poorer quality than labelling only the end categories; if poorly defined categories generate confusions, the more poorly defined categories there are, the greater the number of confusions that will occur
b) respondent centred effects rating scales suffer from a number of respondent-centred biases, such as the tendency for some respondents to choose central response options rather that extreme ones, to agree rather than to disagree, and to be positive rather than negative respondent-centred response biases are inversely dependent upon the degree to which the topic and the categories have been properly defined, and directly influenced by the number of categories
or, include labelled end categories and an appended measure either of the importance of the topic or of the respondents sureness as a substitute for the middle categories
3. be attached to statements that are of the same level of generality as the behaviour that is to be predicted c) Make responses meaningfully comparable by: 1. explicitly instructing respondents that they are to make respondent centred responses 2. allowing respondents to look over the whole range of items before responding to each one
16
Editting rules
standard methodological advice in the past has been to formulate questions in accordance with a list of principles of good practice which have been codified from past experience, and then to pilot or trial these questions on a small sample of respondents (i.e. twenty to thirty) drawn from the target population
Editting rules to aid the formulation of questions a) Make sure that the topic has been clearly defined. b) Be clear both about the information that is required about the topic and the reason for wanting this information. c) Make sure that the topic has been defined properly for the respondents by: avoiding the use of 'blab' words (i.e. words that are so abstract or general that they lack specific empirical referents) avoiding words that are unlikely to be understood by all respondents either because they are rarely used in everyday life, or are specialist (i.e. jargon) words d) Make sure that the question is relevant to respondents by: using an appropriate filter avoiding asking for information respondents are likely to have forgotten avoiding hypothetical issues e) Make sure that the question is not biased by: ensuring balance in the introduction to the question (e.g. some people like X, and some people dislike X. - Do you like X or dislike X?) ensuring that sets of response options are complete ensuring that sets of response options are balanced avoiding using words that are likely to invoke stereotypical reactions
17
f) Eliminate complexities that prevent respondents from easily assimilating the meaning of the question by: avoiding asking 2 or more questions at once avoiding the use of words that have several meanings checking whether the question has been worded as simply as possible avoiding the use of too many 'meaningful' words in the one question avoiding the use of qualifying clauses and phrases and the addition of complicating instructions which cause respondents to start to answer, before they have been exposed to the whole question if qualifying clauses and phrases have to be used, they should be placed at the beginning rather than at the end of the question making sure that the question is as short as possible avoiding the use of both negatives and double negatives g) Ensure that respondents understand what kind of answer is required by: setting the question in context informing respondents why the question is being asked informing respondents what will be done with the information they give specifying the perspective that respondents should adopt
Piloting questions
more useful for uncovering aspects of questions that will cause interviewers to have difficulty than for discovering whether or not the respondents interpret the questions as intended interviewers should work in pairs so that one can concentrate fully on conducting the interview while the other is free to listen to and observe both the interviewers and the respondents behaviour
Question testing
three techniques (these usually limit attention to the in-depth testing of no more than six or seven questions at a time): 1) asking respondents to rephrase questions in their own words 2) the double interview 3) asking respondents to think aloud as they answer each question
Chap 13 Concluding comments The TAP paradigm for constructing questions Tap
properly define questions so that each respondent clearly understands what is being talked about
Applicability
establish the applicability of the question to each respondent to ensure they have the information being asked
Perspective
specify the perspective that respondents should adopt when answering the question so that each respondent gives the same kind of answer 18