The document discusses the problem of ecological validity in interviews and surveys. It argues that while sociologists widely use interviews and surveys, there is little theoretical foundation for how these tools work. It suggests that linguistic and cognitive processes should inform our understanding of how questions are comprehended and answered in artificial research contexts, which often differ from natural settings. The paper aims to outline some cognitive and linguistic issues that can provide a stronger theoretical basis for interviews and surveys and improve their ecological validity.
Original Description:
Original Title
S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf
The document discusses the problem of ecological validity in interviews and surveys. It argues that while sociologists widely use interviews and surveys, there is little theoretical foundation for how these tools work. It suggests that linguistic and cognitive processes should inform our understanding of how questions are comprehended and answered in artificial research contexts, which often differ from natural settings. The paper aims to outline some cognitive and linguistic issues that can provide a stronger theoretical basis for interviews and surveys and improve their ecological validity.
The document discusses the problem of ecological validity in interviews and surveys. It argues that while sociologists widely use interviews and surveys, there is little theoretical foundation for how these tools work. It suggests that linguistic and cognitive processes should inform our understanding of how questions are comprehended and answered in artificial research contexts, which often differ from natural settings. The paper aims to outline some cognitive and linguistic issues that can provide a stronger theoretical basis for interviews and surveys and improve their ecological validity.
Interviews, Surveys, and the Problem of Ecological Validity
Author(s): Aaron V. Cicourel
Source: The American Sociologist, Vol. 17, No. 1 (Feb., 1982), pp. 11-20 Published by: American Sociological Association Stable URL: http://www.jstor.org/stable/27702491 . Accessed: 28/10/2013 15:50 Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp . JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org. . American Sociological Association is collaborating with JSTOR to digitize, preserve and extend access to The American Sociologist. http://www.jstor.org This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions The Problem of Ecological Validity 11 tions that a classic can serve. We may believe that students' minds are expanded by reading Durkheim without our having to believe Durkheim has many true gener alizations about the causes of suicide. George Herbert Mead can symbolize what is distinctive in symbolic interactionism even if we cannot quite figure out how to test the hypothesis of the independence of the "F from the "me," and to turn it into a puzzle for routine science. And orte can enjoy the taste of Marx's famous passage in The 18th Brumaire about French peas ants forming a vast mass, without that beauty being undermined when we find some regions of modern France where the peasants vote Communist. What is destructive about admiration of the classics, then, is the halo effect, the belief that because a book or article is useful for one purpose, it must have all the virtues. INTERVIEWS, SURVEYS, AND THE PROBLEM OF ECOLOGICAL VALIDITY* Aaron V. Cicourel University of California, San Diego The American Sociologist 1982, Vol. 17 (February): 11-20 Despite the fact that virtually all social science data are derived from some kind of discourse or textual materials, sociologists have devoted little time to establishing explicit theoretical foundations for the use of such instruments as interviews and surveys. A key problem always has been the lack of clear theoretical concepts about the interpretation of interview and survey question and answer frames. We lack a theory of comprehension and communication that can provide a foundation for the way that question-answer systems function, and the way respondents understand them. The paper briefly describes the possible relevance of linguistic and cognitive processes for improving our understanding of interviews and surveys. The theoretical foundations of interviews and surveys also must address the way that artificial circumstances become necessary to guarantee adequate study designs. These artificial circumstances often violate ecological validity, or the way interviews and survey questions are constructed, understood, and answered, as contrasted with the way that field notes and tape-recordings of natural settings are used to address the same or comparable substantive and theoretical issues. Social scientists have relied on inter views for a long time. There is little reason to doubt their value and utility for many theoretical and practical purposes. There exists a huge literature on the virtues and drawbacks of interviews that use open ended questions and surveys that use close-ended questions. Yet there is little in the way of theory that would link inter * Presented at the thematic section "Fact or Ar tifact: Are Surveys Worth Anything?" held at the 1980 American Sociological Association Meetings, New York, August 27, 1980. The other speaker was Howard Schuman, taking a less critical view of sur vey research. I am grateful to Michael Cole, Roy D'Andrade, and Hugh Mehan for their valuable re marks and suggestions on a much longer first draft of the paper. [Address correspondence to: Aaron V. Cicourel; Department of Sociology; University of California, San Diego; La Jolla CA 92037.] views and surveys to more general issues of communication and comprehension. Those researchers who are convinced that interviews and surveys are basic research tools for the sociologists are concerned about improvements in survey design and use, but see little point in challenging their routine use. In this paper I want to suggest a few cognitive and linguistic issues that can clarify our understanding of the pro cesses and mechanisms underlying the use of interviews and surveys. I also want to suggest some theoretical ideas that can strengthen the ecological validity of inter view and survey methods and findings. The necessity of writing a brief paper does not permit me to discuss old issues about current interview and survey prac tices that I hope are obvious to This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions 12 The American Sociologist sociologists: for example, the way that preliminary qualitative interviews nor mally precede the construction of inter view schedules and surveys; and the way that pre-testing with in-depth interviews helps to create questions that respondents can understand, while helping to create answer categories that reflect the thinking of respondents and not simply the re searcher. Hence I will avoid discussing the range of practices that are necessary to insure quality control, such as using different types of questions on the same topic, exploring the significance of changes in wording, and other procedures too numerous to mention here. Interviews and surveys usually occur within a broader context of interaction that includes complex cognitive and lin guistic activities within a set of in stitutionalized and emergent socio cultural constraints. The questions used in surveys almost always are framed in a textual format that displays features in common with I.Q., aptitude, and reading tests. Virtually all social science data are de rived from some kind of discourse or tex tual materials such as reports, written ac counts of observations, interviews, audio or video recordings of natural settings, historical or contemporary documents, minutes of meetings, newspapers, maga zines, and the like. Questionnaires mailed to respondents presuppose something about the way people are able to analyze textual materials. For example, in re sponding to questions in a reading test, the respondent must utilize several sources of knowledge that the researcher used in interviews and surveys, therefore, pre sumes a theory of communication and comprehension that seldom is addressed by sociologists. The remainder of the paper will suggest aspects of communica tion and comprehension that can help es tablish some theoretical foundations for interviews and survey research. Aspects of memory and comprehension presupposed in surveys and interviews Recent work on learning and reading comprehension (Bransford et al., n.d.) reminds us that our research instruments stringently control the information re sources available to the subject or respon dent. In tests and questionnaires "normal procedures" presuppose an agreement or social contract between researcher and respondent: the contract does not permit the use of other persons (nor the inter viewer) in order to decide on the meaning of the question and the appropriateness of a response. Normal group or peer sources of help are blocked. The test or question naire item is assumed to be self explanatory or self-contained. These con ditions contrast with the possibility of consulting a friend or colleague or return ing to a textual report, newspaper article, book, and the like, during or after an ini tial reading of the text. The interview and survey seek to re strict the question frame, and in the case of surveys, the answer frame. The goal is to restrict the question in such a way as to anticipate and even designate (in surveys) the range of responses that can be used. The aggregation of responses requires a few choices that either are formally part of the questionnaire or are imposed on open-ended responses. We impose infor mation processing restrictions on the re spondent because they enable us to aggre gate and summarize a large amount of in formation in a fairly succinct way. But we pay a price, and we need to understand the costs in order to improve the reliability and validity of interview and survey data. We need a better understanding of the role of memory and the way questions are comprehended. Norman (1973) has noted several relationships between the organi zation of memory and answers to ques tions: a question may not evoke an appro priate recall if it is phrased differently from the storage format. Norman calls this the "paraphrase problem" because the "best" answer to a question may be the use of another question by the respondent in order to clarify what is intended by the original question. Norman's reference to memory brings up the problem of how people store information and combine general and specific sources of informa tion in order to reveal what they think is addressed by the question they are asked. Norman is interested in the pre-processing that occurs before an answer to a question This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions The Problem of Ecological Validity 13 can be produced. In the case of surveys we are faced with the paradox that the respondents are not encouraged to pro vide us with reasons or explanations about their answers, yet such information gives us clues about how the question was un derstood. Norman (1973) states that retrieval of information from memory requires con struction by the respondent because of the paraphrase problem. It is difficult, there fore, to show the way questions and an swers are articulated by respondents be cause no simple algorithm can be iden tified that would enable us to specify a sequence of instructions or steps or ac tions that directly connects questions and answers. We need, therefore, an under standing of the comprehension process and the construction of responses in inter views and surveys. Aspects of language presupposed in surveys and interviews I want to suggest a parallel between as pects of modern linguistic theory and sur veys. Language can refer to a lexicon and individual words as a set of carefully con structed ideals that can be studied and described independently of actual lan guage use in social settings. The linguist's syntax-based theory of language provides the ideal structures the survey researcher needs for constructing standardized ques tionnaires whose forced-choice responses can be analyzed independently of the way persons in daily life actually discuss or pose and answer questions of each other within the constraints of daily practices in socio-cultural organizations. The linguist's normative theory of lan guage describes idealized prescriptive and proscriptive rules that are recipes for de ciding what are socially acceptable or un acceptable sentences. Survey questions are equally normative because they occur in highly restrictive settings that have little or nothing to do with actual discussions or practices in group or informal organi zational activities. But the linguist's and survey researcher's idealized language structures and substantive questions about the world of opinions, attitudes, be liefs, and moral judgments are an integral part of the way researchers and the public governmental agencies conceptualize and interpret the world around them. The paradox we face is that our surveys and interviews only indirectly reflect as pects of the daily life settings of those we interrogate, yet these instruments can be the basis for the development of policy by organizations in many complex nation states. If we recognize that questions and an swers are speech acts (Austin, 1962; Searle, 1969), we can make some sug gestions about the way that the structure of language can improve our understand ing of interviews and surveys. Speech act theory seeks to combine the analysis of the propositional content of an utterance with its illocutionary force; the intention of a speaker to act on the world by the use of a promise, assertion, command, and the like. Speech acts enable the researcher to establish a functional meaning for an ut terance by the way they are classified as statements about the world, a speaker's act on the world, or a symbolic represen tation of an event in the world. A meta language was felt to be necessary for dis cussion of speech acts. The way in which surveys and inter views are conducted presupposes a model of conversational behavior that has been ignored by most sociologists. The notion of a speech act model has been extended to the idea of "conversational postulates" by Grice (1975), and derived from his more general notion of the cooperative principle. The principle refers to a kind of directive to the speaker to formulate all aspects of his or her utterance in a way that will permit participants of a conver sation to facilitate to the utmost the achievement of the explicit and tacitly agreed upon aims of the conversation. Grice identifies four categories that are designed to orient the speaker to be as informative as possible but not more in formative than seems necessary. Nor should the speaker say anything believed to be false or anything that lacks sufficient evidence. The speaker also is to be rele vant and to be brief and orderly while trying not to be ambiguous or obscure in his or her use of expressions. Notice that the term speaker would apply to both the This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions 14 The American Sociologist interviewer and respondent in the case of surveys and interviews. Another theoretical issue contained in Grice's work includes the notion of con versation implicature. In everyday con versation, listeners are expected to make inferences that do not necessarily follow from the premises or statements given, yet the statements are necessary for the com prehension of the discourse or text. The notion of conversational implicature is central to surveys and interviews as sub sets of discourse and textual activities be cause our questions require that the re spondent go beyond the information given in survey questions and presume that the utterances can be expanded in order to pursue their implications and derive co herence from what is said. Speech act theory can help us under stand the way that variations in the textual content and structure of interview and survey questions guide the kinds of in terpretations that will be made, and how these attributions of meaning will influ ence the construction of a response or the selection of an option in a set of forced choice responses. Some aspects of question-answer systems Question-answer systems deal with a sub-set of speech act theory. The ques tions employed are for the most part direct attempts to elicit information. In everyday English we often use indirect speech acts and our questions do not always follow an interrogative format. Scheduled inter views and surveys presume a respondent who is aware of the general procedure. A specific style of interrogation is used and respondents assume a response stance that differs markedly from routine every day conversation. But this response set is quite similar to occasions when someone is being interviewed for a job or is taking a test. The formal aspects of conversation out lined by Grice indirectly parallel some formal properties of question-answer systems described by Harrah (1973). Whereas Grice specifies some general conditions governing all conversations, including aspects of the reasoning neces sary for successful exchanges, Harrah in dicates conditions that are especially rele vant to the questioner. In Harrah's (1973) model the ques tioner: (a) Is presumed to know what the problem is about. (b) Knows how to express the question in an effective manner. (c) Knows what the set of possible alterna tives can be. (d) Can claim that one of the alternatives is true. (e) Does not know which alternative he wants to know. (f) Believes the respondent can help him if a particular question is put properly. But there are various logics of questions, notes Harrah, and they will vary according to the social situation in which questions are used. Thus in a classroom setting a teacher puts questions to students and knows the answers expected. Harrah de scribes some of the conditions of a Ph.D. examining committee where their ques tions may be directed as much to each other as to the student. In the Ph.D. ex amination the knowledge base of the re spondent is presumed to be the problema tic issue. When a physician asks questions of a patient, the knowledge base of the respondent may be viewed as a source of new information. But this information re quires particular types and sequences of questions and answers. The patient's an swers also may become problematic de pending on the kinds of attributions made to the patient because of age, mental status, social position, and physical con dition and appearance, to mention a few key variables that affect all question answer systems. Questionnaire items are not merely in dividual, self-contained texts, but become the basis for inferring macro-structures that resemble those reported by re searchers working on text comprehension (Kintsch and van Dijk, forthcoming; van Dijk, 1972). The respondent seeks a more comprehensive understanding of the dif ferent questions asked despite the re searchers attempts to randomize the pre sentation of questions that are linked by hypotheses in the research design. This search for a pattern on the part of the respondent is part of an attempt to create This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions The Problem of Ecological Validity 15 an explanation that goes beyond whatever cryptic information is supplied by the interviewer or questionnaire. The respon dent becomes an active participant in the survey or interview and seeks to develop his or her own hypotheses about what is going on and what intentions the re searcher projects by the kinds of ques tions asked. Discourse and textual analysis as comprehension A key issue in the study of comprehen sion is trying to estimate what the respon dent brings to the reading test, interview, or survey. There are several strategies available for studying comprehension. In unpublished work by David Rumelhart, comprehension by subjects is measured by asking them to read a story line by line while indicating what is happening after each line. An unpublished project in prog ress on reading comprehension by Charles Fillmore and Paul Kay uses reading test items (that strongly resemble survey questions) as the basis for interrogating the child about his or her understanding of the text of each test question. More abstract types of textual analysis include the identification of topics or themes and their continuity over a large textual domain (Grimes, 1980). A large and growing literature exists here that ad dresses textual analysis (Dressier, 1977; Halliday, 1967). I will not attempt to re view current work on the analysis of texts but only mention the broad goal of iden tifying those structures that would serve as a basis for interpreting specific portions and general aspects of a text. Particular conventions of language use, such as fol lowing certain forms of language structure when writing a letter, a report, in structions for filling out the necessary pa pers for a bank loan, and the like, generate expectations about how topics are intro duced, developed, and terminated. We need to know how interview and survey questions as texts are interpreted by the researcher and respondents. Current work on the analysis of discourse and textual materials can help us develop a theoretical foundation for understanding and im proving interviews. The theoretical foun dations of interviews and surveys must include the way that artificial circum stances necessary to guarantee adequate study designs can violate the ecological validity of findings vis-?-vis what takes place in daily life settings. Restating the ecological validity issue for sociology Social scientists often are so pre occupied with creating an adequate study design that they overlook the ecologi cal validity problem: Do our instruments capture the daily life conditions, opinions, values, attitudes, and knowledge base of those we study as expressed in their natu ral habitat? Recent work by Cole et al. (1978) re views the history and current efforts to deal with the problem of ecological va lidity. Cole et al. refer to the revolutionary impact of Wundt's laboratory psychology for the experimental study of mind in arti ficially constructed and simplified envi ronments. Can we extend the elegance and control of laboratory research to field settings? In sociology it is difficult to study everyday settings while using carefully formulated surveys. Much of survey re search can be viewed as the application of rigorous techniques to data elicited in simplified and artificial social envi ronments. In sociology and psychology ecological validity remains a minor issue because studies of the social organization of the laboratory and the interview or survey settings often are relegated to minor roles when data are analyzed. Psychologists have demonstrated renewed interest in pursuing laboratory-derived problems in natural settings and incorporating every day tasks into laboratory settings. In sociology this would mean contrasting the way interview and survey questions are constructed, understood, and answered, with the way that field notes and tape recordings of natural settings are used to address the same or comparable substan tive and theoretical issues. Psychologists are sensitive to problems associated with task definition, mental overload, and possible differences in the This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions 16 The American Sociologist way remembering, thinking, and attending to activities occur in laboratory and non laboratory settings. But they do not study the cultural definitions of everyday life that are part of the laboratory setting and that are invoked necessarily when at tempting to conduct controlled research in non-laboratory settings. Psychologists are not sensitive to what their subjects must be able to do to make themselves appear as normal members of a group and the larger society in order to perform ade quately in an experiment or in an every day setting. The subjects and the exper imenter both must rely on their everyday tacit knowledge in order to satisfy stated and unstated social conditions that must be followed if the research is to be consid ered successful. Sociologists are sensitive to the fact that many problems are associated with the way questionnaires are administered, coded, and organized for analysis. But they are insensitive to the information processing problems associated with these tasks. Because so many surveys are done in the same culture in which the re searchers also are native, and because we gradually have socialized our respondents to be fairly docile to the demands of sur veys, especially since everyday life cir cumstances often force them to submit to such activities, we have little knowledge about the social practices of survey re search within field settings and within re search centers where the analysis takes place. When we administer surveys in other cultures we incorporate natives who have been trained in the same method and who can tacitly negotiate the cultural dif ferences. During everyday interaction the mem bers of a group who routinely discuss political, economic, and social events are sensitive to group resources of informa tion and the interpersonal constraints that are imposed on exchanges, and also are aware of the knowledge limitations of the members of the group. The ecological va lidity issue addresses the extent to which responses to interview and survey ques tions reflect or represent the daily actions of a collectivity. We must compare the way collective discussions about topics covered in interviews and surveys parallel or differ from the way such themes are presented in the formal setting created by research goals. A partial examination of the ecological validity issue can be found in recent work by Schuman (1966), and Schuman and Presser (1977; 1977-1978; 1979; forth coming) where they show that changes in the wording of the questions often lead to changes in response patterns. The work by Schuman and Presser also contains valuable information on differences in the use of open and closed questions. The authors (forthcoming:9-10) note that if re spondents are given a question about which they know nothing, many will an swer the question if there is no explicit "don't know" option. Many respondents are willing to admit ignorance. Respon dents also are said to make an "educated (though wrong) guess" about a topic de spite their being uninformed about the issue. The authors note that these re sponses are like "non-attitudes" in the sense that there probably was no prior thought about the attitude before pre sented by the interviewer. The studies by Schuman and Presser seek to resolve possible problems in inter viewing and surveys as a way of enhanc ing their reliability and validity. These studies, though limited in scope and depth, are valuable contributions to the minimal work that has been done by re searchers working within social science who strongly support these methods in their existing form. Many serious prob lems of reliability and validity remain be cause of normal practices that are devoid of adequate concern for theoretical foun dations. For example, we restrict our col lection of information to a few categories in order to restrict the number of compari sons we have to make. The respondent's knowledge of the world is not a problema tic issue. In a laboratory study or in sur veys with fixed-choice questions, subjects or respondents bring categorical mech anisms with them, but the actual "pack ages of information" they employ must be either recoded or tailored to the particular conditions of the experiment or survey question and forced choices provided. The range of speech acts becomes se verely restricted in survey research. The This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions The Problem of Ecological Validity 17 respondent's cultural resources available for searching one's memory are con strained, and this limits the respondent's ability to make comparisons. The survey question introduces co-variation by the wording of the question. The following examples are from a national survey by E. C. Ladd, Jr. and S. M. Lipset (as cited by S. Lang (1978)). The questions were sent to college and university professors. (1) The statements below relate to teaching and student performance. Does each correctly reflect your per sonal judgment? (i) The students with whom I have close contact are seriously underprepared in basic skills?such as those re quired for written and oral communi cation. (a) Definitely yes (b) Only partly (c) Definitely no (2) "Grade inflation" is a serious academic standards problem at my institution. (a) Definitely yes (b) Only partly (c) Definitely no (3) American higher education should ex pand the core curriculum, to increase the number of basic courses required of all undergraduates. (a) Definitely yes (b) Only partly (c) Definitely no The questions assume a sample of re spondents familiar with the content of the items (in the present case American col lege and university professors) as well as with the meaning of "definitely yes," "only partly," and "definitely no." The questions and the responses reflect a co-variation that enables a reader to parse the question easily. The answers permit easy aggrega tion so that these and other questions can be cross-tabulated with size of school, age of respondent, discipline or field of study and the academic rank of the respondent. In the present case these questions were mailed out to the informant. There was no way for the respondent to obtain informal clarification about questions from the interviewer. The problem solving aspects of answering a fixed-choice question are severely limited. There were no open ended questions to make it possible to ob serve some elements of a problem-solving strategy at work (including the limitations of the respondent's knowledge base) as the interviewee and interviewer negoti ated the questions and answers. The questions on the basic skills of students, grade inflation, and the idea of expanding the core curriculum attribute an expertise to the respondent that cannot be chal lenged. There is no possibility of exploring the individual experiences of teachers within the same subject area much less across disciplines. Nor can we distinguish respondents by the level of college or uni versity classes they teach. We have no information on the background of their students. The categories created by the re searcher must be negotiated individually by each respondent. But the researcher's categories provide ready-made classes and the response set generates automatic criteria for deciding class membership. It is difficult to interpret the meaning of these personal judgments vis-?-vis the ex periences and knowledge base of a pre sumed group of "experts." The expertise is an automatic creation of identifying a population known as being college and university teachers. The responses we obtain, however, remain ambiguous per sonal judgments. The ecological validity of the response is not addressed. Class membership and interview and survey categories An important aspect of surveys and open-ended interviews is the extent to which a concept or class membership is presupposed in the way we elicit infor mation from respondents. Our ability to perceive, remember, and talk about some object or event as an instance of a class or concept we are presumed to know is fun damental to the way we use surveys and open-ended questions. Public and private bureaucracies so cialize their employees to the use of categories that can subsume a variety of activities under identifiable classes and thus confer a stability on the world that enables them to go beyond the informa tion given. We negotiate the assignment of an object or event or some aspect of in formation to a class initially on the basis of This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions 18 The American Sociologist expectations of its perceptible elements, and then begin to infer some of the ob ject's or event's nonperceptible attributes. This is like saying that we construct the identification of typical features that en able us to claim class membership for an object or event, or we claim that certain features suggest one or more possible classes that trigger a search for additional elements that enable us to choose among several possible candidates of classes. In our perception of speech, for example, we may be forced to imagine or recall or search for information that extends be yond conventional or dictionary interpre tations of what is said because of the so cial setting. Our ability to make the non perceptible "visible" and hence integral to the invocation of a class or concept is a basic process necessary for all social in teraction and bureaucratic practices. A recent paper by Medin and Smith (in press) distinguishes between three views of concepts. The first or classical view requires that a concept have common properties which become necessary and sufficient to define the concept. Every member of a class can be specified by the properties they all must possess through a single description of all members. Medin and Smith note that attacks of this view revolve around the properties of descrip tion that must be true of all members. The second view of concepts is called the prototype or poly the tic position. The focus of this view is on the way instances of a concept can vary in th? degree to which they all share certain properties. This view says that the different instances can vary in the extent to which they will embody the concept. A single description of some may again suffice, but the prop erties of the description are now only true for most but not all of the members. Some instances of the class will possess more of the critical properties than will other in stances. Those instances possessing more critical properties are said to be more rep resentative of the concept in question. The third view described by Medin and Smith is called an exemplar notion. This view of a concept states that no single representation exists for an entire class or concept, but instead only specific repre sentations of the class's exemplars occur. The example given by Medin and Smith for this third view is that of the class of persons who might be called suicidal. The example of persons who might be suicidal is used by Medin and Smith to compare the three views briefly. They note that the classical view failed when used by clinicians because no necessary and sufficient common properties could be found to define all people who have suicidal tendencies. The polythetic view fails, say Medin and Smith, because it falls short of revealing how someone decides that a particular person is suicidal. Fol lowing a suggestion by Twersky and Kahneman (1973), Medin and Smith con tend that because clinicians are not likely to use a single description of all persons with suicidal tendencies, they might in stead make the decision about someone being suicidal by comparing the individual to other persons known to be suicidal. The exemplar view would result in the class of people with suicidal tendencies being rep resented by separate descriptions for various people known to be members of the class of suicidal persons. Two key questions raised by Medin and Smith here are as follows: (1) Is it possible to have a single or unitary description for all members of the class? (2) Can we say that all of the properties specified in a unitary description are true of all members of the class? According to Medin and Smith the classical view would answer both ques tions with an affirmative response, while the polythetic view would say yes to (1) and no to (2). The exemplar view would say no to (1) and consider (2) to be irrelevant. When we use fixed-choice question naire items the respondents are expected to be able to recognize the classes of ob jects stated in each item as self explanatory. This expectation derives from the assumed pretesting of each questionnaire item prior to sending out or utilizing the final questionnaire. The ex tent to which the respondent possesses the necessary knowledge base in order to answer the question is seldom a relevant issue. The possibility that the concepts or classes presented to the respondent may This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions The Problem of Ecological Validity 19 not be defined clearly in his or her mind is also seldom an issue. In both cases the forced-choice nature of the survey guar antees an "adequate" response so long as the respondent is willing to check off one of the choices available. The researcher's versus the respondent's conception of a particular class or concept is presumed to be resolved by the pretest done to the final questionnaire. The issue of possible social classes used is not directly testable. What is testable is the way different respondents can be distinguished by some measure of social class as determined by the way fixed-choice questions are constructed, answered, and coded. Having created several social classes by one set of ques tions, we examine other questions an swered differentially and can attribute the differences to social class membership. Surveys are presumed to be hypothesis-driven and hence a way of testing theory. But the technical aspects of the instrument make it difficult to clarify theoretical concepts or classes said to have motivated the use of the survey. Theoretical concepts are subservient to the mechanics of creating and imple menting questions and their coding. The discourse and textual processes and mechanisms that provide the theoretical basis for surveys remain unexamined, as are the social constraints and practices of the society's social stratification system that enable the researcher to utilize such an esoteric and indirect instrument to learn about the everyday activities and beliefs of the members of a group. Conclusion Part of the paper has been critical of survey research. I have slighted several issues. For example, conscientious survey researchers seek a form of quasi-experi mentation with questionnaire items in which a particular question frame is re tained but a particular word (or perhaps phrase) is altered. Important differences often are found because of these changes (cf. Schuman and Presser, 1981), even if it is not always clear what sort of reasoning we should attribute to the respondents. What is important for the survey re searcher is the patterning that occurs or emerges that gives us more confidence in the overall survey. When the same ques tions are used across different groups or with the same group at different times, and similar, or the same, patterning emerges, then the researcher feels consid erably more confident that the question naire is reflecting something significant about the respondents' opinions, atti tudes, or beliefs. Knowing that some identifiable group was more in favor of some action or law this year than last year is part of the formal patterning that is sought. There still can be problems here when particular researchers throw out items that do not seem "to work," but the general idea is to avoid using question naire items as if responses on a given oc casion can be treated in some absolute way. The goal is to look at some group or category relative to other groups or categories at specific periods of time, and not how many respondents endorse a given item at a particular time. The survey researcher seeks to control the way a data base is generated by creat ing restricted conditions under which in formation is to be elicited, coded, and an alyzed. The conditions simplify and dis tort the daily life activities of those groups and institutions we seek to understand and predict, but the controls and restricted data base are highly valued by many social scientists because they foster a sense of scientific rigor in our research. Another source of control in survey re search can be found in the enormous ad vances that have occurred in sampling theory and the researcher's ability to sam ple different respondents. What is more difficult is the sampling of behavior. In the case of voting behavior we find a fairly close correspondence between what people say in response to a questionnaire item and the way they actually vote. But other topics do not fare as well, and some not well at all. We are not clear about the behavior or activities the survey ques tionnaire items are said to index. People are not very accurate in describing their own behavior when asked to respond to direct questions. The primary difficulty remains the absence of strong theories. Instead of using strong theories we invari ably rely on the detection of patterning in This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions 20 The American Sociologist survey responses in order to guide us in making theoretical explanations after the fact. Theory seldom guides social re search explicitly; we depend on research findings to decide which theoretical con cepts seem appropriate. Sophisticated survey researchers surely can find "answers" or "replies" to the is sues I have raised and will point to the use of other sources of data or additional checks or strategies that I have not cov ered in the paper. W? need strong theories to decide whether a particular method and the data it yields tells us something worth know ing. We all are forced to deal with the same problem of interpretation regardless of whether we use surveys, census mate rials, vital statistics, extensive interview ing, participation observation, or audio or video tapes. The interpretation issue is seldom the focus of survey research, much less any other type of research in sociology. Notions like limited capacity processing, comprehension of discourse and textual materials, and language use in socially constrained contexts, remain pe ripheral topics in sociology. Yet they ad dress the interpretation issue in several explicit ways. Can we afford the con venience of ignoring these issues? REFERENCES Austin, John L. 1962 How To Do Things With Words. Cam bridge, MA: Harvard University. Bransford, John D., Barry S. Stein, Tommie S. Shelton, and Richard A. Owings n.d. "Cognition and adaption: The importance of learning to learn." Unpublished manuscript. Cole, Michael, Lois Hood, and Ray P. McDermott 1978 "Ecological niche picking: Ecological inva lidity as an axiom of experimental cognitive psychology." Unpublished manuscript. Dressier, Wolfgang U. (ed.) 1977 Current Trends in Textlinguistics. Berlin, NY: de Gruyter. Grice, H. Paul 1975 "Logic and conversation." Pp. 41-58 in P. Cole and J. Morgan (eds.), Syntax and Semantics. New York: Academic. Grimes, Joseph E. 1980 "Context structure patterns." Paper pre sented at the Nobel Symposium on Text Processing, Stockholm, August. Halliday, Michael A. K. 1967 "Notes on transitivity and theme in En glish." Journal of Linguistics 3:199-244. Harrah, David 1973 "The logic of questions and its relevance to instructional science." Instructional Sci ence 1:447-467. Kintsch, Walter and Teun A. van Dijk forth- "Toward a model of text comprehension coming and production." Psychological Review. Lang, Serge 1978 "The professors: A survey of a survey." The New York Review of Books 25:38-42. Medin, Douglas and Edward Smith n.d. "Representation and processing of lexical concepts." Unpublished manuscript. Norman, Donald A. 1973 "Memory, knowledge, and the answering of questions." Pp. 135-165 in R. S. Solso (ed.), Contemporary Issues in Cognitive Psychology: The Loyola Symposium. Washington, DC: Winston. Schuman, Howard 1966 "The random probe: A technique for evaluating the validity of closed questions." American Sociological Review 21:218-222. Schuman, Howard and Stanley Presser 1977 "Question wording as an independent vari able in survey analysis." Sociological Methods and Research 6:151-170. 1977- "Attitude measurement and the gun control 1978 paradox." The Public Opinion Quarterly 41:427-438. 1979 "The open and closed question." American Sociological Review 4:692-712. 1981 Questions and Answers in Attitude Sur veys: Experiments on the Effects of Ques tion Form, Wording and Context. New York: Academic Press, forth- "Public opinion and public ignorance: The com- fine line between attitudes and non ing attitudes." American Journal of Sociology. Searle, John R. 1969 Speech Acts. Cambridge: Cambridge Uni versity Press. Tversky, Amos and Daniel Kahneman 1973 "Availability: A Hueristic for judging fre quency and probability." Cognitive Psy chology 5:207-232. van Dijk, Teun A. 1972 Some Aspects of Text Grammers. The Hague: Mouton. This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PM All use subject to JSTOR Terms and Conditions