Journal of School Psychology

1978 Vol. 16, No. 4

0022-4405/78/16(D-4)334500.95
~) 1978 The Journal of School Psychology, Inc.

ADVERSARY EVALUATION
AND THE SCHOOL PSYCHOLOGIST
BLAINE R. WORTHEN

THOMAS R. OWENS

Utah State University

Northwest Regional Educational Laboratory

Summary: This article describes adversary evaluation and explains its use in evaluation
of school-based programs by school psychologists. The authors discuss the history of
adversary evaluation, the various models which have been used, and major research
findings. Three diverse examples of adversary evaluation are presented, along with
criteria for evaluating such models. In addition the article raises some potential concerns
regarding the use of adversary evaluation, and suggests potential roles that school psychologists might play within this paradigm.
INTRODUCTION
Most evaluation " m o d e l s " or approaches allow an evaluator to use the insights of
school psychologists or other relevant specialists in formulating issues and interpreting
findings. Yet it is apparent that some evaluation strategies encourage this approach
more than others. This article presents one relatively new approach to evaluation--the
adversarial approach--which has particular promise for appropriately involving school
psychologists and other specialists in evaluation studies. The adversarial approach will
be examined in terms of its potential for providing and integrating relevant and useful
data about school-based programs and services.
Over a decade ago, Guba (Note 1) suggested that educational evaluation might well
adopt or adapt aspects of the legal paradigm. But five years apparently passed before
the first clear-cut example of adversary evaluation in education was conducted
(Owens, Note 2). Since that time, conceptual work by Wolf (Note 3), Owens (1973)
and Levine (1974) has stimulated further interest in the adversary approach, and
several evaluation studies have used at least some of the concepts of adversarial
proceedings (e.g., Kourilsky & Baker, 1976; Nafziger, Benson, & Worthen, 1977;
Stake & Gjerde, 1975; Wolf, 1975; Hiscox & Owens, Note 4; Levine, Note 5; Wolf &
Tymitz, Note 6).
Even a cursory examination of these existing "adversary evaluations" reveals
extensive differences in purpose and underlying conception of what constitutes appropriate adversarial behavior. To illustrate this diversity, we are presenting three examples, each unique in the nature of the adversary proceedings used and in the timing of
the evaluation relative to program operation. This discussion is followed by a presentation of the basic rationale for this approach, along with a consideration of its potential
strengths and limitations, and criteria for judging the adequacy of an adversary evaluation. Finally, potential contributions of school psychologists to such evaluation studies
are outlined.
334

Worthen and Owens

335

THREE EXAMPLES OF ADVERSARY EVALUATION

The examples selected illustrate how an adversary approach can be applied before,
during and after program operation. In the first example, an adversary evaluation was
used to determine the viability of alternative programs or proposals to adoption. In the
second example, the approach was used in evaluating an ongoing process. In the third
example, the approach was used to conduct a summative evaluation of a program
which had been in existence for some time.
Adversary Evaluation of Proposed Alternatives. Perhaps the best example of how
adversary proceedings can be applied to proposed alternatives is the recent study by
Wolf and Tymitz (Note 6). In this study, the judicial approach developed by Wolf
(Note 3) was used to evaluate selected provisions of Public Law 94-142 (Education for
all Handicapped Children Act). The Bureau of Education for the Handicapped endorsed use of this judicial approach to assess the feasibility of implementing individual
educational programs legally mandated in the legislation, with hearings to be conducted in four states: Alabama, New Jersey, Washington and Wisconsin. The particular approach used was intended to de-emphasize the importance of "winning the
case," while emphasizing the importance of cross examination and other judicial
procedures in reaching a better understanding of the issues. After the two advocates-called case analysts--had conducted extensive interviews with those knowledgeable
about the issues, and had built their cases, a hearing resembling a jury trial was held.
This hearing called for use of a hearing officer, panel (jury) and witnesses. To facilitate
understanding of the issues, the case analysts consulted with each other during their
early preparation, as well as during the hearing, and the panel and hearing officer were
encouraged to question the witnesses.
Adversary Evaluation of Ongoing Processes or Programs. Perhaps the best example of applying the Adversary approach to an ongoing process is Levine's (Note 5)
evaluation of an experimental feature of the Clinical-Community Psychology program
of the State University of New York (Buffalo) Department of Psychology. This evaluation took the form of a jury trial, and focused on an experimental procedure which was
being used to evaluate Ph.D. candidates in place of a more traditional examination.
The new screening process was innovative in the deadline and penalty procedure and in
the use of two papers in place of an examination. Students were also allowed to select
their own committees and topics, in consultation with faculty committees.
An interdisciplinary group---lawyers, psychologists, statisticians, educators, and
philosophers--studied potential problems in applying a jury trial approach to evaluation, and planned the trial. After preparing a statement of standards for judging Ph.D.
screening procedures, they issued a formal complaint stating that the new procedure
violated those standards. The complaint and the response to the complaint developed
by this interdisciplinary group became the issues for the trial.
Opposing counsels (advanced law students) were each assisted by teams of three
graduate students who worked as case aides. The two teams shared information, but
developed separate trial strategies. Evidence was required to be relevant. Examination
and cross examination of witnesses was part of the trial. The jury, consisting of faculty
and graduate students, was allowed to address a limited number of questions to witnesses. They were asked to reach consensus on the validity of each of four questions
addressed in the complaint, and make recommendations for change. The jurors were

336

Journal of School Psychology

nearly unanimous in their votes, and the program was found "not guilty" on most
charges.
Adversary Evaluation of Complete Programs. Among the most comprehensive
program evaluations to use the adversary approach was the Nafziger et al. (1977)
evaluation of Hawaii's statewide, primary-grade team teaching program. ~ This
9-year-old program called for teams of three teachers to instruct from 50 to 70
students--a group that would normally be divided into two classrooms. The program
goal was to maximize the intellectual, social, emotional and physical growth of students by providing for greater individualization through team teaching.
Adversary techniques used in this study differed from those used in the judicial
model. Ten experienced evaluators agreed to form two opposing evaluation teams and
to jointly collect and share a common set of data. Before being assigned to advocate or
adversary teams, the evaluators developed a study design and data collection and
analysis procedures which they unanimously agreed would fairly, accurately and comprehensively portray the program. Eight of the evaluators were then placed on two
well-balanced teams which were randomly assigned to pro and con positions, one to
present the most convincing possible arguments for program continuation, and the
other for program termination. The remaining two evaluators were assigned as arbiters
to supervise the evaluation and to ensure that both teams were able to collect needed
ancillary data, and that they had equal resources and opportunity to develop and
present their cases.
A modified debate model, rather than a judicial hearing model, was selected for the
final presentation. The debate format seemed sufficiently flexible to allow broader
coverage of issues and more efficient use of the limited time available for presentation
of the findings. The presentation (including initial statements and rebuttals by both
teams) was first given in a closed meeting for Hawaii Department of Education staff,
the State Board of Education and selected legislators. It was subsequently compressed
into 1 hour, videotaped and repeated twice on statewide television. Presentations
allowed for questions from the audience, and television viewers were invited to voice
their views or concerns by phone or letter.
Based on information obtained through this adversary evaluation, decision makers
in Hawaii concluded that the team teaching program was not successfully meeting its
goals, and decided to phase it out and divert the resources into more promising
alternatives.

THE ESSENTIAL NATURE OF ADVERSARY EVALUATION

That the foregoing examples illustrate very different applications of the adversarial
approach should come as no surprise, since at this stage "adversary evaluation" is
little more than a rubric under which to describe a collection of widely divergent
strategies. As yet, adversary evaluation offers no unified framework meant to prescribe
the course of all such evaluations. However, a simple common thread runs through all
emerging conceptions of this approach. In its broad sense, the term refers to all

~An example of another comprehensive adversary evaluation, this one using the judicial
model, is a study of the Indiana University Division of Teacher Education (Wolf, 1975).

Worthen and Owens

337

evaluations where there is planned opposition in the viewpoints of different evaluators

or evaluation teams. Webster's sense of "contending with, opposing" is central to this
general definition. The fact that an evaluation approach includes a planned effort to
generate opposing points of view within the overall evaluation is the sine qua non
here.
Given this simple definition, it is possible to examine in greater detail the rationale
and assumptions which have led proponents of the adversary approach to argue for its
use. In large part, their enthusiasm is spurred by concern over the adequacy of more
traditional evaluation approaches in providing information useful to decision makers in
determining the fates of complex educational and social programs. Wolf (1975), for
example, has argued that many traditional school and program evaluation studies
become preoccupied with quantitative data, and de-emphasize or ignore human
judgment--a factor emphasized in the adversarial approach. House (1976) notes that
many traditional evaluation studies seem dependent on the utilitarian ethic, ignoring
the need to consider multiple outcomes and measures, emphasizing the magnitude of
test score gains without consideration for processes used to achieve them, disregarding
the pluralistic values of diverse groups interested in the evaluation, and often favoring
the growth of the higher social classes at the expense of the lower.
Certainly judgment and pluralism are not incompatible with many traditional evaluation approaches. Nevertheless, such analyses have led several evaluators to seek new
evaluation models which favor a more pluralistic ethic and the application of personal
judgment.
Proponents seem to hold at least eight assumptions regarding adversarial approaches to evaluation:
1. Evaluators are not by definition objective spectators capable of rendering unbiased
judgments about programs, based solely on canons of scientific inquiry. Quite the
contrary, evaluators are subject to certain biases that should be exposed by an
"honest opposition" charged with examining the issue from another point of view.
2. Whenever evaluation studies use multiple variables or data sources, the essence of
evaluation lies as much or more in the judgmental decision of how to weight the
evidence as in the collection and analysis of data. The fundamental evaluation
instrument remains human intelligence--applied to whatever data are yielded
through the evaluation study.
3. Social and educational phenomena are complex and multidimensional; therefore,
an effective evaluation must attempt to portray as many dimensions as possible,
revealing the interrelationships that exist. Qualitative as well as quantitative data
are needed to accomplish this task. Human testimony can be as valid as test scores.
4. Evaluation occurs in a pluralistic society in which differing values must be addressed.
5. Decision makers generally are (or should be) interested in considering various
interpretations of evaluation data, rather than being satisfied with the conclusions of
a single evaluation.
6. An adversary approach can reveal information that would likely be overlooked in
a traditional evaluation approach.
7. Important decisions regarding large-scale programs are seldom made by a single
individual. Therefore, an effective evaluation must communicate findings and interpretations to a broad audience, including all relevant decision makers.

338

Journal of School Psychology

8. Research on information dissemination (Bettinghaus & Miller, Note 7), indicates

that two-way communication is usually more effective than one-way communication in spreading new ideas. Adversary evaluation encourages dialogue between
evaluators and decision makers, especially during the presentation of evaluation
data.
Some may justifiably argue that important assumptions are omitted from this list,
or are inherent in any adversary model, while others are more germane to one particular adversary strategy. Yet, these assumptions underly many proposals that use of the
adversary approach in educational evaluation be expanded (e.g., Wright & Sachse,
Note 8).
Adversary evaluation has also gained some ardent adversaries. Popham and
Carlson (1977) cite what they view as "deep, dark deficits" in the adversarial approach. Their criticisms have sparked sharp response (e.g., Jackson, 1977), and when
evaluators meet, informal discussions among proponents and opponents of the model
seem increasingly adversarial. The criticisms and commendations offered by
evaluators lead one to reflect on which criteria are most appropriate in judging the
relative merits of this new approach to evaluation.

CRITERIA FOR JUDGING ADVERSARY EVALUATIONS

In assessing the utility and effectiveness of any new evaluation model, it is important to review existing criteria for meta-evaluation to determine their relevance. One
would anticipate a priori that at least a central core of criteria would apply to any
evaluation, yet it must be recognized that evaluation approaches differ because purposes and priorities differ, and new approaches may result in new or modified criteria.
Failure to review the criterion issue carefully could lead one to reject a new model of
evaluation because it failed to meet criteria in which it was never intended to excel.
In recent years, several evaluation theorists have proposed criteria for judging
educational evaluations. Perhaps the most generally discussed and widely applied are
the 11 general criteria proposed by Stufflebeam (1974). Four of these criteria deal with
technical adequacy: internal validity (whether the findings are true); external validity
(whether the findings are generalizable); reliability (whether the data are accurate);
and objectivity (whether the data are likely to be interpreted similarly by different
competent judges). Six deal with utility of the evaluation: relevance (whether the
findings relate to the purposes of the program); importance (whether the evaluation
covers the program's most essential features); scope (whether the evaluation addresses
all important questions); credibility (whether the evaluation encourages trust in the
results); timeliness (whether evaluation findings are available in time for use in making
decisions); and pervasiveness (whether the findings are disseminated to all intended
audiences). The final criterion is cost-effectiveness (whether evaluation costs can be
kept low without sacrificing quality).
In general, these 11 criteria seem appropriate for judging an adversary evaluation.
Indeed, a well-conducted adversary evaluation should be judged favorably on all
criteria traditionally applied to evaluation studies. 2 In addition, aspects of the adver~There is a significant possibility, however, that the manner in which the adversary process is
planned and carried out could lead to violation of some criteria, as shall be shown in a later
section of this article.

Worthen and Owens

339

sary approach seem especially relevant to certain criteria. For example, the cross
examination of witnesses, rebuttal of arguments by the adversary, and the opportunity
for the decision makers to question witnesses and other data sources are features that
can help ensure internal validity, reliability, objectivity, relevance and credibility. The
public nature of adversary hearings helps satisfy the criterion of pervasiveness. The
adversaries' efforts to synthesize data and present the most salient arguments help
ensure relevance, importance and scope.
Two criteria need clarification. As in any evaluation study, the issue of external
validity appears important only if some form of sampling is used, or if the program
being evaluated is intended for broader application.
The criterion of objectivity seems applicable to most data used in most adversary
evaluations. If human testimony is to be more widely used, however, there is also need
for acceptance of relevant subjective data. For example, the personal motives of a
witness, though entirely subjective in nature, may be very relevant in a hearing.
The potential of the adversary approach to meet the criterion of cost effectiveness is
discussed in a later section on strengths and limitations.
In addition to those discussed above, three other criteria appear important in judging an adversary evaluation: parity of advocates, audience understanding, and use of
results, a Parity of advocates refers to achieving an equitable balance in capabilities
between two adversaries or teams. Audience understanding reflects the extent to which
users of the evaluation data are able to understand both the program or process under
study and the outcomes of the evaluation. Use of results involves the extent to which
the findings are used by the individual or agency for which the study was conducted. ~
It would seem that the public nature of adversary evaluation would increase the
likelihood that these results would be useful. A well planned, carefully conducted
adversary evaluation should be rated favorably on each of these additional criteria. It is
suggested that all 14 criteria discussed in this section be used in judging any adversary
evaluation.
BEYOND SPECIFIC CRITERIA:
S O M E G E N E R A L Q U E S T I O N S AND C O N C E R N S
As is evident from the criteria discussed earlier, the adversary approach has real
merit if applied with prudence and judgment in those situations where it would be both
appropriate and advantageous. This does not mean we accept without reservation all
the claims made by proponents of adversary evaluation. Overly zealous supporters
may not be sufficiently introspective to seek and correct flaws which may exist in the
adversary concept. Moreover, preoccupation with the irrelevant paraphernalia of some
adversary approaches could cause evaluators to overlook the real benefits--and
problems--that can result from adversary evaluation.
Several issues central to the future of adversary evaluation in education are dis~The latter two may be equally relevant to other evaluation approaches.
4It is recognized that many factors which influence the use or non-use of evaluation findings
are idiosyncratic or intangible. Witness the frequent poor evaluations which are used as a basis for
action, whereas some impeccable evaluations which meet all the above criteria are shelved. It still
seems important, however, to determine whether the findings were used before reaching overall
judgments of the worth of the study; this would be especially true if one summed across several
studies to determine the frequency with which studies employing a particular approach yielded
results that were used.

340

Journal of School Psychology

cussed in detail elsewhere (Worthen & Rogers, Note 9); space does not permit complete repetition here. Nevertheless, three issues particularly relevant to the focus of this
paper merit summary. 5
Potential Pitfalls of the Legal Paradigm. Wolf (1975), Hiscox and Owens (Note
4), and Wolf and Tymitz (Note 6) have shown that the legal paradigm has intriguing
possibilities for some evaluations. Yet the legal model need not limit our conception of
adversarial evaluation. Worse, in careless hands, such approaches can deteriorate to
the level of trivial courtroom melodrama that quickly sours the uninitiated on the whole
notion of "adversarying." Use of juries and cross-examination techniques may be
appropriate in applying the legal paradigm to adversary evaluation, but cloaking the
person presiding over an educational hearing in a black robe seems as pretentious and
inane as placing powdered wigs on Congressional senators. More serious use of the
legal model can precipitate a seductive slide into what might be termed "indictment
mentality," thereby doing a disservice both to evaluation efforts and to the programs
being evaluated. Adversary evaluation literature which invokes the legal model often
contains such terms as "statement of charges" (Hiscox & Owens, Note 4), and
"guilty or not guilty" (Levine, Note 5). Such terminology may be appropriate given a
formal complaint against an educational program, as in the recent investigation of the
University of Massachusetts School of Education programs. But formal complaints,
plaintiffs and litigants are conspicuously absent from the typical educational
evaluation--and rightly so. Educational evaluation should be an instrument for improving educational programs, not for determining their " g u i l t " or "innocence." Although evaluators must of necessity judge a program's worth, that action seems a far
cry from placing a program "on trial" for specific charges.
Not only is the vocabulary of the legal model objectionable, but the model is
practicable only when there is a specific problem to be solved. Already too many
persons tend to view evaluation as something to do when a program is in trouble. It
would be unfortunate if this tendency were exacerbated, and evaluations conducted
only in response to a formal complaint regarding an offending program. We fear this
orientation may be a side effect of basing evaluations on the legal model or, for that
matter, on any model applicable only in problem-solving situations. It would be far
more salutary if educators viewed evaluation as a routine requirement for keeping
programs operating at maximum efficiency.
Obviously, one should not merely dismiss the legal paradigm as wholly inappropriate; but it would be equally foolhardy to unreservedly accept it as the best approach
to conducting adversary evaluations in education. Congressional hearings, interviews
conducted jointly by partisan interviewers, and debate paradigms are alternatives
worth considering.
Problems of Extremism in Adversary Evaluation. Whenever there are adversaries, there is always the threat of polarization leading to extreme claims. Though
extremism is not inevitable, it is quite possible that strong pro and con positions
encourage adoption of a polar decision without due consideration of the full range of
possible alternatives. In the traditional evaluation, where an evaluator is charged with
~The remaining five issues not summarized here deal with the absence of a clearly delineated
adversary model of evaluation, the impact of challenging evidence on the credibility of the
evaluation, the absence of adequate guidelines for managing adversary evaluations, the potential
impact of adversary evaluation on evaluation ethics, and the scarcity of educational evaluators
trained in conducting adversary evaluations.

Worthen and Owens

341

strict neutrality and objectivity, the middle ground might well receive as much attention as both ends of the spectrum. But what about adversary proceedings, in which the
antagonists may focus unswervingly on the extremes, deliberately----or sometimes,
unwittingly--ignoring the middle? Which best serves the decision maker: conflict or
compromise, contrast or convergence, polarization or plea bargaining? Does the adversary approach lend itself to the gathering of diagnostic information so often needed
by the thoughtful decision maker?
In shifting evaluation responsibilities from one objective and neutral evaluator to
two biased protagonists, educators stand both to gain and lose. The gain is likely to be
an increase in the spectrum of data and interpretations available to decision makers;
few other evaluation approaches seem as capable of extending the range in both
positive and negative directions as does the adversary method. Yet there is a potential
loss when unnecessary polarization shifts attention away from the middle ground,
wherein information essential to rational decision making may lie. Though it may not
be an explicit characteristic of the adversary approach, many such evaluations proceed
as if guided by an unspoken obligation to present two equally convincing cases, pro
and con. Of course, no one would tolerate a dispassionate adversary who presented a
weaker case than the data merited; but what about the advocate who errs in the other
direction, who feels compelled to keep up with the opposition, even if it means
straining or ignoring the data? Here we part philosophical company with some colleagues who seem to sincerely believe that a program is not well represented unless pro
and con arguments are equally convincing. In a forensic society wherein debate seldom
influences a proposition, that orientation seems appropriate, but not in an evaluation
where the outcome can influence real programs and real people.
Like the legal paradigm, the debate paradigm carries with it many irrelevancies that
should be eliminated before it is applied to educational evaluation. But there is a
critical difference: the touchstones of debate are polemics and persuasion--not truth,
which is central to the validity of evaluation studies. Debates surely depend upon
careful use of facts. But seldom is the debater forced to adhere as closely to unadorned
truth as is the conscientious evaluator. Logic can provide a permissive climate for
manipulating data until its form is favorable. Probably more sophistry results from
debaters' perversions of syllogistic logic than from any other self-deception known to
man. At least part of this tendency can be traced to the need to build strong cases on
flimsy foundations.
We recommend that decision makers think carefully about the charge they give to
adversary evaluators. An appropriate mandate would seem to be that of presenting the
most positive and most negative possible cases on the basis of existing evidence.
Within that framework, an evaluator might be encouraged to employ all the techniques
of persuasion available, provided adversarial zeal does not lead to flights of fancy or
specious arguments that confound or distort the evidence. Another suggestion for
keeping argumentation within reasonable bounds: we propose that adversaries consult
with one another after carefully reviewing all available data, that they identify areas of
agreement (e.g., concede points on which evidence is patently one sided), and argue
only the more equivocal points which remain in contention.
Deciding When an Adversary Evaluation Is Appropriate. Even the most enthusiastic advocate of adversary evaluation is unlikely to argue that this approach would
be appropriate for any evaluation. The adversary approach is ambitious, and can be
costly. It would seem wise, therefore, to reserve its use for cases which warrant the

342

Journal of School Psychology

increased investment of time and money. In an effort to determine which factors

should govern decisions about when to use the adversary approach, a follow-up questionnaire was sent to decision makers and evaluators involved in the Nafziger et al.
(1977) evaluation cited earlier. Respondents were asked when they thought it would be
appropriate to use an adversary evaluation. A majority of the 27 respondents indicated
that an adversary evaluation would be appropriate in the following instances:
1. When the program is controversial and public opinion regarding its value is
polarized (93%);
2. When decisions must be made about whether to continue or terminate a program
(81%);
3. When the program is large and affects many people (77%);
4. When there are many different audiences for the evaluation report (65%); and
5. When the evaluation is conducted by persons external to the program (56%).
Very few respondents (15%) felt adversary evaluation would be appropriate if conducted by internal evaluators seeking ways to improve the program.
Respondents' reactions to the cost of the evaluation were also sought. This question
was prompted in part by the frequency with which evaluators (e.g., Popham &
Carlson, 1977) have questioned whether the advantages of the adversary approach
warrant the high cost.
In fairness, the cost depends on how the evaluation is conducted. If a full-blown
courtroom procedure is employed, the cost is likely to be proportionately quite high for
the amount of data produced. If every phase of an evaluation involves the kind of
two-party cross-checking Wright and Sachse (Note 8) describe, the resultant data may
be better, but one need not be a mathematical whiz to predict that the cost will double.
Even a simple debate used as a vehicle for presenting findings from a standard evaluation increases expense.
The real question, however, is not what is the cost, but what is the cost effectiveness? On this level, it seems apparent that benefit must be argued on grounds that
adversary evaluation increases such things as representativeness of the data, fairness of
the instruments, communication between evaluators and decision makers, and identification of all the pros and cons. Whether adversary evaluation really provides more
benefits will remain an open question until someone sees fit to research the issue. In
the interim, the survey of Hawaii educators is provocative. When asked if they felt the
information produced by the adversary approach was worth the cost of involving two
teams, 78% said it was, and another 15% said it was worth more than the cost. Only
7% felt it was not worth the cost. Of course, reactions to a specific evaluation should
not be generalized, but they demonstrate that even a relatively costly adversary evaluation can be justified in the public's view."
POTENTIAL ROLES FOR SCHOOL PSYCHOLOGISTS
IN A D V E R S A R Y E V A L U A T I O N S
As school psychology emerges as a field in its own right, the function of evaluation
as a guide to decision making becomes critical, as this special issue illustrates. School
psychologists have gone beyond the role of helping individual students; they now
eAlso, it might be reiterated that the basic adversary concept can be implemented without the
heavy costs associated with some approaches discussed earlier.

Worthen and Owens

343

influence educational programs and policies (Maher et al., Note 10). Given this expanded responsibility, it is important for school psychologists to be aware of new
evaluation approaches that facilitate involvement of educators and citizens in identifying important issues, that permit qualitative and quantitative data to be combined in an
understandable way, and that allow decision makers to question experts and weigh
evidence presented from diverse points of view.
The use of adversary evaluation in education can open new and meaningful roles to
school psychologists--roles that take advantage of the special skills and training these
psychologists offer. Five possible roles are (a) principal adversaries, 7 (b) consultants to
the principal adversaries, (c) expert witnesses, (d) jury or panel members, and (e)
trainers of other school personnel in the use of adversary evaluation.
On some issues, school psychologists may be the most appropriate persons to
actually prepare and present arguments in an adversary hearing. For example, suppose
the issue were the feasibility of alternative approaches to improving student discipline
practices. School psychologists might be best qualified to prepare and present the case
because of their ability to identify useful practices, as well as their awareness of the
shortcomings inherent in opposing plans. Shore (1977) also suggests this role for
school psychologists.
The school psychologist may also be called on by others who take responsibility for
preparing and presenting the case. For example, in any adversary evaluation of the new
law aimed at educating all handicapped children (Public Law 94-142), the procedures
designed to implement the law will almost certainly come under scrutiny. School
psychologists should be among the consultants asked to examine the procedures, cite
their merits and deficits, or identify areas in which further information is necessary to
support judgments of worth.
As an expert witness, a school psychologist may be called upon to testify about the
validity of a particular test for a given student population. At other times, the psychologist may be called upon to highlight significant research findings or discuss the effects
of a new program upon exceptional students.
A panel of informed professionals and citizens, called upon to act as a " j u ~ " in an
adversary hearing, should include school psychologists and other specialists whenever
their insights seem relevant to the program under investigation. A psychologist could
also help other panel members judge the adequacy of any supporting evidence presented.
Furthermore, a school psychologist can play an important role in helping other
professional personnel understand the importance and utility of program evaluation.
In this training role, the psychologist can assist teachers, administrators and others in
understanding the purposes of adversary evaluation, situations in which it would be
appropriate, and ways of employing it in evaluating their particular school programs or
services.
CONCLUSION
We have attempted to describe the adversary concept as an intriguing new approach
to evaluation of school-based programs and services. The paper includes examples of
widely divergent uses of the approach, each of which illustrates the characteristic
7These might be "prosecutor and defense attorney" in the courtroom model, case analysts in
other types of judicial approaches, or presenters of pro and con arguments in a debate paradigm.

344

Journal of School Psychology

common to all: planned opposition within the evaluation. In addition, the paper covers
some underlying assumptions of adversarial approaches, and outlines criteria for judging the adequacy of a particular adversary evaluation. It cites questions and potential
pitfalls to be considered by anyone intending to use the adversary approach, and offers
five suggestions for possible ways in which school psychologists might contribute to
an adversary evaluation.
Does the adversary approach result in better evaluations? Does it really offer the
advantages claimed by its proponents, or does it suffer from the serious deficits
described by its distractors? In our opinion, it is too early to answer these questions
with certainty. Our hope is that the adversary approach will be tried by those who share
our desire to improve and refine it for use in appropriate educational settings. It may or
may not live up to the promises of its avid supporters, but we believe the adversary
approach to educational evaluation deserves a fair trial.

REFERENCE N O T E S
1. Guba, E. G. Evaluation infield studies. Address at evaluation conference sponsored by the
Ohio State Department of Education, Columbus, Ohio, 1965.
2. Owens, T. R. Application of adversary proceedings for educational evaluation and decision
making. Paper presented at the annual meeting of the American Educational Research Association, New York City, 1971.
3. Wolf, R. L. Application of select legal concepts to educational evaluation. Unpublished
doctoral dissertation, University of Illinois, 1973.
4. Hiscox, M. D., & Owens, T. R. Attempts at implementing an educational adversary model.
Paper proposed for presentation at Third Annual Pacific Northwest Educational Research and
Evaluation Conference, Seattle, 1975.
5. Levine, M. Experiences in adapting the jury trial to the problem of educational program
evaluation. Unpublished manuscript, State University of New York at Buffalo, 1976.
6. Wolf, R. L., & Tymitz, B. L. Enhancing policy formulation through naturalistic judicial
inquiry procedures: A study of the individual education program component of Public Law
94-142. Technical Report, Indiana Center for Evaluation, September 1977.
7. Bettinghaus, E. P., & Miller, G. R. A dissemination system for state accountability programs. Unpublished document, Michigan State University, 1973.
8. Wright, W. J., & Sachse, T. Payoffs of adversary evaluation. Paper presented at the annual
meeting of the American Educational Research Association, New York City, April 1977.
9. Worthen, B. R., & Rogers, W. T. Uses and abuses of adversary evaluation: A consumer's
guide. Paper presented at the annual meeting of the American Educational Research Association, New York City, April 1977.
10. Maher, C. A., et al. Program evaluation and the school psychologist: A synoptic model for
the conduct of evaluative activity in school settings. Paper presented at the annual meeting of
the American Psychological Association, San Francisco, August 1977.
REFERENCES
House, E. R. Justice in evaluation. In G. V. Glass (Ed.), Evaluation studies review annual.
Beverly Hills, Cal.: Sage Publications, Inc., 1976.
Jackson, G. Adversary evaluation: Sentenced to death without a fair trial. Letter to the editor in
Educational Researcher, November 1977, 6(10), 2.
Kourilsky, M., & Baker, E. L. An experimental comparison of interaction, advocacy and adversary evaluation. CeDR Quarterly, 1976, 9(2), 4-8.
Levine, M. Scientific method and the adversary model, some preliminary thoughts. American
Psychologist, September 1974, 29(9), 661-677.
Nafziger, D. H., Benson, J., & Worthen, B. R. 3 on 2 evaluation report, 1976-77. Volume I,
Technical Report. Portland, Ore.: Northwest Regional Educational Laboratory, January
1977.

Worthen and Owens

345

Owens, T. R. Educational evaluation by adversary proceeding. In E. R. House (Ed.), School

evaluation: The politics and process. Berkeley, Cal.: McCutchan Publishing Corporation,
1973.
Popham, W. J., & Carlson, D. Deep dark deficits of the adversary evaluation model. Educational Researcher, 1977, 6(6), 3--6.
Shore, K. The judicial evaluation model in theory and practice. School Psychology in New
Jersey, 1977, 19(I), 6-8.
Stake, R. E., & Gjerde, C. An evaluation of TCITY: The Twin City Institute for Talented Youth.
Report #1. Kalamazo, Mich.: Evaluation Center, Western Michigan University, November
1975.
Stufflebeam, D. L. Meta evaluation. Occasional paper No. 3. Kalamazoo, Mich.: Evaluation
Center, Western Michigan University, 1974.
Wolf, R. L. Trial by jury: A new evaluation method, 1. The process. Phi Delta Kappan,
November 1975, 185-187.
Blaine R. Worthen
Department of Psychology
Utah State University
Logan, Utah 84322

Thomas R. Owens
Northwest Regional Educational Laboratory
710 S.W. Second Avenue
Portland, OR 97204