JOURNAL OF PERSONALITY ASSESSMENT, 83(3), 256276

Copyright 2004, Lawrence Erlbaum Associates, Inc.

Rorschach Assessment of Changes Following

Psychotherapy: A Meta-Analytic Review RORSCHACH
GRNNERD
CHANGE ASSESSMENT

Cato Grnnerd
Department of Psychology
University of Oslo

I examined Rorschach assessment of personality changes following psychotherapy. I con-

ducted a comprehensive literature search to find all studies using the Rorschach method at least
twice for the same participant in connection with psychotherapy. I conducted meta-analyses for
38 samples, and I performed regression analyses to identify moderating factors. Across all Ror-
schach scores, the total weighted sample effect size was r = .26, and nearly half the variables
obtained effect sizes higher than .30. Several moderating factors were found. Most important,
effect sizes increased with longer and more intensive therapy. More concern for interscorer reli-
ability was associated with larger effect sizes, whereas a higher degree of scorer blinding was
associated with smaller effect size magnitudes. Predicted levels of change based on the regres-
sion models indicated substantial increases in effect size with longer therapies. The data indi-
cate that many elements in the Rorschach are valid indicators of change despite the poor reputa-
tion the method has acquired within psychotherapy research.

The Rorschach method is viewed by many as the clinical tool
par excellence. Critics, on the other hand, have been quite
strong in their dismissal of the Rorschach in clinical and fo-
rensic settings (e.g., Garb, 1999; Garb, Wood, & Nezworski,
2000; Wood, Lilienfeld, Garb, & Nezworski, 2000). In psy-
chotherapy research, the Rorschach method, along with other
so called projective instruments, has largely been discarded
in recent years because of poor psychometric qualities, reli-
ance on inference, and their reflection of orientations that
emphasize unconscious processes (Lambert & Hill, 1994, p.
73). Although Lambert and Hill acknowledge the fact that
the Comprehensive System (CS; quoting the second edition;
Exner, 1986) has overcome some of these problems, the Ror-
schach method is still viewed as too time consuming and
costly. Self-report and observer inventories are preferred.
During the last decades, studies using the Rorschach method
have gradually disappeared from reviews on psychotherapy
(e.g., Garfield, 1978, 1994; Meltzoff & Kornreich, 1970).
The Rorschach method is not widely used in psychother-
apy research, but among the studies that exist, even fewer
studies have used the method to evaluate therapy progress by
testing before and after treatment. Uncertainty regarding
temporal stability might be one reason for this. However, in a
recent meta-analysis (Grnnerd, 2003), I concluded that
temporal stability levels are moderate to high. The varying
levels of stability seen among some variables were, among
other factors, related to the length of the retest periods (i.e.,
longer intervals were associated with less stability) and the
number of participants in the samples (i.e., larger samples
were associated with higher stability). Furthermore, some of
the variables with the lowest stability levels reflect state-like
features of personality, which cannot be expected to remain
as stable as those reflecting trait-like features. Ample evi-
dence can also be found of acceptable and often excellent
interscorer reliability levels in Rorschach studies (Acklin,
McDowell, Verschell, & Chan, 2000; McDowell & Acklin,
1996; Meyer, 1997a, 1999a; Meyer et al., 2002). Concerns
about reliability should therefore not be a decisive factor
against the Rorschach method.
The general scepticism toward the Rorschach method
might also be an important factor when assessment methods
are selected for a study. Some previous meta-analyses on the
validity of the Rorschach method (Atkinson, Quarrington,
Alp, & Cyr, 1986; Parker, 1983; Parker, Hanson, & Hunsley,
1988; West, 1998) have been criticized for various faults and
biases (Garb, Florio, & Grove, 1998, 1999; Garb et al.,
2000), although some of the critical arguments have them-
selves been disputed (Hiller, Rosenthal, Bornstein, Berry, &
Brunell-Neuleib, 1999; Parker, Hunsley, & Hanson, 1999).
Other analyses have clearly indicated, however, that the Ror-
schach method can yield acceptable effect size magnitudes.
Hiller et al. (1999) found unweighted and weighted effect
sizes of r = .29 and r = .26, respectively, based on crite-
rion-oriented validity studies. The Rorschach method ob-
 RORSCHACH CHANGE ASSESSMENT
tained larger effect sizes than the Minnesota Multiphasic
Personality Inventory (MMPI; Hathaway & McKinley,
1943) when behavioral outcome criteria were used such as
hospitalization, discontinuation from therapy, and so forth.
The MMPI, on the other hand, showed larger effect sizes
when it was compared with other self-report criteria. Hiller et
al. (1999) suggested that the MMPI and other self-report in-
struments share method variance through a similar focus on
how people present themselves. The Rorschach method, on
the other hand, might be more related to what people actually
do as defined by behavioral criteria.
Hiller et al. (1999) examined the validity of various
variables from the Rorschach method in a diverse set of stud-
ies. Two other meta-analyses have also shown encouraging
results for more focused assessments using specific vari-
ables, namely the Rorschach Prognostic Rating Scale (Meyer
& Handler, 1997, 2000), and the Rorschach Oral Depend-
ency Scale (Bornstein, 1999). Both studies concluded that
the Rorschach method can be a highly valid predictor of ob-
jective criteria such as psychotherapy outcome or depend-
ency behavior. Meyer and Archer (2001) summarized data
from all available meta-analyses and reanalyzed the Parker et
al. (1988) data based on the criticism raised against this study
(Garb et al., 1998). Meyer and Archer found that the Ror-
schach method performed neither better nor worse than other
assessment methods. Across all methods, effect sizes gener-
ally hovered around r = .30. For the Rorschach, effect sizes
from general and focused meta-analyses ranged from .14 to
.46 (see Meyer & Archers Table 4, p. 493; a simple average
of listed studies yielded r = .34).
Meta-analyses of psychotherapy studies have yielded re-
sults in the same range as the Rorschach studies. Smith and
Glass (1977) in their classic study of benefits from psycho-
therapy found an average effect size of r = .32 (d = .68). On
average, though, effect size levels for the benefits of treat-
ment seem to have been somewhat lower. Lipsey and Wilson
(1993) summarized data from 302 meta-analyses studying
various kinds of treatments. The average effect size from
these studies was r = .24 (converted from d = .50, assuming
equal number of participants in treatment and comparison
groups), whereas a more limited selection of 156 carefully
controlled studies yielded r = .23 (d = .47) on average.
Forty-one of the 302 meta-analyses examined psychotherapy
studies only and are therefore more comparable with the pur-
pose of this study, and they yielded an average of r = .33 (d =
.63, simple average computed from Lipsey & Wilsons,
1993, Table 1, Point 1.1 and 1.2). Later meta-analyses of the
benefits from psychotherapy have obtained similar results.
Grissom (1996) summarized 68 meta-analyses and found
that the typical difference between a therapy and a control
group was r = .35 (d = .75). Westen and Morrison (2001)
found effect sizes of r = .15, r = .37, and r = .41 for
manualized treatments of depression, panic disorder, and
generalized anxiety disorder, respectively (ds = .3, .8, and .9,
respectively). Martin, Garske, and Davis (2000) found the
257
association between working alliance and outcome was r =
.22. Finally, Shadish, Navarro, Matt, and Philips (2000) ex-
amined the benefits of psychotherapy using studies in clini-
cally representative conditions. Shadish et al. found effect
sizes of r = .20 and .15 for random effects and fixed effects
models, respectively.
Rorschach studies and psychotherapy studies in general
thus have seemed to yield similar effect size magnitudes. The
question remains how the Rorschach performs in the field of
psychotherapy evaluations. Meyer and Archer (2001) called
for more data on the Rorschachs locus of effectiveness to en-
hance effectiveness and utility of Rorschach testing. As
pointed out in another extensive literature review (Meyer et
al., 2001), different assessment instruments yield different
types of information, and getting a clearer picture of the
strengths and weaknesses of the Rorschach method is there-
fore important. To achieve this, Meyer and Archer stressed the
importance of doing meta-analyses directed toward more lim-
ited areas of application such as more specific scorings, inter-
pretations, and contexts. Global meta-analyses may give
useful information on the general yield of an instrument com-
pared with other instruments but are inherently limited by their
lack of focus on more limited and specific applications. Some
Rorschach variables may be valuable in some contexts but not
others; some may be generally valuable across contexts,
whereas others may not prove valuable at all.
This analysis is an attempt to move in such a direction by
aiming at the very practical and every day clinical task of as-
sessing progress in psychotherapy. I made an examination
into studies using the Rorschach method to assess personal-
ity changes following psychotherapy to attempt to clarify
whether this dismissal of the Rorschach method in psycho-
therapy research is warranted. Furthermore, I make a tenta-
tive comparison with temporal stability data from a previous
meta-analysis (Grnnerd, 2003). Considering the two top-
ics together is important, as temporal stability has a direct
bearing on the validity of the results. According to general
psychometric theory, reliability is a prerequisite for validity
(e.g., Grove, Andreassen, McDonald-Scott, Keller, &
Shapiro, 1981). Reliability limits the validity that can be ob-
tained for a measure (the level of validity is limited to the
square root of the reliability). Short-term mood fluctuations
and long-term personality changes can also be observed in
temporal stability data. Most of the constructs I want to mea-
sure are not entirely stable. The challenge for psychotherapy
research in this regard is to identify clinically meaningful
changes as opposed to changes that might be statistically sig-
nificant but do not bear any clinical importance.
In this study, I evaluated the usefulness of the Rorschach
method. Whether psychotherapy is effective or not is not an
issue here, only whether the Rorschach method can detect ac-
tual changes. Psychotherapy research has shown that psy-
chotherapy is effective (e.g., Bergin & Garfield, 1994;
Grissom, 1996; Lipsey & Wilson, 1993; Seligman, 1995),
and this general conclusion is used as a basis for this study. I
258
touch on the relative strengths and weaknesses of the Ror-
schach method compared with other instruments in psycho-
therapy evaluation. Specifically, I report change data from
other instruments, but I make no attempt to gather represen-
tative data for these instruments to compare with the Ror-
schach method.
STUDY 1: GLOBAL ESTIMATES OF CHANGE
Method
Literature search. I did a comprehensive literature
search in the PsycINFO database from 1921 to October 2003
aiming at finding all studies that reported changes following
psychotherapy with the Rorschach method. The search terms
were also used in Grnnerd (2003) to find temporal stability
studies. I included journal articles, books, and dissertations
in the search and used the terms stabilit* or retest* or
(consisten* and time), combined with (therap* or
psychotherap* or treat*) and (outcom* or contin* or
prognos* or predict* or improv* or chang*). I excluded stud-
ies of specific effects of drugs, medicines, surgery, or other
purely somatic treatments together with studies of children
and adolescents, meaning that the mean age of the partici-
pants at retest had to be at least 18 years. I also excluded
group and consensus Rorschach studies. All these restric-
tions were implemented in the search with the following ex-
clusion terms: family rorschach*, joint rorschach*, consen-
sus rorschach*, group rorschach*, ag=childhood, surgic*,
brain damag*, brain lesion*, and case stud*. Language was
restricted to English, Norwegian, Swedish and Danish.
This initial PsycINFO search returned 562 references. In
addition, I reviewed indexes and reference lists in other
available material1 including standard Rorschach textbooks
(Beck, 1950; Klopfer, Ainsworth, Klopfer, & Holt, 1954;
Piotrowski, 1957; Rapaport, 1949; Schachtel, 1966; Schafer,
1954), Rorschach reviews (Goldfried, Stricker, & Weiner,
1971; Jensen, 1959; Lerner, 1975; Rickers-Ovsiankina,
1960), and psychotherapy reviews (Lambert & Hill, 1994;
Meltzoff & Kornreich, 1970; Orlinsky, Grawe, & Parks,
1994). As work progressed and studies were retrieved in full
text, I checked all reference lists for relevant studies that
were not identified in the PsycINFO search.
I retrieved and examined more than 600 study abstracts to
find studies in which participants were undergoing psycho-
therapy, were tested at the beginning and at least once during
or after psychotherapy, and in which Rorschach data reflect-
ing changes from baseline to retest was analyzed. Because of
poor indexing of many studies in the PsycINFO database,
some studies slipped through the exclusion criteria. During
1
I thank Gregory J. Meyer for providing additional references that
were not located in the search process.
GRNNERD
examination of the abstracts, I enforced the search exclusion
criteria and a number of other exclusion criteria, as explained
in the following, before a study was retrieved in full text. I
liberally defined psychotherapy as a nonexperimental thera-
peutic intervention over a minimum period of 1 month of an
established type such as psychoanalysis, client-centered
therapy, or brief cognitive therapy. I excluded nonstandard
administrations of the Rorschach card, such as group admin-
istrations, modified cards, alternative series (e.g.,
Behn-Rorschach Test; Zulliger, 1941) as well as studies in
which the participants were given only some of the 10 cards.
I also excluded studies involving participants with disabili-
ties such as mental retardation or deafness. Medication was
defined as active treatment in this context, and therefore, I
only included studies comparing psychotherapy groups with
nonmedicated control groups.
Based on the examination of the abstracts, I retrieved 158
studies in full text. These were studies that either were clearly
relevant or studies in which it could not be determined from
the abstract or the keywords whether it fulfilled the inclusion
and exclusion criteria or not. If there was any doubt, I re-
trieved the study in full text. The full text examination
showed that the majority of studies fell clearly short of the
criteria. Other studies seemingly reported relevant data, but I
excluded them on closer examination as explained in the Ap-
pendix. In addition, Glatt and Karon (1974) included partici-
pants not undergoing therapy, but B. Karon (personal
communication, October 7, 2003) kindly provided the raw
data from the participants who were in therapy. Kavanagh
(1985) only reported data from the variables that were signif-
icant, and Kavanagh could not be reached to provide addi-
tional data. From the Method section, it was possible,
however, to determine which other variables were examined
that were not significant. A secondary source (Hamlin &
Albee, 1948) was used to code data from Muench (1947),
which could not be retrieved despite persistent attempts. Sev-
eral attempts were made to contact the authors of the ex-
cluded studies listed in the Appendix to inquire if additional
data was available. No additional data was obtained from
these contacts.2
From the set of 158 studies retrieved in full text, 24 studies
had reported relevant and usable data and were included in
these analyses.
2
The following authors were contacted during August to October
2003, but they were not able to provide additional data: Noble A.
Endicott (Endicott & Endicott, 1963), Harold S. Zamansky
(Zamansky & Goldman, 1960), John E. Exner, Jr. (regarding retest
period in the Exner, 1974, study) and Judy L. Kantrowitz
(Kantrowitz, Katz, & Paolitto,1990; Kantrowitz, Katz, Paolitto,
Sashin, & Solomon, 1987a, 1987b; Kantrowitz, Paolitto, Sashin,
Solomon, & Katz, 1986). The authors of the following studies could
not be reached: Barry, Blyth, and Albrecht (1952); Cadman,
Misbach, and Brown (1954); Rioch (1949); Kavanagh (1985);
Schwager and Spear (1981); Dudek (1970; regarding retest period
for the psychotherapy group); and Spero (1984).
 RORSCHACH CHANGE ASSESSMENT
Coding. In some cases, a study provided data from one
sample of participants, and the study would then be coded as
one sample. Other studies reported data from two separate
samples of participants, each receiving a different type of
therapy. I coded these subsamples as separate samples in
these analyses. Furthermore, Lipsey and Wilson (2001) rec-
ommended that data from simple pretest to posttest compari-
son analyses and from treatment versus control analyses
should be analyzed separately because pretest to posttest
comparison designs will generally yield higher effect sizes
than treatment versus control designs. In cases in which both
these types of data were reported for the same participants, I
coded the studies as two separate samples that each went into
separate analyses.
I considered all data reflecting changes from baseline to
the final testing. Exner and Andronikof-Sanglade (1992),
Glatt and Karon (1974), Haimowitz and Haimowitz (1952),
and Weiner and Exner (1991) reported data from two or three
retests, but only the final retest was coded in Study 1 (Study 2
included all retests). Doherty (2001) and Glatt and Karon
(1974) both reported data based on the same participants, and
I therefore coded them as one sample, identified as Glatt and
Karon (1974). In cases in which the study reported data from
other methods alongside the Rorschach, I also coded this data
to allow a comparison of the Rorschach method with other
methods.
I did not code subgroups within samples separately unless
the subgroup represented a specific form of therapy. Blatt,
Ford, Berman, Cook, and Meyer (1988) presented data for
subgroups classified as anaclitic and introjective, but I coded
only the change data for the whole sample. Similarly, I did
not consider subgroups in Glatt and Karon (1974) treated by
experienced versus inexperienced therapists.
I thus divided the 24 included studies into 29 samples.
Twenty-two of these reported pretest to posttest analysis
data, and the remaining 7 samples reported treatment versus
control analysis data. One study reported both pretest to
posttest data as well as control data for the same participants
(Dudek, 1970). I coded different kinds of result statistics (r, t,
F, 2, p, and z) from Rorschach data as well as data from
other assessment instruments individually as reported (I call
these result statistics entries in this study). I only coded p val-
ues in cases in which no other more exact statistics were re-
ported or could be calculated from raw data. In cases in
which only probability limits were given (e.g., p < .05), I
coded the limits as reported. I assigned an effect size of zero
to results reported as nonsignificant. If probability levels
were based on two-tailed tests, I converted them to one-tailed
values, that is, the p value was divided by two. This proce-
dure allows the direction of the effect to be coded (Rosenthal,
1991a).
In five samples in which only probability limits were re-
ported (Coons, 1957; Coons & Peacock, 1970; Exner, 1974;
Exner & Andronikof-Sanglade, 1992; Weiner & Exner,
1991), I used the raw data to calculate 2 values, which in
259
turn were converted to effect size estimates. In another sam-
ple (Baker, 1998), I used means and standard deviations to
calculate an independent samples t test. The appropriate test
for these data would be a paired samples t test, but the re-
quired data were not reported. Treating the samples as inde-
pendent effectively underestimates the difference, thus
providing a conservative estimate in this context. Campo,
Dow, and Tuset (1988) reported z-test data for pretest to
posttest differences, whereas the appropriate test for the re-
ported design would have been a paired samples t test. I
therefore coded the data as t values, which resulted in a more
conservative estimate of the differences. Krout, Krout, and
Dulin (1952) contained a few discrepancies between the ta-
bles and the text, but only the table data was considered. The
Weiner and Exner (1991) study contained two errors, but the
correct data was kindly provided by I. B. Weiner (personal
communication, August 5, 2003).3
All authors had made predictions about the expected di-
rection of the changes for each of the variables they exam-
ined. The studies could therefore be classified as
confirmatory studies in which a number of variables thought
to be important for assessment of changes had been selected
by the authors based on theory and/or previous empirical
data. Each entry was classified as representing an effect in
the predicted direction, a null result, or an effect in the oppo-
site of the predicted direction. All effect sizes that were in the
direction that had been hypothesized by the author(s) were
given a positive sign, and all in the opposite direction were
given a negative sign. I coded data that resulted in effect sizes
of zero as null results.
I did not code the Diagnostic Interview for Borderline
(Gunderson, Kolb, & Austin, 1981) data reported in Baker
(1998) and the Bell Adjustment Inventory (Bell, 1934) data
in Hamlin and Albee (1948). In both cases, high scores on
these instruments were used as selection criteria for the sam-
ples, and changes in the scores from pretest to posttest were
tested and found significant. This design creates conditions
for regression toward the mean to mix with true changes.
Because the exact amount of regression artifacts in
nonrandomized pretest to posttest designs cannot be estab-
lished (Campbell & Kenny, 1999), I excluded the data as a
precaution.
I did not check the coding for reliability, which may be re-
garded as a weakness of this study. After the initial coding,
however, I made two thorough checks of the complete cod-
ing with more than a year passing in between each check. In
both rounds, errors were detected and corrected, and signifi-
cantly less errors were detected in the last round. Although
this procedure cannot replace a proper reliability check, it
should inspire confidence in the accuracy of this coding.
3
In Weiner and Exners (1991) Table 3, in the data for Adj D > 0
from the first testing of the short-term sample, the correct percentage
is 23, not 3 as reported. In the same table, the correct entries for Sum
Shading > FM + m are 32, 36, 13, 14**, 10, 11, 9, and 10.
260 GRNNERD
Analyses. The correlation coefficient r was used as an
estimator of effect size magnitude. Rosenthal (1991a) pre-
sented methods to convert a variety of statistics into r. The ad-
vantage of using r over the other commonly used effect size
statistic, Cohens (1988) d, is that the same conversion is
used for dependent and independent t tests and that less data
is generally needed from the studies to compute effect sizes.
The r values should not be interpreted as traditional correla-
tion coefficients; instead, they represent a general indicator
of the size of an effect on a scale ranging from 1 to +1. Al-
though r may not be widely used in the therapy literature, a
number of meta-analyses involving the Rorschach method
have used the correlation coefficient as the effect size estima-
tor (e.g., Bornstein, 1998, 1999; Hiller et al., 1999; Parker et
al., 1988; Romney, 1990). Hemphill (2003) summarized in-
formation from two reviews of meta-analytic findings. One
review covered different forms of psychological interven-
tion, and the second covered different assessment instru-
ments. Hemphill suggested the following empirically based
guidelines for interpreting effect size magnitudes: low r <
.20, middle r = .20 to .30, and high r > .30.
I did coding and basic effect size conversions by manually
entering data and formulas onto a Quattro Pro spreadsheet
(Corel, 2000). The data was then transferred to SPSS (2001)
for analysis.
To obtain an average effect size for a sample, all calcu-
lated effect sizes from that sample were used to compute an
average. As an example (see Table 1), the three probability
limits of p = .05, .46, and .08 reported in Barendregt (1957)
were converted into effect sizes of r = .27, .02, and .23, re-
spectively. They were in turn combined into a total average
effect size of r = .18 for that sample (when exact, nonrounded
effect sizes were used).
To obtain the global estimates of change effect sizes
across samples, a weighted average based on
nontransformed sample correlational effect sizes was calcu-
lated using the sample size N as the weight. Field (2001) ar-
gued that weighted averages give the most accurate
estimation of the population average. Feild also demon-
strated that averages based on Fishers Zr transformed cor-
relations can lead to overestimations when the population
effect sizes are variable and especially so for large effects.
Conversely, averages based on raw correlations slightly un-
derestimated the population effect sizes, but to a lesser de-
gree than the overestimation based on transformed
correlations. The single predictorcriterion relationship in
this study (i.e., changes due to treatment) also calls for
weighted averages based on nontransformed correlations to
be used.
Heterogeneity was estimated by calculating Q, a 2-dis-
tributed estimate. Significant Q values indicate that system-
atic factors are associated with variations between estimates.
I address these factors in Study 2 following.
I did not correct estimates for attenuation (or level of reli-
ability) for two reasons. First, estimates could be corrected
for imperfections before entering the analyses, but in line
with Rosenthal (1991b), the real-world levels are pre-
ferred. On the practical side, too many studies failed to report
interscorer reliability levels, and the type of statistics varied.
Also, conversion from percentage agreement to kappa-type
statistics is problematic (Acklin, 1999; Grnnerd, 1999;
McDowell & Acklin, 1996; but see also Meyer, 1997a,
1999b).
In addition to the analysis of variables averaged within sam-
ples, I also analyzed different Rorschach scores and variables
across samples (the term variable is used for convenience in
this context). In cases in which a sample had more than one ef-
fect size for a variable, I averaged the results within the sample
first. Each sample thus contributed only one entry for each
variable. Most often, a single score defined a variable. As an
example, I coded entries from nine samples for the Human
Movement score, M, and these entries defined the cases for the
analysis of the variable M. In other instances, the number of
entries for a specific score was rather small, and a more encom-
passing variable was constructed. Form was one such exam-
ple, which included four entries for Lambda > .99, two for F%
and one for NonF%. Two special variables were also defined,
Constructs and Global. The Constructs variable consisted of
clinical ratings, which in these studies were based on psycho-
analytic theory, and included definitions for, for example,
Anxiety, Attitude Toward Self, Hostility, and so forth. Only
examples were given for the scoring criteria, but judging from
these reports, they were based both on traditional scoring cate-
gories, such as Movement, Shading, and so forth, as well as
more informal clinical evaluations of, for example, the pres-
ence of anxiety in the Rorschach data. The Global variable
consisted of instances in which clinicians identified which one
of two protocols was taken before and which one was taken af-
ter treatment. No specific criteria of change were given to the
judges of the studies in advance, so the judges decided for
themselves what constituted important signs of improvement.
In total, I defined a list of 32 variables. I calculated
weighted averages and levels of heterogeneity for each of
these (e.g., the effect size for Form Level was rw = .25, with a
nonsignificant level of heterogeneity).
Finally, I compared the Rorschach effect sizes to effect
sizes obtained from other assessment instruments in the same
samples. These analyses should only be regarded as tenta-
tive, however, based on two major limitations. First, contrary
to the Rorschach results, the results from other instruments
did not provide a representative selection of studies using
these instruments. A full meta-analysis of how any particular
instrument performs when evaluating therapy changes might
yield different effect size magnitudes than those reported in
these studies. Consequently, the other instruments were only
coded according to broad methodological types without fur-
ther individual attention. The types were self-report, ability,
and case rating. I discarded two instruments that did not fit
these categories (the Thematic Apperception Test [Murray,
1938] and the Id Ego Superego Test [Dombrose & Slobin,
 RORSCHACH CHANGE ASSESSMENT 261
TABLE 1
Studies Evaluating Change Following Psychotherapy Divided Into Samples

Retest
Study Diagnostic Category Treatment Modality Period N rw

Baker (1998) Borderline NR 38 22 .06

Barendregt (1957) Bronchial asthma Group therapy 19 36 .18
Blatt et al. (1988) Schizophrenia Intensive psychotherapy 15 90 .07
Campo, Dow, & Tuset (1988) NR Psychoanalysis; psychoanalytically 18 30 .14
oriented therapy
Carr (1949) NR Nondirective psychotherapy NR 9 .21
Coons (1957)
Interaction Group Control Mostly schizophrenia Interaction oriented group therapy 2 41 .36
Insight Group Control Mostly schizophrenia Insight oriented group therapy 2 42 .07
Coons & Peacock (1970) Mostly schizophrenia Group therapy 1 56 .14
Dudek (1970)
Psychoanalytic Group I Neurosis Psychoanalysis NR 16 .46
Psychoanalytic Group II Schizophrenia Psychoanalysis 20 10 .13
Psychotherapy Groups Control Neurosis and schizophrenia Psychoanalysis 67 55 .63
Exner (1974) Depression, schizophrenia, NR NR 181 .18
and character disorder
Exner & Andronikof-Sanglade
(1992)
Brief, second retest Mixed Brief therapy, various modalities 10 35 .19
Short, second retest Mixed Short therapy, various modalities 25 35 .39
Gaylin (1966) NR NR 4 57 .19
Glatt & Karon (1974), Schizophrenia Psychoanalysis 20 15 .20
Psychotherapy subgroup, third
retest
Gunderson et al. (1984)a, second Schizophrenia Insight-oriented therapy; reality-adaptive, 24 74 .20
retest supportive therapy
Haimovitz & Haimovitz (1952), NR Client-centered therapy and/or group 15 10 .80
second retest therapy
Hamlin, Berger, & Cummings NR NR 15 16 .77
(1952)b
Kavanagh (1985) Mixed Psychoanalysis; psychoanalytic 33 33 .05
psychotherapy
Krout, Krout, & Dulin (1952)
Psychoanalytic group Mixed Psychoanalytic therapy 12 19 .43
Nonanalytic group Mixed Nonanalytic therapy 13 14 .37
LaFrance (1971) Drug dependent convicts Group therapy 6 18 .81
Mintz, Schmeidler, & Bristol (1956) NR Psychoanalysis 20 20 .66
Peterson (1954) NR NR 3 42 .08
Peyman (1956) Schizophrenia Group therapy 6 32 .41
Rice (1973) NR Client-centered therapy 2 48 .27
Weiner & Exner (1991)
Short, third retest Mixed Rational emotive therapy; Gestalt therapy; 48 88 .28
modeling therapy; assertiveness therapy
Long, third retest Mixed Dynamically oriented therapy 48 88 .43

Note. N = 1,202. Retest period = the average retest period for all participants in the sample reported in whole months; rw = the average weighted effect size
obtained for each sample; NR = not reported.
aDesign presented in Stanton et al. (1984). bSecondary source for Muench (1947).

1958]) from the analyses because both came from the same
sample (LaFrance, 1971). Second, the included samples var-
ied as to how many other instruments were used. In studies in
which many instruments were used, all were compared to the
same Rorschach effect size and these pairs of observations
were therefore not independent.
In this study, I took several steps to ensure that the coding
and analyses were conservative. Given the force of recent crit-
icisms of the Rorschach method, an underestimation of effect
sizes was more desirable than an overestimation. This should
also be taken into account when the results are evaluated.
Results
Table 1 summarizes the key descriptive characteristics of the
29 included samples and their obtained effect sizes (see Table
6 in Study 2 for a complete overview of the sample coding
and average effects). The total number of participants in all
samples amounted to 1,202. Average retest periods varied
from 6 weeks to 5 years, with an average of 18.7 months. I
coded a total of 266 Rorschach entries, and I also coded 201
entries from various other assessment methods from 10 of the
samples.
262 GRNNERD
Two samples had obtained negative effect sizes. In TABLE 3
Baker (1998), the group of borderline participants showed a Combined Level and Heterogeneity
for 32 Variables
mixed pattern of changes on the Rorschach, with the aver-
age tipping below zero (r = .06). Some indexes were also Variable k N rw Q
positive, however, so the picture was not clear. In Coons
and Peacock (1970), an interaction effect occurred with a Sum Shd > FM + m 4 246 .58 2.82
D score 4 246 .58 3.69
parallel treatment given to half the group. The subgroup Constructs 3 43 .50 0.66
that received psychotherapy and a random ward treatment FC 2 26 .50 4.41*
showed a higher degree of change (r = .11) than those who Color Balance 6 310 .49 24.55***
Organization 5 303 .45 15.43**
received organized ward treatment (r = .38). Splitting the Global 6 222 .44 9.32
sample into two subsamples to separate the effect of the Populars 4 97 .38 1.31
ward treatment would compromise the coding of data from DEPI 4 246 .37 2.19
FM + m 2 26 .36 2.71
other methods in the study. The overall average sample ef- Afr 6 272 .36 7.91
fect size would also remain the same because the two sub- CDI 4 246 .36 9.30*
groups were equally large. For both reasons, the results All H Content 5 259 .34 15.48**
Shading 6 272 .32 9.07
were coded as belonging to a single sample, resulting in a Adj D 4 246 .30 10.46*
negative effect size overall. M 9 430 .30 25.59**
Average sample-based effect sizes are shown in Table 2. S 2 70 .28 0.60
Form 7 329 .27 12.34
The average for the 22 pretest to posttest design samples was
R 5 187 .26 7.46
rw = .25; for the 7 treatment versus control design samples it r & (2) 5 352 .22 1.50
was rw = .28. The difference between the two was not signifi- Miscellaneous Variables 10 329 .21 7.51
cant, t(27) = .232, p = .818. The total average for all 29 sam- EA & EB 5 575 .20 3.78
a&p 4 246 .17 2.61
ples was rw = .26. Heterogeneity was not significant for any Miscellaneous Content 5 303 .17 5.79
of the three sample groups. A fail-safe N analysis indicated Originals 1 57 .16
that 404 studies with null results are needed to invalidate the Special Scores 6 358 .14 8.35
Form Level 8 437 .12 5.25
results for the pretest to posttest design sample group, and 59 SumC 3 116 .10 5.72
studies are needed for the treatment versus control design W&D 4 89 .10 8.91*
samples. For all 29 samples, the fail-safe N was 790. Based Depth 2 176 .09 0.00
Object Relations 2 123 .03 0.01
on the lack of difference between the two sample subgroups, A% 2 26 .00 0.00
the average of all 29 samples is referred to as the main result
in the following. Note. Table sorted by effect size. k = number of samples; N = the number of
participants in the k samples; rw = the weighted average; Q = 2 distributed
Table 3 presents the average effect sizes for the 32 vari- heterogeneity statistic with df = k 1; Constructs = clinical evaluations of
ables reported in the samples. Weighted effect sizes varied personality constructs such as anxiety, attitude toward others, and libido;
between .00 and .58. The average effect size for the 32 vari- Color Balance = (FC+) (FC) CFC, CF + C > FC + 1, FC/FC + CF + C;
ables was r = .29. The diverse impact of treatment on the dif- Organization = Zd < 3.0, Zf; Global = clinical evaluation of change or
ferent variables was clearly illustrated. Using Hemphills improvement for the protocol as a whole; All H Content = H, H < (H) + Hd +
(Hd), Pure H < 2; Shading = c, T = 0, T > 1; S = S > 2; Form = F%, NonF%,
Lambda > .99; r & (2) = Fr + rF, Fr + rF > 0, 3r + (2)/R < .33, 3r + (2)/R > .43;
TABLE 2 Miscellaneous Variables = Adaptive Regression, Primary Process Thinking,
Sample-Based Effect Sizes Human Experience Variable, Rorschach Prognostic Rating Scale,
Annihilation Anxiety, Buhlers Basic Rorschach Score, Carrs signs,
Sample Group k N Q rw CI FSN Muenchs signs, Haimowitzs signs, Determinant changes, Damage content,
Hostile content, Oppression content; EA & EB = Ambitence, EA, EA < 7,
Pretest to posttest Extratensive, Introversive; a & p = Mp > Ma, p > a + 1; Miscellaneous
design 22 948 18.38 .25 .19 to .31 404 Content = Content changes, Intellect > 5; Special Scores = Confab, Contam,
Treatment versus Fab, SumSpSc > 6, TDI, WSum6; Form Level = F + %, X + % < .70, X % >
control design 7 280 5.85 .28 .17 to .39 59 .20, FQ+/o change, FQu change, FQ change; W & D = D+/D, W+/W, W,
All 29 1202 24.14 .26 .20 to .31 790
(W Dd)/R; Depth = FD > 2; Object relations = various DCOS and MOA
Note. Other averages were also calculated: unweighted averages based on indexes. See Table 6 for references.
raw correlations, unweighted averages based on Fishers Zr-transformed
correlations, and weighted averages based on Fishers Zr-transformed
correlations (weighted by the square root of N 3, the inverse of the standard (2003) empirically based classifications, only 10 variables
error of Zr). All were somewhat higher than the averages listed in the table but were of small magnitude (r < .20), whereas 14 were in the
not dramatically different. The weighted averages across all samples based high range (r > .30). Only 8 variables showed significant het-
on transformed correlations were .29, .34, and .30 for unweighted raw r,
erogeneity. More than half of the variables consisted of en-
unweighted Zr, and weighted Zr, respectively. k = the number of samples; N =
the total number of participants included in the samples; Q = 2 distributed tries from less than five samples, some from as few as one
heterogeneity statistic; rw = the average r weighted by N; CI = 95% and two, and the data from these variables must obviously be
confidence interval; FSN = fail-safe N. interpreted with caution.
 RORSCHACH CHANGE ASSESSMENT
Table 4 shows the Rorschach results compared with the
other instruments reported in 10 of the samples. Based on the
lack of significant differences between pretest to posttest de-
sign samples and treatment versus control samples, all sam-
ples were analyzed together. Also note carefully that the
effect sizes were weighted by sample size in the analyses,
which caused the high degrees of freedom.
First of all, the 19 samples that only reported Rorschach
results obtained significantly higher Rorschach effect size
magnitudes than the 10 samples that reported results from
several instruments: Rorschach only, rw = .29; Rorschach
and others, rw = .20, t(1,230) = 7.944, p < .000, two-tailed. In
the 10 samples that reported multiple instruments, the Ror-
schach effect sizes were significantly smaller than the effect
sizes of other assessment instruments. Interestingly enough,
the difference was caused by ability tests and case ratings,
which were higher in magnitude than Rorschach and
self-report methods.
Summary
Sample level effect sizes were consistently situated in the up-
per middle range of effects typically observed in psychologi-
cal studies (Hemphill, 2003). The results can be interpreted
to reflect two underlying findings. First, these effect sizes
were on the same level and even somewhat larger than the av-
erage effect size of r = .23 based on 156 meta-analyses of
treatment studies (Lipsey & Wilson, 1993). This shows that
in this group of studies, psychotherapy has had an effect that
is comparable to other psychotherapy studies.
Second, the Rorschach method reflects these changes in
the same magnitude as other instruments. This is indicated
both by these findings compared to prior meta-analyses and
TABLE 4
Effect Size Magnitudes for the Rorschach
Method Compared to Other Assessment
Methods in 10 Samples
Grouping Test df p E rw
Rorschach t = 2.808 1, 504 .005
versus others
Rorschach 10 .20
All other
instruments 17 .23
Instrument type F = 13.383 1, 502 < .000
Rorschach 10 .20
Self-report 5 .18
Ability 5 .23
Case rating 7 .25
Note. N = 471 in 10 samples. E = the number effect sizes entered from the
10 samples; rw = the weighted average; self-report = Katz Adjustment Scale
(2), Soskis Attitude Towards Illness, Camarillo Dynamic Assessment Scale,
Client Outcome Rating; ability = Wechsler Adult Intelligence Scale (4),
Visual-Verbal Test; case rating = Menninger Scales, Fair weather,
StraussHarder Symptoms, Hospital Adjustment Scale, Psychotherapy
Outcome Interview, Behavior Rating, Therapists outcome rating. See Table
6 for references.
263
by the findings in Table 4 in which the same studies had com-
pared the Rorschach method to other assessment instru-
ments. The comparison in Table 4 indicates that ability tests
and outcome ratings have shown somewhat higher levels of
change, but the studies that only have reported Rorschach
data showed significantly larger effect sizes than the
multimethod studies. Thus, the question is raised whether the
10 multimethod samples are representative of Rorschach
studies. If they are not, which there is some reason to believe,
the main implication would be that the Rorschach method is
neither inferior nor superior when making assessments of
personality changes.
Practically half the variables in Table 3 displayed effect
sizes in the high range (r > .30) indicating that some variables
are more useful than others when making assessments of per-
sonality changes. Also notice that the variables Constructs
and Global both obtained impressive effect sizes of rw = .50
and rw = .44, respectively. These results are important be-
cause the variables represent a more holistic approach to per-
sonality assessment that has more direct links to core
concepts in clinical practice.
STUDY 2: MODERATOR INFLUENCES
ON CHANGE ESTIMATES
Method
Literature search. The same 24 studies selected in
Study 1 were used in this study. The coding was more elabo-
rate, however, and included additional samples as detailed in
the following.
Coding. Exner and Andronikof-Sanglade (1992),
Glatt and Karon (1974), Haimowitz and Haimowitz (1952),
and Weiner and Exner (1991) have all reported data from
two or three retests. Each retest in these studies have been
coded as separate samples and added to the samples from
Study 1. Although this coding creates a degree of statistical
dependence among moderator variables, it allows a more
thorough analysis of the influence of treatment duration on
effect size magnitudes.
The 24 included studies were thus divided into 38 sam-
ples, 9 more than in Study 1. All the additional samples re-
ported pretest to posttest analysis data, making a total of 31
samples with in this group. The remaining 7 samples re-
ported treatment versus control analysis data. The two sam-
ple subgroups were still not significantly different, t(36) =
0.330, p = .743. All 38 samples were therefore analyzed to-
gether in this study.
The effect size coding of the additional samples was per-
formed as reported in Study 1. In addition, a number of sam-
ple characteristics were coded as moderator variables, as
presented in Table 5. Concern for Interscorer Reliability and
Blindscoring were coded because of their methodological
264 GRNNERD
TABLE 5
Moderator Variable Coding

Variable Code Description

Sample level
Concern for Interscorer Reliability 0 Not reported or no control
1 Partial control: Inappropriate method (percentage agreement), statements that reliability checks
were made but without report of data, calculations not based on data from reported subjects, low
agreement with appropriate method (mean kappa or ICC below .60)
2 Full control: Kappa or ICC, high agreement (M > .59)
Blindscoring 0 Not reported or no blinding
1 Partial blinding: Ambiguous report of blinding procedure, or a degree of blinding was applied, for
example, the same person collected all protocols and then scored them in random order (not by
pretest and posttest pairs)
2 Full blinding: Scorer(s) completely blind to pretest and posttest information
Publication Type 0 Unpublished/non-peer-reviewed: Book, dissertation, report, secondary source
1 Minor journal: Peer-reviewed journal, not coded as major
2 Major journal: Journal of Personality Assessment, Journal of Consulting and Clinical Psychology,
Journal of Clinical Psychology, Psychotherapy
Retest Period Average retest period for the sample converted into months
Therapy Load (Therapy Type + 1) Sessions per Week
Helper variables for Therapy Load
Sessions per Week Average number of sessions per week in therapy; missing values replaced by 1
Therapy type 0 Group therapy
1 Individual and group therapy
2 Individual therapy; default if not reported

Note. Only moderator variables included in one or more of the regression models are listed. ICC = intraclass correlation coefficient.

importance. As previously noted, reliability constrains valid-
ity, and poor interscorer reliability may have a detrimental
effect on effect sizes. Also, given the time span of the rele-
vant studies and the increased focus on interscorer reliability
in Rorschach data in recent years, the effects of reliability on
effect sizes in this context were of interest. When nothing
was mentioned about interscorer reliability, the study was
given the code 0. Other studies presented agreement esti-
mates that were problematic, insufficient, or inappropriate,
and these were coded 1. This coding included the use of per-
centage agreement, which has been criticized as inappropri-
ate, especially for low base rate data (Grnnerd, 1999;
Wood, Nezworski, & Stejskal, 1996). Similarly, if reliability
checks were not made of the data in the study but rather a ref-
erence was made to the scorers previous achievements, the
study was also coded 1. Finally, if agreement levels were
low, although using appropriate estimates, it was also coded
1. The study was coded 2 only when appropriate methods
were used and interscorer reliability was acceptable (> .59).
Blindscoring is important when repeated measures are ap-
plied in clinical contexts, as they were in many of the in-
cluded studies. Expectancy effects are well documented
(e.g., Rosenthal & Rubin, 1978), and blindscoring has there-
fore become a central methodological feature of high quality
studies. When no mention was made of blinding, the study
was coded 0. In cases in which the report of the blinding pro-
cedure was ambiguous or unclear, the study was coded 1.
This was also the case when a pseudoblinding was applied, as
when the administrator scored protocols at random and not
ordered in pretest and posttest pairs. The study was coded 2
only when the scorers were different from the administrators,
and the protocols were masked for pretest and posttest infor-
mation during scoring.
The Publication Type moderator addressed possible pub-
lication bias. Publications in core journals were coded 2,
publications in other peer-reviewed journals were coded 1,
and the remaining publications were coded 0. Publication
bias was an issue in the debate over a previous meta-analysis
involving the Rorschach method (Parker et al., 1988). In this
literature search, I aimed at finding all studies reporting rele-
vant data and thereby also provide a good opportunity to ex-
amine the possible influence of publication bias.
Retest Period was thought to be of major importance in
this context. Psychotherapy research has generally con-
cluded that longer therapies are associated with higher effect
size magnitudes (Orlinsky et al., 1994), and the same result
could therefore be expected in this data. Therapy duration
was highly similar to the retest period of the final test in most
cases. Initially, therapy duration was coded as a separate
moderator but was finally abandoned because of scarce in-
formation in the included studies; and in the group of studies
in which it was reported, the correlation with Retest Period
was very high.
Therapy Load was calculated as an index of therapy inten-
sity based on the assumption that individual therapy in gen-
eral can be regarded as more intensive than group therapy
and that more frequent sessions can be expected to result in a
higher degree of change.
Several other moderators were initially also coded but
were either not influential in the analyses (the exclusion limit
was p > .15 as described in the Analyses section following),
or they were highly problematic. Therefore, I give only a
 RORSCHACH CHANGE ASSESSMENT
brief description of these moderators here.4 The sample level
moderator variables were Attrition (addressing the concern
that patients dropping out could be different from those who
stayed in therapy), Number of Scores (a measure of scoring
complexity that again could be related to level of reliability),
Publication Year (addressing possible changes in method-
ological quality over the years), Randomized Design (many
studies used naturalized samples that could be thought to dif-
fer from more experimentally oriented designs), Schizophre-
nia versus Nonschizophrenia (schizophrenia patients may
show lower extent of change due to more chronic illness),
Scoring System (addressing possible differences between the
CS, other standard systems, and nonstandard systems) and
Severity of Pathology (based on the same concern as with the
schizophrenic participant; this variable was rated along the
neurotic-schizophrenic spectrum). Publication Year and
Scoring System were influential in the analyses, but both
contributed to overfitted regression models and were there-
fore discarded.
Analyses. The focus of Study 2 was to establish
whether moderator variables were systematically related to
variations in effect sizes levels. This meant that focus was
shifted away from absolute effect size levels, as analyzed in
Study 1, to the relative influence of moderator variables on
effect size magnitudes. Average levels were therefore of no
special interest and should be expected to be generally lower
in magnitude due to the inclusion of several retests. This fol-
lows the previously mentioned finding from psychotherapy
research that longer therapies are generally associated with
larger effect sizes. We should therefore expect that at least on
average, the first retests yield smaller effects than subsequent
retests.
Moderator influences were analyzed with the stepwise lin-
ear regression procedure in SPSS (2001). Outliers were
winsorized at 3 SDs from the mean before the data entered
the regression analyses. The method of weighted least
squares (WLS) was applied in the regression analysis, using
the sample size as the weight. WLS is needed when the sam-
ple variance differs between estimates, and the procedure is
generally more robust when the data have skewed distribu-
tions and moderators are intercorrelated (Steel &
Kammeyer-Mueller, 2002).
Meyer (2000) noted that the traditional alpha level of p =
.05 is too strict when building stepwise regression models,
especially when the number of cases in the analysis is small.
Following Meyers (2000) recommendations, alpha levels
were set to p = .10 for inclusion and p = .15 for exclusion. The
moderators Concern for Interscorer Reliability,
Blindscoring, and Publication Type were treated as con-
4
The full list of moderator variable codings is available from Cato
Grnnerds Web site at http://www.psykomatikk.no.
265
tinuos variables because they reflected an assumed increase
in quality with increasing values.
A relatively new development within the field of
meta-analysis is to use regression models to predict the levels
obtainable from an ideal study or in specific modifications of
single studies (e.g., Roberts & DelVecchio, 2000; Shadish et
al., 2000). The obtained regression equation provides an ac-
count of the relative influence of the various moderators but
also provides a possibility to enter the best possible values
for each moderator back into the equation. In practice, this is
done by multiplying the ideal moderator values with their
corresponding unstandardized regression coefficients. This
procedure allows a researcher to address specific shortcom-
ings in the data set and estimate what kind of change can be
achieved when the effects of these limitations are contained.
In line with Grnnerd (2003), I applied this method in these
analyses.
I conducted two analyses. The first analysis was sample
based and included the average effect size across all Ror-
schach scores in each of the 38 samples listed in Table 6. The
moderators presented in Table 5 were used as potential pre-
dictors of the sample-based effect sizes.
In the second analysis, regression analyses were per-
formed separately for variables that were used in at least five
samples. Note carefully, however, that compared to the data
listed in Table 3, these analyses included data from all the re-
test intervals, and therefore, the number of entries and aver-
age levels of change were in many cases different from those
listed in Table 3. The 2 miscellaneous variables were omit-
ted, which left 13 variables to be analyzed. As an example, 6
entries for the Affect Ratio, Afr, defined the cases of one re-
gression analysis (no Afr data was available from the studies
with several retests, so these cases were equivalent to those
listed in Table 3) as did 17 entries for M (the 9 entries from
the final retests in Table 3 and an additional 8 entries from in-
termediate retests), 16 entries for Form Level (8 from final
retests and 8 from intermediate retests), and so forth. Given
the low number of entries for most variables, the potential
moderator variables were restricted to Retest Period and
Therapy Load.
The inclusion of several retests of the same participants
creates a degree of dependence between the moderator vari-
ables and the effect sizes themselves. First, when results
from several retests are reported, they are not independent re-
sults. The baseline will be the same for all retests, thereby
limiting the variance between the various retests. Second,
when all retests in a study are defined as separate samples, all
the sample-level moderators except Retest Period will be
identical, for example, Concern for Interscorer Reliability
was the same for all retests, Publication Type was the same,
and so forth (note, however, that because of attrition, the
number of participants was not always the same for all retests
within a study). This coding practice will effectively weight
the moderators according to the number of retests from a
study. As previously noted, WLS regression has proven to be
266
more robust in the face of moderator nonindependence but
can only partly rectify its influence. To assess the extent of
the problem, I ran a control analysis including only the final
retest of each sample to remove moderator interdependence.
If the obtained model from this analysis was similar to the
main analysis model, the impact of nonindependence should
not be decisive in these results.
Results
Table 6 shows the complete coding of all retest intervals for
all samples including both Rorschach data and data from
GRNNERD
other methods (parts of this data were also presented in Table
1). According to a common guideline (Curran, West, &
Finch, 1996), none of the moderators were highly skewed,
and no transformations were therefore necessary.
Table 7 shows all generated regression models. Statis-
tically significant prediction equations were computed for
the sample-level analysis and for 4 of the 13 specific vari-
ables. For the sample-level analysis, Retest Period proved
most influential. Increased length of time in therapy was re-
lated to increased effect sizes. The next factors were related
to methodology. Effect sizes were higher in magnitude in
studies coded with higher levels of Concern for Interscorer

TABLE 6
Coding and Effect Sizes for All 38 Samples

Study Scoring System Moderator Codinga N E PP rw TC rw

Baker (1998) CS 38/P/n/N/f/S/1.0/m/34/2/m/60/92/U 22 4 .06

Barendregt (1957) Own 19/N/n/S/f/N/2.0/lo/46/2/g/0/3/Mi 36 4 .18
Blatt et al. (1988) ABZb 15/N/r/N/f/S/4.0/m/21/12/i/10/41/Mi 90 23 .07
Menninger scales (Harty et al., 1981) 6 .33
Fairweather symptoms (Fairweather et al., 1960) 1 .14
StraussHarder symptoms (Strauss & Harder, 4 .16
1981)
WAIS (Wechsler, 1944) 3 .18
Campo, Dow, & Tuset (1988) Own 18/N/n/S/n/N/3.0/lo/27/9/i/0/9/Mi 30 9 .14
Carr (1949) Beck NR/N/n/N/n/S/NR/m/NR/3/g/0/27/Ma 9 2 .21
Coons (1957)
Interaction Group Control Clin.ev.c 2.3/P/r/N/f/N/3.0/m/NR/3/g/3/1/Mi 41 1 .36
WAIS 3 .27
Insight Group Control Clin.ev. 2.3/P/r/N/f/N/3.0/m/NR/3/g/3/1/Mi 42 1 .07
WAIS 3 .02
Coons & Peacock (1970) Clin.ev. 1.4/N/r/N/f/N/5.0/m/29/5/g/5/1/Mi 56 6 .14
WAIS 1 .35
Hospital Adjustment Scale (McReynolds & 1 .18
Ferguson, 1953)
Dudek (1970)
Psychoanalytic Group I Klopfer NR/N/n/S/n/C/2.0/lo/29/6/i/0/3/U 16 11 .46
Psychoanalytic Group II Klopfer 20/N/n/S/n/C/2.0/h/29/6/i/0/3/U 10 11 .13
Psychotherapy Groups Control Klopfer 67/N/n/N/n/C/2.0/m/32/6/i/0/3/U 55 1 .63
Exner (1974) CS NR/N/n/N/n/C/NR/m/0/NR/g/0/3/U 181 4 .18
Exner & Andronikof-Sanglade (1992)
Brief, first retest CS 3.5P/n/S/p/C/1.0/lo/25/3/i/0/45/Ma 35 27 .25
Katz Adjustment Scale (Katz & Lyerly, 1963) 7 .44
Brief, second retest CS 10/P/n/S/p/C/1.0/lo/25/3/i/0/45/Ma 35 27 .19
Katz Adjustment Scale 7 .44
Short, first retest CS 10/P/n/S/p/C/1.0/lo/25/3/i/0/45/Ma 35 27 .38
Katz Adjustment Scale 7 .44
Short, second retest CS 25.5/P/n/S/p/C/1.0/lo/25/3/i/0/45/Ma 35 27 .39
Katz Adjustment Scale 7 .44
Gaylin (1966) Standardd 4/N/n/S/f/S/1.0/lo/29/3/i/0/19/Ma 57 13 .19
Glatt & Karon (1974), Psychotherapy subgroup
First retest Standard 6/P/r/S/f/S/1.0/h/27/3/i/0/21/Ma 14 8 .39
Second retest Standard 12/P/r/S/f/S/1.0/h/27/3/i/0/21/Ma 11 8 .18
Third retest Standard, HBBe 20/P/r/S/f/S/1.0/h/27/3/i/0/21/Ma 15 12 .20
Gunderson et al. (1984), second retest Holtf 24/P/n/S/f/S/1.4/h/27/4/i/22/108/Mi 74 4 .20
Visual-Verbal Test (Feldman & Drasgow, 1951) 2 .28
Camarillo Dynamic Assessment Scale (May & 4 .01
Dixon, 1969)
Soskis Attitude Towards Illness Questionnaire 4 .02
(Soskis & Bowers, 1969)
Psychotherapy Outcome Interview (Gunderson 8 .21
& Gomes-Schwartz, 1980)
Haimovitz & Haimovitz (1952),
First retest, posttherapy Own 2.4/F/n/N/f/N/1.5/m/35/3/m/0/10/U 56 10 .27
Second retest, followup Own 15.4/F/n/N/f/N/1.5/m/35/3/m/82/10/U 10 1 .80
(continued)
 RORSCHACH CHANGE ASSESSMENT 267

TABLE 6 (Continued)

Study Scoring System Moderator Codinga N E PP rw TC rw

Hamlin, Berger, & Cummings (1952) Muenchg 15.4/P/n/N/n/S/1.0/m/0/3/i/0/20/Ma 16 2 .77

Kavanagh (1985) DCOS, MOAh 32.7/P/n/N/f/S/4.0/m/32/12/i/0/33/Mi 33 .05
Krout, Krout, & Dulin (1952)
Psychoanalytic group Own 12/P/n/N/n/N/4.0/m/27/12/i/0/37/Ma 19 10 .43
Nonanalytic group Own 13/P/n/N/n/N/1.0/m/27/3/i/0/37/Ma 14 10 .37
LaFrance (1971) Holt 6/P/n/N/n/S/1.0/m/25/1/g/0/52/Mi 18 1 .81
TAT 1 .58
Id Ego Superego Test 1 .81
Behavior rating 1 .80
Mintz, Schmeidler, & Bristol (1956) Clin.ev. 20/P/n/N/f/N/3.0/m/30/9/i/0/1/Ma 20 1 .66
Peterson (1954) Standard 3/N/n/S/n/S/NR/lo/0/NR/i/0/30/Ma 42 12 .08
Peyman (1956) RPRSi 6/N/r/S/n/S/2.0/h/31/2/g/11/58/Mi 32 6 .41
Rice (1973) Standard 2.3/N/n/S/n/S/2.0/lo/28/6/i/0/3/Ma 48 1 .27
Therapist outcome rating 1 .37
Client outcome rating 1 .30
Weiner & Exner (1991)
Short, first retest CS 13/N/n/S/f/C/1.0/lo/23/3/i/0/48/Ma 88 27 .23
Short, second retest CS 29/N/n/S/f/C/1.0/lo/23/3/i/0/48/Ma 88 27 .26
Short, third retest CS 48/N/n/S/f/C/1.0/lo/23/3/i/0/48/Ma 88 27 .28
Long, first retest CS 13/N/n/S/f/C/2.4/lo/26/7/i/0/48/Ma 88 27 .18
Long, second retest CS 29/N/n/S/f/C/2.4/lo/26/7/i/0/48/Ma 88 27 .35
Long, third retest CS 48/N/n/S/f/C/2.4/lo/26/7/i/0/48/Ma 88 27 .43

Note. E = the number of entries coded from the study; PP = pretest to posttest design data; TC = treatment versus control design data; rw = weighted average
sample effect size; first line of each study presents Rorschach data, thereafter other multiple assessment instruments as specified; CS = Comprehensive System;
WAIS = Wechsler Adult Intelligence Scale; TAT = Thematic Apperception Test.
aKey to moderator coding: Retest Period (months)/Concern for Interscorer Reliability (N = not reported; P = partial; F = full)/Randomized Design (n = not

randomized; r = randomized)/Schizophrenia (N = nonschizophrenia; S = schizophrenia)/Blindscoring (n = no blinding; p = partial; f = full)/Scoring System (N =

nonstandard; S = standard; C = CS)/Sessions per Week/Severity of Pathology (lo = low; m = middle; h = high)/Subject Age/Therapy Load/Therapy Type (g =
group; m = mixed; i = individual)/Attrition (percentage)/Number of Scores/Publication Type (U = unpublished; Mi = Minor; Ma = Major). bAllison (A), Blatt (B),
and Zimet (Z; 1988), including Developmental Concept of the Object Scale (DCOS; Blatt, Brenneis, Schimek, & Glick, 1976) and Mutuality of Autonomy Scale
(MOA; Urist, 1977). cGlobal clin.ev. (clinical evaluation) of improvement from test to retest. dVarious basic scores from Rorschach (1921/1942), Beck (1950),
Klopfer, Ainsworth, Klopfer, and Holt (1954), and/or Friedman (1953). eHurvich, Benveniste, and Brodys (HBB; 1998) Annihilation Anxiety scoring. fHolts
(1956) Primary Process scoring. gMuenchs (1947) adjustment and improvement signs. hSee footnote b. iKlopfer et al.s (1954) Rorschach Prognostic Rating
Scale (RPRS).

TABLE 7
Regression Models

Model k R Constant r B SE B

Sample 38 .664 .150

Retest Period .38 .058 .002 .538***
Scorer Blinding .37 .103 .031 .446***
Concern for Interscorer Reliability .15 .132 .049 .410***
Publication Type .05 .072 .039 .269*
Variable
M 17 .561 .141
Retest Period .56 .007 .003 .561**
Color Balance 14 .833 .191
Retest Period .76 .007 .002 .655***
Therapy Load .55 .030 .014 .366*
Form Level 16 .555 .238
Therapy Load .55 .015 .006 .555**
All H Content 13 .587 .080
Therapy Load .59 .050 .021 .587**

Note. N = 1,202. Models were not generated for R, Afr, Form, Shading, EA & EB, r & (2), Organization, Special Scores, or Global. The remaining variables in
Table 3 were drawn from less than five samples. The number of participants the variables were based on was the same as in Table 3. k = the number of samples
contributing data to the analyses; r = the univariate correlation between the effect size and the moderator.
*p < .10. **p < .05. ***p < .01.
268 GRNNERD
Reliability and lower in magnitude in studies coded with
higher levels of Blindscoring. Finally, increasing values on
the Publication Type moderator was related to slight in-
creases in effect size levels. In a control analysis based only
on the final retest interval, a rather different model was re-
turned including Randomized Design. The model was some-
what overfitted, though, based on the moderators relatively
high standard error (.090) and restricted range and rather un-
likely predicted values. Whether the control model suggested
that moderator nonindependence did cause problems is
therefore uncertain, but some caution should probably be ex-
ercised in the interpretation of the main model.
For the specific variables, Retest Period and/or Therapy
Load was a factor in all four variable models. Increasing re-
test periods and increasing intensity, as defined by Therapy
Load, was associated with increased effect size magnitudes
except for one case. Form Level effect sizes seemed to de-
crease with increasing therapy load, a finding that should be
interpreted with caution. The raw data indicated that modera-
tor dependence was a problem in this particular variable be-
cause three of the four included studies each contributed data
from two or three retest intervals.
Table 8 presents the predicted levels obtained by entering
ideal values for the included moderators based on the ob-
tained regression models in Table 7. The ideal value for
Therapy Load was defined as individual therapy twice a
week. Retest Period, which was effectively defined as ther-
apy duration in this context, was defined as brief therapy of 6
months, short-term therapy of 1 year, and long-term therapy
of 2 years. Publication Type was defined as a major journal,
and Concern for Interscorer Reliability and Blindscoring
were both set to the highest level.
TABLE 8
Observed Effect Size Levels for Final
Retests and Predicted Effect Size Levels
Based on the Regression Models
Predicted
Model Observed rw 6m 1y 2y
Sample .25 .39 .42 .49
Variable
M .30 .18 .22 .30
Color Balance .49 .53 .57 .65
Form Level .12 .15
All H Content .34 .38
Note. Sample average drawn from Table 2 and variable averages from
Table 3. rw = weighted averages from final retests; 6 m = a 6-month retest
period; 1y = 1 year; 2 y = 2 years. Values entered in sample model: Concern
for Interscorer Reliability = 2 (full control); Blindscoring = 2 (full blinding);
Publication Type = 2 (major journal); Retest Period as specified in column
header. Values entered in the variable models: Sessions per Week = 2;
Therapy Type = 2 (individual therapy), resulting in Therapy Load = 6; Retest
Period entered as specified in column header. Levels for the models that did
not include Retest Period as a moderator are presented in the 6 m column.
The predicted increases in levels of change were sub-
stantial and impressive. The model for the sample-level re-
sults predicted an increase up to r = .49 after 2 years.
Human movement (M) and Color Balance both showed
steady increases with time, a finding among the most inter-
esting in this analysis.
Summary
The most important and clear finding from the regression
analyses of moderator influences was that increasing length
and intensity of the therapy was associated with higher de-
grees of personality changes. Such a finding is important and
portrays a meaningful development as therapy progresses.
The validity of the results from the global estimates of
change (Study 1) is thus strengthened even further.
Second, two important factors related to methodology
were revealed. A higher level of Concern for Interscorer
Reliability was associated with larger change estimates, in-
dicating that the authors who conscientiously attended to
the reliability of their data seemed to be rewarded by over-
all higher levels of validity. The other factor demonstrated
a methodological pitfall. A higher degree of blindscoring
was associated with smaller effect sizes, which could very
well be explained by expectancy effects. When scorers or
judges know which protocol is taken pretherapy and which
is taken posttherapy, it seems that the postprotocols were
rated more favorably than preprotocols, thereby inflating
the degree of change. The relationship between Publication
Type and effect size levels could also be taken as an indica-
tion of methodological quality, although only indirectly.
One can assume that peer-reviewed journals publish studies
of higher quality than non-peer-reviewed publications. The
question is whether the difference between major and mi-
nor journals can be interpreted in the same way. There
should be no reason to doubt the standard of the journals
categorized as major, however, and the same interpretation
is therefore reasonable.
The predicted levels of change over therapy duration
showed impressive gains in effect sizes. Especially the sam-
ple model result was impressive because the model included
a negative factor in Blindscoring. A note of caution should be
made about extending the predictions too far. The assump-
tion of a linear relationship may not hold and especially not
when the relationship between time and effect size is
stretched beyond the time scope of these studies. Other stud-
ies have in fact indicated a logarithmic relationship between
dose and effect, with larger gains in the beginning of treat-
ment and then smaller and smaller gains as therapy pro-
gresses (Howard, Moras, Brill, Martinovitch, & Lutz, 1996).
This data was not extensive and coherent enough to allow a
thorough examination of possible curvilinear relationships.
The results do show, however, that when retest period and
specific methodological issues are considered, the Ror-
schach may yield substantial effect sizes over time.
 RORSCHACH CHANGE ASSESSMENT
DISCUSSION
In this meta-analysis of studies using the Rorschach method
to assess personality changes following psychotherapy, the
method showed moderate sensitivity. Overall results were
strong. A combined sample effect size of rw = .26 is a moder-
ate effect size according to Hemphills (2003) guidelines but
still substantial in this context. Meyer et al. (2001) noted that
effect sizes from a wide selection of psychological assess-
ment studies based on appropriate criteria and without
method confounds typically fall between .15 and .30. Meyer
et al. (2001) argued that one generally should be pleased
(p. 134) when one encounters effect sizes around .30. Other
meta-analyses of psychotherapy studies have yielded much
the same effect size magnitudes as these Rorschach results as
shown in the beginning of the article. These results also show
that the Rorschach method seemed to perform as well as
other instruments. The authors of the original studies clearly
found the selection of instruments to be relevant and useful,
most likely because their selection provided a multifaceted
account of changes in various personality domains. It is en-
couraging to find that the Rorschach method clearly also had
its place. From the perspective of this analysis, the negative
reputation that the Rorschach method has acquired within the
field of psychotherapy research should be reconsidered.
These samples are based on a variety of participants with
varying types of pathology, with very different types of ther-
apies of varying duration, and with a variety of predictors and
designs. They represent a selection of clinically representa-
tive studies mostly being conducted with naturalized sam-
ples from everyday clinical practice. This feature is a
strength concerning the generalizability of the results. The
fail-safe N analysis showed that 790 unpublished studies
with null results need to be included before these results are
rendered nonsignificant, another indication that the findings
are sturdy.
The predicted levels obtained from the regression mod-
els inspire further confidence in the results. Substantial in-
creases were found in effect size magnitudes when retest
periods and methodological features were considered. Most
important, longer and more intensive psychotherapy
yielded larger effect sizes than shorter and less intensive
therapy. This finding is in line with the broader psychother-
apy research literature, which has generally concluded that
longer therapies are associated with greater improvement
(Orlinsky et al., 1994). Howard, Kopta, Krause, and
Orlinsky (1986) showed that increasing doses of therapy re-
sulted in higher ratings of satisfactory outcome. The rela-
tionship was logarithmic, indicating that the quickest gains
were achieved in the early phases of therapy, and then the
gains are smaller and smaller as therapy length increases.
Howard, Lueger, Maling, and Martinovitch (1993) concep-
tualized the change pattern in the phase model of psycho-
therapy. The doseeffect relationship has been confirmed in
another important data set, the Consumer Reports study
269
(Seligman, 1995, 1996). Studies included in this analysis
that reported several retests for various samples (Exner &
Andronikof-Sanglade, 1992; Gunderson et al., 1984;
Haimowitz & Haimowitz, 1952; Weiner & Exner, 1991)
also lend strong support to this conclusion in which the in-
crease in effect sizes with longer therapy duration was
clearly demonstrated within the same groups. Only one
study (Glatt & Karon, 1974) defied this trend in which the
first retest reached the highest level and then sinking from r
= .39 to .18 and finally .20. The proportion of missing data
was relatively high in this material, which could account
for the somewhat atypical pattern.
Much of this data is based on pretest to posttest compari-
son designs, and I showed in a previous study (Grnnerd,
2003) that temporal stability decreases with increasing test
intervals. The largest portion of the data showed changes in
the direction predicted by the examiners, however, thus con-
tradicting the assertion that low temporal stability should be
related to larger effect sizes. Such a result would be unlikely
if changes were unrelated to therapeutic interventions.
Rather, one would see a picture of changes going in either
way despite the predicted directions. I therefore conclude
that temporal stability is not a major concern in this context.
Clinicians who are using the Rorschach method in their
daily work might find the substantial effect sizes obtained for
construct evaluations and global clinical evaluations of par-
ticular interest. Clinical use of the Rorschach method has of-
ten been criticized for lacking scientific credibility and even
being speculative. These data contradict such a general asser-
tion, and show that proper, conceptually driven, theoretically
based use of the Rorschach can yield highly valid results. The
locus of responsibility is thus shifted away from the Ror-
schach as a method and onto the user who must carry the re-
sponsibility of treating Rorschach data according to
clinically and scientifically sound principles.
On the other hand, the diversity of the included studies
makes this analysis too diffuse to say which types of therapy,
of what length, and for what kind of disorders the Rorschach
method is most sensitive. Very few studies have used the
Rorschach method to assess therapy progress, a surprisingly
low number compared with the large amount of research on
the Rorschach method in general and compared with the
methods widespread use in clinical settings. In this analysis,
the low number of samples resulted in a clustering of certain
types of data within single samples, for example, that object
relations data came from only two studies, and almost all CS
data from two others. The regression models were in many
cases somewhat fragile. During the course of this work, it
was noticeable how small changes in the data selection or
type of analysis caused the models to change quite quickly.
This posed a particular problem for the variable-level analy-
ses in Tables 3, 7, and 8 for which the data was the most
scarce. A reasonable question to ask is therefore whether the
variation in the levels of change across different variables as
seen in Table 3 is a result of true differences or a lack of data.
270 GRNNERD
Looking at the substantial interpretation of the various
variables, identifying patterns is difficult, probably also due
to the restricted amount of data. One also needs to consider
the possibility that some personality characteristics may
change but are not very well captured by Rorschach vari-
ables. With these problems in mind, a gross and tentative
grouping of the variables can be made.
In the lower half of Table 3, we find several variables re-
lated to self-concept, self-perception, and interpersonal rela-
tions such as reflection and pairs (r & (2)), Depth, Object
Relations, and Active and Passive movement (a & p). The
exceptions were Human Movement (M), Human Content
(All H Content), and Shading, which were in the range of .30
to .34. Human Movement also showed a strong increase in
predicted levels of change with time (Table 8).
In the upper half of Table 3, one finds nearly all variables
related to stress and control capacity on one hand and affect
on the other. Variables related to stress and coping, such as
the D score, Coping Deficit Index (CDI), and Adjusted D
score (Adj D), were in the upper portion of the list. An even
more marked feature was the clustering of affective features
in the upper half, with the sole exception of the weighted
Color scores (SumC). Color Balance, Form Color (FC),
Stimulation (SumShd > FM + m), Affective Ratio (Afr), and
partly Shading and the Depression Index (DEPI) all dis-
played large effect sizes. In addition, Color Balance showed
a strong gain in predicted degree of change over time (Table
8).
The remaining variables in Table 3 related to information
processing (W & D, Form, Organization) as well as
mediational and ideational aspects of personality (Populars,
Form Level, FM + m, Special Scores). These variables were
more spread across the range of effect size magnitudes and
did not seem to form any particular pattern.
The clustering may suggest that features related to self
perception and to some extent interpersonal relationships are
less susceptible to change than affective and coping features.
Interpersonal relationships and self-perceptions are funda-
mental features of personality, and problems related to these
areas often form core issues in long-term therapies. Given the
limited average length of the therapies, less change could and
perhaps even should be expected for these variables. The fact
that Human Movement started out on a relatively low level
and then showed increases with time in treatment may sug-
gest, however, that ideational features of interpersonal rela-
tionships are susceptible to changes given enough time. If
this interpretation is feasible, then thinking differently about
relations to other people may in turn provide the basis for
changing perceptions about ones self, which may be a much
slower process. These features correspond to
characterological symptoms in the phase model of psycho-
therapy outcome (Howard et al., 1993), which showed a
slower recovery rate than acute and chronic distress.
Helping a person to handle affects and stress more effi-
ciently, on the other hand, should be a more manageable task
in shorter therapies, which could explain the higher effect
sizes found for these variables. Affect and stress are more at
the front of the clinical picture and therefore more easily ac-
cessible for the patient and the therapist. Again, this corre-
sponds quite well with the phase model, which showed the
most rapid recovery rates for acute distress and then some-
what slower for chronic distress symptoms. The increase in
predicted levels for Color Balance might suggest that therapy
might be quite effective in bringing affects under more delib-
erate control and to increase the efficiency of secondary pro-
cesses. In more cognitive terms, the patient is better able to
cope with stress, anxiety, and other emotions.
A further note should be taken of the predicted increases
in M and Color Balance over time. Initially, this result might
suggest a link to Hermann Rorschachs conceptualizations of
Erlebnistypus (Rorschach, 1921/1942) and to current con-
ceptualizations of Experience Actual (EA; Exner, 2003). If
one looks at the components of EA, which are M and SumC,
one finds, however, that SumC showed a weak effect. SumC,
and thereby EA, has been criticized for being a problematic
indicator of problem-solving resources (Lajoie, 1998), and
these results may underline this criticism. If one is to infer
that therapy leads to improved functioning, which should be
a reasonable inference, it seems that Color Balance would be
a better conceptualization of affective coping resources than
SumC.
For specific variables examined in this study and in
Grnnerd (2003), the effect sizes for changes due to treat-
ment were correlated with the effect size for temporal stabil-
ity. The correlation was close to zero. The temporal stability
levels for this group of variables were high, however. Except
for variables reflecting state-like features (FM + m, Shading)
and Animal Content, all stability coefficients ranged from
.70 to .84, in many cases with even higher predicted levels
based on regression models with moderators. To compare
more directly, temporal stability levels for all types of vari-
ables in a scoring system are needed. This is especially perti-
nent for those variables that are important for interpretations
and for the CS with its widespread use and reputation for sci-
entific vigor. Even more important, the reliability of each
variable, both interscorer reliability and temporal stability,
should be considered within each study when data is ana-
lyzed. Only this can give a direct assessment of how reliabil-
ity levels affect effect size magnitudes.
These results can be used as a starting ground for improve-
ments in methodology. The particular challenges facing re-
searchers using the Rorschach method have been pertinently
pointed out several times (e.g., Exner, 1995), and these re-
sults add further weight to the importance of these concerns.
Researchers should ensure that proper blinding is applied in
administration and scoring. Expectancy effects are well doc-
umented in drug treatment studies (Fisher & Greenberg,
1997) and in other contexts (Rosenthal & Rubin, 1978),
showing that researchers have a tendency to find what they
are looking for. These results also gives the same suggestion.
 RORSCHACH CHANGE ASSESSMENT
Ensuring proper blinding of testretest information will most
likely result in lower effect size magnitudes, but the results
will also be more valid.
Next, the reliability of the data should be secured by a
proper interscorer reliability analysis. The reliability should
also be incorporated into the analyses, for example, by dis-
carding variables with low reliability. The recent debate con-
cerning interscorer reliability in Rorschach studies (e.g.,
Meyer, 1997a, 1997b, 1999b; Meyer et al., 2002; Wood et
al., 1996; Wood, Nezworski, & Stejskal, 1997) has been
helpful in highlighting the importance of securing reliability,
and methods have been presented for doing so. The problem
should therefore be smaller today than when many of the
studies included in this analysis were published.
Two of the included studies, Exner and
Andronikof-Sanglade (1992) and Weiner and Exner (1991),
deserve special attention and can serve as illustrations of a
general point. These studies are important because they
have been written by central figures behind the CS and
therefore serve as models for other studies (e.g., Abraham,
Lepisto, Lewis, Schultz, & Finkelberg, 1994; Carlson &
Lindgren, 2002; de Ruiter & Hildebrand, 2000). They
should be rewarded for their general focus on clinically im-
portant aspects of the data, but concerns can also be raised.
In these studies, the number and percentage of persons in a
group who were outside a cutoff point on a specific vari-
able on the first test were presented. This figure was then
compared with the same number and percentage on retest
on the same variable. In this way, a dichotomy was created
based on whether a person was inside or outside an area as-
sumed to have interpretational importance. Significant dif-
ferences between the groups were tested by a chi-square
analysis with correction for continuity. The results of the
analyses were represented as probability limits for each
variable.
First, applying a type of analysis in which the participants
to a larger extent are their own controls is preferable rather
than only comparing group means or frequencies. More em-
phasis is thus put on idiographical aspects of the data, which
should increase the clinical validity of the results. In psycho-
therapy research, this is very important given the highly indi-
vidual configuration of pathology and personality changes.
The possibility of individual changes canceling each other at
the group level should be constricted. If two participants
cross the cutoff point in opposite directions from test to re-
test, the resulting group change will be zero. Data from the
stability analysis also clearly suggested the advantages of
testretest correlations over other statistics in which only
group differences are tested. Second, general cutoff points
might not be relevant for the participants at hand. The more
severe the pathology, the more problematic this can be, as
initial deviations from cutoff points could be larger. This ef-
fectively reduces the statistical power of the study. Given the
quite strong results in the mentioned studies, this remains a
principal objection, but other studies that have followed the
271
same analysis have failed to obtain significant results.
Another objection can also be raised. When less seriously
impaired participants are studied, a larger portion of the
group will be expected to move across the cutoff points. The
observed changes might be too minute to have any clinical
significance, or the changes might be caused by random fac-
tors. Reporting the magnitude of the observed changes is
therefore important also, as Abraham et al. (1994) did (this
study examined adolescents and was therefore not included
in this analysis). Otherwise, depending on the actual partici-
pants and the choice of cutoff levels, the possibility of both
Type I and Type II errors are immanent in these types of anal-
yses.
It is advisable that the units of analysis are as close as pos-
sible to clinically important aspects of the data. To enhance
methodology in this field further, more care should be taken
to develop methods to assess changes that focus on the clini-
cal usefulness of selected elements in the Rorschach method.
These methods should consider many structural and substan-
tial aspects of the data. First, the distributional characteristics
are important. Skewness reduces testretest correlations
(Dunlap, Chen, & Greer, 1994), and this needs to be ac-
counted for in temporal stability studies and possibly also in
change assessments. The reliability of the variable is impor-
tant, and the analysis should directly incorporate the effects
of interscorer reliability and temporal stability. This could be
done according to classic reliability theory as a correction for
attenuation or perhaps more appropriately according to
generalizability theory (e.g., Shavelson & Webb, 1991). On
the substantial side, such methods should take both clinically
meaningful cutoff points, the magnitude of the change, and
the expected change for any given individual into account
(for a recent review of such methods, see Wise, 2004). The
expected pattern of changes based on each persons individ-
ual personality configuration should be considered, espe-
cially because of the Rorschach methods alleged ability to
capture these configurations.
These analyses do not suggest that the Rorschach method
necessarily should be the preferred instrument in psychother-
apy studies. What can be concluded, however, is that these
data show that the Rorschach method is not inferior in this
context. It is a time-consuming method, and in large-scale
projects, this might be a decisive factor. The relatively low
number of participants in each of these samples might also
speak of this concern. On the other hand, it is a question of
gains and losses related to the purpose at hand. Several au-
thors of these studies have pointed to the important fact that
different participants change differently according to their
psychopathology and that the Rorschach method meaning-
fully captures the individual nature of these changes. Given
the positive results of this study, the Rorschachs idiographic
nature does not come at the expense of more quantitative use.
Rather, they are complementary aspects of the same method,
which should encourage more use of the Rorschach method
in psychotherapy research in the future.
272 GRNNERD
ACKNOWLEDGMENTS
A paper with a brief outline of an early version of this study
was presented at the Sixteenth International Congress of Ror-
schach and Projective Methods in Rome, Italy, September
2002. The results were presented at the seventh European
Rorschach Association Congress in Stockholm, Sweden,
August 2004. I thank Ellen Hartman for valuable comments
on the text and Monica Martinussen for good advice on
meta-analytic techniques at the start of this project. Jukka
Nyblom provided valuable help with some of the raw data
analyses and suggested the weighting in the intermethod
analyses. Stine Vogt provided useful language corrections for
an early version of this manuscript. Sarah Elisabeth Zeld
kindly did the actual job of recovering the Glatt and Karon
(1974) data from the archives.
REFERENCES
References marked with an asterisk indicate studies included in the
meta-analysis.
Abraham, P. P., Lepisto, B. L., Lewis, M. G., Schultz, L., & Finkelberg, S.
(1994). An outcome study: Changes in Rorschach variables of adoles-
cents in residential treatment. Journal of Personality Assessment, 62,
505514.
Acklin, M. W. (1999). Behavioral science foundations of the Rorschach
Test: Research and clinical applications. Assessment, 6, 319326.
Acklin, M. W., McDowell, C. J., Verschell, M. S., & Chan, D. (2000).
Interobserver agreement, intraobserver reliability, and the Rorschach
Comprehensive System. Journal of Personality Assessment, 74,
1547.
Allison, J., Blatt, S. J., & Zimet, C. N. (1988). The interpretations of psycho-
logical tests. Washington, DC: Hemisphere.
Atkinson, L., Quarrington, B., Alp, I. E., & Cyr, J. J. (1986). Rorschach va-
lidity: An empirical approach to the literature. Journal of Clinical Psy-
chology, 42, 360362.
*Baker, L. C. (1998). Rorschach assessment in borderline personality disor-
der: A follow-up study. Unpublished doctoral dissertation, Michigan State
University, East Lansing.
*Barendregt, J. T. (1957). A psychological investigation of the effect of
group psychotherapy in patients with bronchial asthma. Journal of Psy-
chosomatic Research, 2, 115119.
Barry, J. R., Blyth, D. D., & Albrecht, R. (1952). Relationships between
Rorschach scores and adjustment level. Journal of Consulting Psychol-
ogy, 16, 3036.
Beck, S. J. (1950). Rorschachs Test: Vol. 1. Basic processes (2nd ed.). New
York: Grune & Stratton.
Bergin, A. E., & Garfield, S. L. (1994). Handbook of psychotherapy and be-
havior change (4th ed.). New York: Wiley.
Berman, A. L. (1972). Videotape self-confrontation of schizophrenic ego
and thought processes. Journal of Consulting and Clinical Psychology,
39, 7885.
Blatt, S. J., Brenneis, C. B., Schimek, J. G., & Glick, M. (1976). Normal
developement and psychopathological impairment of the concept of the
object on the Rorschach. Journal of Abnormal Psychology, 85, 364373.
*Blatt, S. J., Ford, R. Q., Berman, W., Cook, B., & Meyer, R. (1988). The as-
sessment of change during the intensive treatment of borderline and
schizophrenic young adults. Psychoanalytic Psychology, 5, 127158.
Bornstein, R. F. (1998). Interpersonal dependency and physical illness: A
meta-analytic review of retrospective and prospective studies. Journal of
Research in Personality, 32, 480497.
Bornstein, R. F. (1999). Criterion validity of objective and projective de-
pendency tests: A meta-analytic assessment of behavioral prediction. Psy-
chological Assessment, 11, 4857.
Cadman, W. H., Misbach, L., & Brown, D. V. (1954). An assessment of
roundtable therapy. Psychological Monographs, 68, 384.
Campbell, D. T., & Kenny, D. A. (1999). A primer on regression artifacts.
New York: Guilford.
*Campo, V., Dow, N., & Tuset, A. (1988). Rorschach, O.R.T. and follow-up.
British Journal of Projective Psychology, 33(2), 3153.
Carlson, A. M., & Lindgren, T. (2002, September). Methodological prob-
lems when using the Rorschach in the measurement of psychotherapy out-
come. Paper presented at the 17th International Congress of Rorschach
and Projective Methods, Rome, Italy.
*Carr, A. C. (1949). An evaluation of nine nondirective psychotherapy cases
by means of the Rorschach. Journal of Consulting Psychology, 13,
196205.
Cohen, J. (1988). Statistical power analysis or the behavioral sciences.
Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
*Coons, W. H. (1957). Interaction and insight in group psychotherapy. Ca-
nadian Journal of Psychology, 11, 18.
*Coons, W. H., & Peacock, E. P. (1970). Interpersonal interaction and per-
sonality change in group psychotherapy. Canadian Psychiatric Associa-
tion Journal, 15, 347355.
Corel. (2000). Quattro Pro (Version 9.0 for Windows) [Computer software].
Ottawa, Ontario, Canada: Author.
Curran, P. J., West, S. G., & Finch, J. F. (1996). The robustness of test statis-
tics to nonnormaility and specification error in confirmatory factor analy-
sis. Psychological Methods, 1, 1629.
de Ruiter, C., & Hildebrand, M. (2000, September). The Rorschach in the as-
sessment of treatment outcome in forensic psychiatric patients. Paper pre-
sented at the 6th Congress of the European Rorschach Association, Paris,
France.
Doherty, J. P. (2001). The assessment of annihilation anxiety in the psycho-
therapy of schizophrenia. Unpublished doctoral dissertation, Central
Michigan University, Mount Pleasant.
Dombrose, L. A., & Slobin, M. S. (1958). The IES test. Perceptual and Mo-
tor Skills, 8, 347389.
*Dudek, S. Z. (1970). Effects of different types of therapy on the personality
as a whole. Journal of Nervous and Mental Disease, 150, 329345.
Dunlap, W. P., Chen, R., & Greer, T. (1994). Skew reduces test-retest reli-
ability. Journal of Applied Psychology, 79, 310313.
Endicott, N. A., & Endicott, J. (1963). Improvement in untreated psychiat-
ric patients. Archives of General Psychiatry, 9, 575585.
*Exner, J. E., Jr. (1974). The Rorschach: A Comprehensive System: Vol. 1.
Basic foundations. New York: Wiley.
Exner, J. E., Jr. (1978). The Rorschach: A Comprehensive System: Vol. 2.
Current research and advanced interpretation. New York: Wiley.
Exner, J. E., Jr. (1986). The Rorschach: A Comprehensive System: Vol 1. Ba-
sic foundations (2nd ed.). New York: Wiley.
Exner, J. E., Jr. (Ed.). (1995). Issues and methods in Rorschach research.
New York: Lawrence Erlbaum Associates, Inc.
Exner, J. E., Jr. (2003). The Rorschach: A Comprehensive System: Vol. 1. Ba-
sic foundations and principles of interpretation (4th ed.). New York:
Wiley.
*Exner, J. E., Jr., & Andronikof-Sanglade, A. (1992). Rorschach changes
following brief and short-term therapy. Journal of Personality Assess-
ment, 59, 5971.
Fairweather, G., Simon, R., Beghard, M., Weingarten, E., Holland, J.,
Sanders, R., et al. (1960). Relative effectiveness of psychotherapeutic pro-
grams: A multicriteria comparison of four programs for three different pa-
tient groups. Psychology Monographs, 74, 171185.
Feldman, M. J., & Drasgow, J. A. (1951). A visual verbal test for schizophre-
nia. Psychiatric Quarterly Supplement, 25, 5564.
Field, A. P. (2001). Meta-analysis of correlation coefficients: A Monte Carlo
comparison of fixed- and random-effects methods. Psychological
Methods, 6, 161180.
 RORSCHACH CHANGE ASSESSMENT
Fisher, S., & Greenberg, R. P. (1997). The curse of the placebo: Fanciful pur-
suit of a pure biological therapy. In S. Fisher & R. P. Greenberg (Eds.),
From placebo to panacea (pp. 356). New York: Wiley.
Fishman, D. B. (1973). Rorschach adaptive regression and change in psy-
chotherapy. Journal of Personality Assessment, 37, 218244.
Friedman, H. (1953). Perceptual regression in schizophrenia. An hypothesis
suggested by the use of the Rorschach test. Journal of Projective Tech-
niques, 17, 171185.
Garb, H. N. (1999). Call for a moratorium on the use of the Rorschach Ink-
blot Test in clinical and forensic settings. Assessment, 6, 313317.
Garb, H. N., Florio, C. M., & Grove, W. M. (1998). The validity of the Ror-
schach and the Minnesota Multiphasic Personality Inventory: Results
from meta-analyses. Psychological Science, 9, 402404.
Garb, H. N., Florio, C. M., & Grove, W. M. (1999). The Rorschach contro-
versy: Reply to Parker, Hunsley, and Hanson. Psychological Science, 10,
293294.
Garb, H. N., Wood, J. M., & Nezworski, M. T. (2000). Projective techniques
and the detection of child sexual abuse. Child Maltreatment: Journal of
the American Professional Society on the Abuse of Children, 5, 161168.
Garfield, S. L. (1978). Research on client variables in psychotherapy. In A.
E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behav-
ior change (3rd ed.). New York: Wiley.
Garfield, S. L. (1994). Research on client variables in psychotherapy. In A.
E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behav-
ior change (4th ed.). New York: Wiley.
*Gaylin, N. L. (1966). Psychotherapy and psychological health: A Ror-
schach function and structure analysis. Journal of Consulting Psychology,
30, 494500.
Gibson, R. L., Snyder, W. U., & Ray, W. S. (1955). The counseling process.
Journal of Counseling Psychology, 2, 8390.
*Glatt, C. T., & Karon, B. P. (1974). A Rorschach validation study of the ego
regression theory of psychopathology. Journal of Consulting and Clinical
Psychology, 42, 569576.
Goldfried, M. R., Stricker, G., & Weiner, I. B. (1971). Rorschach handbook
of clinical and research applications. Englewood Cliffs, NJ:
Prentice-Hall.
Grissom, R. J. (1996). The magical number .7 .2: Meta-meta-analysis of
the probability of superior outcome in comparisons involving therapy,
placebo, and control. Journal of Consulting and Clinical Psychology, 64,
973982.
Grnnerd, C. (1999). Rorschach interrater agreement estimates: An empir-
ical evaluation. Scandinavian Journal of Psychology, 40, 115120.
Grnnerd, C. (2003). Temporal stability in the Rorschach method: A
meta-analytic review. Journal of Personality Assessment, 80, 272293.
Grove, W. M., Andreassen, N. C., McDonald-Scott, P., Keller, M. B., &
Shapiro, R. W. (1981). Reliability studies of psychiatric diagnosis. Ar-
chives of General Psychiatry, 38, 408413.
*Gunderson, J. G., Frank, A. F., Katz, H. M., Vannicelli, M. L., Frosch, J. P.,
& Knapp, P. H. (1984). Effects of psychotherapy in schizophrenia: II.
Comparative outcome of two forms of treatment. Schizophrenia Bulletin,
10, 564598.
Gunderson, J. G., Gomes-Schwartz, B. (1980). The quality of outcome from
psychotherapy of schizophrenia. In M. B. B. J. S. Strauss, Jr., T. W.
Downey, S. Fleck, S. Jackson, & I. Levine (Eds.), The psychotherapy of
schizophrenia (pp. 257275). New York: Plenum.
Gunderson, J. G., Kolb, J. E., & Austin, Y. (1981). The diagnostic inter-
view for borderline patients. American Journal of Psychiatry, 142,
277288.
*Haimowitz, N., & Haimowitz, M. (1952). Personality changes in cli-
ent-centered psychotherapy. In W. Wolff & J. A. Precker (Eds.), Success
in psychotherapy (pp. 6393). New York: Grune & Stratton.
Hamlin, R. M., & Albee, G. W. (1948). Muenchs tests before and after
nondirective therapy: A control group for his subjects. Journal of Con-
sulting Psychology, 12, 412416.
*Hamlin, R. M., Berger, B., & Cummings, S. T. (1952). Changes in adjust-
ment following psychotherapy as reflected in Rorschach signs. In W.
273
Wolff & J. A. Precker (Eds.), Success in psychotherapy (pp. 94111).
New York: Grune & Stratton.
Harrower, M. (1958). Personality change and development, as measured by
projective techniques. New York: Grune and Stratton.
Harrower, M. (1965). Psychodiagnostic testing: An empirical approach:
Based on a follow-up of 2000 cases. Springfield, IL: Thomas.
Harty, M. Cerney, M., Colson, D., Coyne, L., Freiswyk, S., Johnston, S., et
al. (1981). Correlates of change and long-term outcome for intensively
treated hospital patients: An exploratory study. Bulletin of the Menninger
Clinic, 45, 209228.
Hathaway, S. R., & McKinley, J. C. (1943). The Minnesota Multiphasic Per-
sonality Inventory. Minneapolis: University of Minnesota Press.
Hemphill, J. F. (2003). Interpreting the magnitudes of correlation coeffi-
cients. American Psychologist, 58, 7879.
Hiller, J. B., Rosenthal, R., Bornstein, R. F., Berry, D. T. R., &
Brunell-Neuleib, S. (1999). A comparative meta-analysis of Rorschach
and MMPI validity. Psychological Assessment, 11, 278296.
Holt, R. R. (1956). Gauging primary and secondary processes in Rorschach
responses. Journal of Projective Techniques, 20, 1425.
Howard, K. I., Kopta, S. M., Krause, M. S., & Orlinsky, D. E. (1986). The
dose-effect relationship in psychotherapy. American Psychologist, 41,
159164.
Howard, K. I., Lueger, R. J., Maling, M. S., & Martinovitch, Z. (1993). A
phase model of psychotherapy outcome: Causal mediation of change.
Journal of Consulting and Clinical Psychology, 61, 678685.
Howard, K. I., Moras, K., Brill, P. L., Martinovitch, Z., & Lutz, W. (1996).
Evaluation of psychotherapy: Efficacy, effectiveness, and patient prog-
ress. American Psychologist, 51, 10591064.
Hurvich, M., Benveniste, P., & Brody, S. (1998). Scoring manual for the
RCS. Unpublished manuscript.
Jensen, A. R. (1959). The reliability of projective techniques: Review of the
literature. Acta Psychologica, Amsterdam, 16, 108136.
Kamil, L. J. (1970). Psychodynamic changes through systematic desensiti-
zation. Journal of Abnormal Psychology, 76, 199205.
Kantrowitz, J. L., Katz, A. L., & Paolitto, F. (1990). Follow-up of psycho-
analysis five to ten years after termination: I. Stability of change. Journal
of the American Psychoanalytic Association, 38, 471496.
Kantrowitz, J. L., Katz, A. L., Paolitto, F., Sashin, J., & Solomon, L. (1987a).
Changes in the level and quality of object relations in psychoanalysis: Fol-
low-up of a longitudinal, prospective study. Journal of the American Psy-
choanalytic Association, 35, 2346.
Kantrowitz, J. L., Katz, A. L., Paolitto, F., Sashin, J., & Solomon, L.
(1987b). The role of reality testing in psychoanalysis: Follow-up of 22
cases. Journal of the American Psychoanalytic Association, 35, 367385.
Kantrowitz, J. L., Paolitto, F., Sashin, J., Solomon, L., & Katz, A. L. (1986).
Affect availability, tolerance, complexity, and modulation in psychoanal-
ysis: Follow-up of a longitudinal, prospective study. Journal of the Ameri-
can Psychoanalytic Association, 34, 529559.
Karon, B. P., & Vandenbos, G. R. (1972). The consequences of psychother-
apy for schizophrenic patients. Psychotherapy, 9, 111119.
Katz, M. M., & Lyerly, S. (1963). Methods for measuring adjustment and so-
cial behavior in the community. Psychological Reports, 13, 503535.
*Kavanagh, G. G. (1985). Changes in patients object representations during
psychoanalysis and psychoanalytic psychotherapy. Bulletin of the
Menninger Clinic, 49, 546564.
Klopfer, B., Ainsworth, M., Klopfer, W., & Holt, R. (1954). Developments
in the Rorschach technique: Vol. 1. Technique and theory. New York:
World Book.
*Krout, J., Krout, M. H., & Dulin, T. J. (1952). Rorschach test-retest as a
gauge of progress in psychotherapy. Journal of Clinical Psychology, 8,
380384.
*LaFrance, S. C. (1971). Developing ego strength in drug dependent persons
through group therapy. Correctional Psychologist, 4(5), 171182.
Lajoie, G. (1998, August). The uncertain validity of the D ScoreA demon-
stration. Paper presented at the 5th Congress of the European Rorschach
Association, Madrid, Spain.
274 GRNNERD
Lambert, M. J., & Hill, C. E. (1994). Assessing psychotherapy outcome and
process. In A. E. Bergin & S. L. Garfield (Eds.), Handbook of psychother-
apy and behavior change (4th ed.). New York: Wiley.
Lerner, B. (1972). Therapy in the ghetto: Political impotence and personal
disintegration. Baltimore: Johns Hopkins Press.
Lerner, B., & Fiske, D. W. (1973). Client attributes and the eye of the be-
holder. Journal of Consulting and Clinical Psychology, 40, 272277.
Lerner, P. M. (1975). Handbook of Rorschach scales. New York: Interna-
tional Universities Press.
Lindberg, D. (1981). A controlled study of 5 years treatment with psycho-
therapy in combination with depot neuroleptics in schizophrenia. Acta
Psychiatrica Scandinavica: Supplementum, 289, 5566.
Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, edu-
cational, and behavioral treatment. American Psychologist, 48,
11811209.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand
Oaks, CA: Sage.
Martin, D. J., Garske, J. P., & Davis, M. K. (2000). Relation of the therapeu-
tic alliance with outcome and other variables: A meta-analytic review.
Journal of Consulting and Clinical Psychology, 68, 438450.
May, P. R. A., & Dixon, W. J. (1969). The Camarillo Dynamic Assessment
Scale. Bulletin of the Menninger Clinic, 33, 135.
McDowell, C., & Acklin, M. W. (1996). Standardizing procedures for calcu-
lating Rorschach interrater reliability: Conceptual and empirical founda-
tions. Journal of Personality Assessment, 66, 308320.
McReynolds, P., & Ferguson, J. T. (1953). Clinical manual for the Hospital
Adjustment Scale. Palo Alto, CA: Leland Stanford University Press.
Meltzoff, J., & Kornreich, M. (1970). Research in psychotherapy. New York:
Atherton.
Meyer, G. J. (1997a). Assessing reliability: Critical corrections for a critical
examination of the Rorschach Comprehensive System. Psychological As-
sessment, 9, 480489.
Meyer, G. J. (1997b). Thinking clearly about reliability. More critical cor-
rections regarding the Rorschach Comprehensive System. Psychological
Assessment, 9, 495498.
Meyer, G. J. (1999a, July). A cross national review of Rorschach interscorer
reliability. Paper presented at the 16th International Congress of Ror-
schach and Projective Methods, Amsterdam, The Netherlands.
Meyer, G. J. (1999b). Simple procedures to estimate chance agreement and
kappa for the interrater reliability of response segments using the Ror-
schach Comprehensive System. Journal of Personality Assessment, 72,
230255.
Meyer, G. J. (2000). On the science of Rorschach research. Journal of Per-
sonality Assessment, 75, 4681.
Meyer, G. J., & Archer, R. P. (2001). The hard science of Rorschach re-
search: What do we know and where do we go? Psychological Assess-
ment, 13, 486502.
Meyer, G. J., Finn, S. E., Eyde, L., Kay, G. G., Moreland, K., Dies, R. R., et
al. (2001). Psychological testing and psychological assessment: A review
of evidence and issues. American Psychologist, 56, 128165.
Meyer, G. J., & Handler, L. (1997). The ability of the Rorschach to predict
subsequent outcome: A meta-analysis of the Rorschach Prognostic Rat-
ing Scale. Journal of Personality Assessment, 69, 138.
Meyer, G. J., & Handler, L. (2000). Correction to Meyer and Handler (1997).
Journal of Personality Assessment, 74, 504506.
Meyer, G. J., Hilsenroth, M. J., Baxter, D., Exner, J. E., Jr., Fowler, J. C.,
Piers, C. C., et al. (2002). An examination of interrater reliability for scor-
ing the Rorschach Comprehensive System in eight data sets. Journal of
Personality Assessment, 78, 219274.
*Mintz, E. E., Schmeidler, G. R., & Bristol, M. (1956). Rorschach changes
during psychoanalysis. Journal of Projective Techniques, 20, 414417.
Muench, G. A. (1947). An evaluation of non-directive psychotherapy. Inter-
national Review of Applied Psychology, 13, 1738.
Murray, H. A. (1938). Explorations in personality. New York: Oxford.
Orlinsky, D. E., Grawe, K., & Parks, B. K. (1994). Process and outcome in
psychotherapy: Noch einmal. In A. E. Bergin & S. L. Garfield (Eds.),
Handbook of psychotherapy and behavior change (4th ed., pp. 270376).
New York: Wiley.
Parker, K. C. H. (1983). A meta-analysis of the reliability and validity of the
Rorschach. Journal of Personality Assessment, 47, 227231.
Parker, K. C. H., Hanson, R. K., & Hunsley, J. (1988). MMPI, Rorschach,
and WAIS: A meta-analytic comparison of reliability, stability, and valid-
ity. Psychological Bulletin, 103, 367373.
Parker, K. C. H., Hunsley, J., & Hanson, R. K. (1999). Old wine from old
skins sometimes tastes like vinegar: A response to Garb, Florio, and
Grove. Psychological Science, 10, 291292.
*Peterson, A. O. D. (1954). A comparative study of Rorschach scoring
methods in evaluating personality changes resulting from psychotherapy.
Journal of Clinical Psychology, 10, 190192.
*Peyman, D. A. R. (1956). An investigation of the effects of group psycho-
therapy on chronic schizophrenic patients. Group Psychotherapy, 9,
3539.
Piotrowski, Z. A. (1957). Perceptanalysis. New York: Macmillan.
Piotrowski, Z. A., & Schreiber, M. (1952). Rorschach perceptanalytic
measurement of personality changes during and after intensive psycho-
analytically oriented psychotherapy. In G. Bychowski & J. L. Despert
(Eds.), Specialized techniques in pyschotherapy (pp. 337361). New
York: Basic.
Pruitt, W. A., & Spilka, B. (1964). Rorschach empathy-object relationship
scale. Journal of Projective Techniques and Personality Assessment, 28,
331336.
Rapaport, D. (1949). Diagnostic psychological testing. New York: Interna-
tional Universities Press.
*Rice, L. N. (1973). Client behavior as a function of therapist style and client
resources. Journal of Counseling Psychology, 20, 306311.
Rickers-Ovsiankina, M. A. (Ed.). (1960). Rorschach psychology. New York:
Wiley.
Rioch, M. J. (1949). The use of the Rorschach test in the assessment of
change in patients under psychotherapy. Psychiatry: Journal for the Study
of Interpersonal Processes, 12, 427434.
Roberts, B. W., & DelVecchio, W. F. (2000). The rank-order consistency of
personality traits from childhood to old age: A quantitative review of lon-
gitudinal studies. Psychological Bulletin, 126, 325.
Romney, D. M. (1990). Though disorder in the relatives of schizophrenics:
A meta-analytic review of selected published studies. Journal of Nervous
and Mental Disease, 178, 481486.
Rorschach, H. (1942). Psychodiagnostics (9th ed., P. Lemkau, Trans.). Bern,
Switzerland: Hans Huber Verlag. (Original work published 1921)
Rosenthal, R. (1991a). Meta-analytic procedures in social research (Rev.
ed.). Baltimore: Sage.
Rosenthal, R. (1991b). Quality-weighting of studies in meta-analytic re-
search. Psychotherapy Research, 1, 2528.
Rosenthal, R., & Rubin, D. B. (1978). Interpersonal expectancy effects: The
first 345 studies. The Behavioral and Brain Sciences, 3, 377415.
Schachtel, E. G. (1966). Experiential foundations of Rorschachs test. New
York: Basic.
Schafer, R. (1954). Psychoanalytic interpretation in Rorschach testing. New
York: Grune & Stratton.
Schwager, E., & Spear, W. E. (1981). New perspectives on psychological
tests as measures of change. Bulletin of the Menninger Clinic, 45,
527541.
Seligman, M. E. P. (1995). The effectiveness of psychotherapy: The Con-
sumer Reports study. American Psychologist, 50, 965974.
Seligman, M. E. P. (1996). Science as an ally of practice. American Psychol-
ogist, 51, 10721079.
Shadish, W. R., Navarro, A. M., Matt, G. E., & Phillips, G. (2000). The ef-
fects of psychological therapies under clinically representative condi-
tions: A meta-analysis. Psychological Bulletin, 126, 512529.
Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer
(Vol. 1). Thousand Oaks, CA: Sage.
Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy out-
come studies. American Psychologist, 32, 752760.
 RORSCHACH CHANGE ASSESSMENT
Soskis, D. A., & Bowers, M. B., Jr. (1969). The schizophrenic experience: A
follow-up study of attitude and post-hospital adjustment. Journal of Ner-
vous and Mental Disease, 149, 443449.
Spero, M. H. (1984). Some pre- and post-treatment characteristics of cult
devotees. Perceptual and Motor Skills, 58, 749750.
SPSS. (2001). Statistical Package for the Social Sciences (Version 11.0.0 for
Windows) [Computer software]. Chicago: Author.
Stanton, A. H., Gunderson, J. G., Knapp, P. H., Frank, A. F., Vannicelli, M.
L., Schnitzer, R., et al. (1984). Effects of psychotherapy in schizophrenia:
I. Design and implementation of a controlled study. Schizophrenia Bulle-
tin, 10, 520563.
Steel, P. D., & Kammeyer-Mueller, J. D. (2002). Comparing meta-analytic
moderator estimation techniques under realistic conditions. Journal of
Applied Psychology, 87, 96111.
Strauss, J. C., & Harder, D. W. (1981). The Case Record Rating Scale: A
method for rating symptoms and social functions data from case records.
Psychiatry Research, 4, 333345.
Urist, J. (1977). The Rorschach test and the assessment of object relations.
Journal of Personality Assessment, 41, 39.
Vidalis, A. A., Preston, T. D., & Baker, G. H. (1990). Is day hospital treat-
ment effective, and can success be predicted? International Journal of So-
cial Psychiatry, 36, 137142.
Watkins, J. G. (1949). Evaluating success in psychotherapy. The American
Psychologist, 4, 396.
Wechsler, D. (1944). The measurement of adult intelligence. Baltimore: Wil-
liams and Wilkins.
*Weiner, I. B., & Exner, J. E., Jr. (1991). Rorschach changes in long-term and
short-term psychotherapy. Journal of Personality Assessment, 56, 453465.
275
West, M. M. (1998). Meta-analysis of studies assessing the efficacy of pro-
jective techniques in discriminating child sexual abuse. Child Abuse and
Neglect, 22, 11511166.
Westen, D., & Morrison, K. (2001). A multidimensional meta-analysis of
treatments for depression, panic, and generalized anxiety disorder: An
empirical examination of the status of empirically supported therapies.
Journal of Consulting and Clinical Psychology, 69, 875899.
Wise, E. (2004). Methods for analyzing psychotherapy outcomes: A review
of clinical significance, reliable change, and recommendations for future
directions. Journal of Personality Assessment, 82, 5059.
Wode-Helgodt, B., Berg, G., Petterson, U., Rydelius, P. A., & Trollehed, H.
(1988). Group therapy with schizophrenic patients in outpatient depart-
ments. Acta Psychiatrica Scandinavica, 78, 304313.
Wood, J. M., Lilienfeld, S. O., Garb, H. N., & Nezworski, M. T. (2000).
The Rorschach test in clinical diagnosis: A critical review, with a back-
ward look at Garfield (1947). Journal of Clinical Psychology, 56,
395430.
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1996). The Comprehen-
sive System for the Rorschach: A critical examination. Psychological Sci-
ence, 7, 310.
Wood, J. M., Nezworski, M. T., & Stejskal, W. J. (1997). The reliability of
the Comprehensive System for the Rorschach: A comment on Meyer
(1997). Psychological Assessment, 9, 490494.
Zamansky, H. S., & Goldman, A. E. (1960) A comparison of two methods of
analyzing Rorschach data in assessing therapeutic change. Journal of
Projective Techniques, 24, 7582.
Zulliger, H. (1941). Behn-Rorschach Test. Bern, Switzerland: Hans Huber
Verlag.
276
Studies Excluded From the Meta-Analysis Despite Appropriate Variables
Study
Barry, Blyth, & Albrecht (1952)
Berman (1972)
Cadman, Misbach, & Brown (1954)
Doherty (2001)
Endicott & Endicott (1963)
Exner (1978)
Fishman (1973)
Gibson, Snyder, & Ray (1955)
Harrower (1958)
Harrower (1965)
Kamil (1970)
Kantrowitz, Katz, Paolitto, Sashin,
& Solomon (1987a)
Kantrowitz Katz, Paolitto, Sashin,
& Solomon (1987b)
Kantrowitz, Katz, & Paolitto (1990)
Kantrowitz, Paolitto, Sashin,
Solomon, & Katz (1986)
Karon & Vandenbos (1972)
Lerner (1972)
Lerner & Fiske (1973)
Lindberg (1981)
Piotrowski & Schreiber (1952)
Pruitt & Spilka (1964)
Rioch (1949)
Schwager & Spear (1981)
Spero (1984)
Vidalis, Preston, & Baker (1990)
Watkins (1949)
Wode-Helgodt, Berg, Petterson,
Rydelius, &Trollehed (1988)
Zamansky & Goldman (1960)
Note. TAT = Thematic Appreception Test; WAIS = Wechsler Adult Intelligence Scales.
Cato Grnnerd
Department of Psychology
Pb 111
FIN80101 Joensuu, Finland
E-mail: cato.gronnerod@psykomatikk.no
Received October 2, 2002
Revised May 6, 2004
GRNNERD
APPENDIX
Reason
Sample included participants who were not in psychotherapy
Experimental intervention using video
Sample included participants who were not in psychotherapy
The study was based on the same participants as in Glatt and Karon (1974), but the data was included in the Glatt
and Karon samples third retest
Sample included participants who were not in psychotherapy
Rreported testretest correlations, which cannot be meaningfully converted to effect sizes reflecting changes
Did not analyze changes in Rorschach data from pretest to posttest
Reported factor analysis of data from multiple tests based on the same participants as in Peterson (1954), which
was included in this analysis
Reported only frequency data
Reported only frequency data
Alternative administration Rorschach cards
Data was based on several assessment instruments (TAT, Rorschach, Draw-a-Person, Cole Animal, & WAIS); no
specific Rorschach data was reported
Same as previous
Same as previous
Same as previous
The study was based on the same participants as in Glatt & Karon (1974) and included participants who were not in
psychotherapy; data was not available from Karon
Alternative administration Rorschach cards
Alternative administration Rorschach cards
Criterion variables possibly also used as selection criteria: The diagnosis of schizophrenia was to be confirmed by
the projective tests the Ro [Rorschach] and the HIT [Holtzman Inkblot Technique] (p. 58); the author did not
specify which indexes were considered as a confirmation of schizophrenia, but neither did he specifically
exclude those that were under investigation
Reported only data from variables that were significant, and it was not possible to determine which other variables
were also investigated
Did not analyze changes in Rorschach data from pretest to posttest
Reported only frequency data
Excluded one patient from the analyses that was atypical compared to the others and another that did not show
indications of change
Reported only data from variables that were significant, and it was not possible to determine which other variables
were also investigated
Alternative administration Rorschach cards
Reported only frequency data
Control participants received medication
Sample included participants who were not in psychotherapy