Themetaanalyticcorrelation

betweentwoBigFivefactors
Somethingisnotquiterightinthewoodshed

PaulBarrett&JeanPierreRolland*

*Universit deParisQuest
NanterreLaDfense,France

April,2009

http://www.pbarrett.net/stratpapers/metacorr.pdf 2 of16

AuntAdaDoomistheinfamousmadwomanintheatticofStellaGibbonscomedynovelColdComfort
Farm(1932);hermindbecameunhingedwhenasachildshesawsomethingnastyinthewoodshed.
Theliteraryphrasemaynottotallycapturetheeffectoftheobservationswemakebelow,but
somethingisnotquiterightaboutthefollowingmetaanalyticresultsreportedinaseriesofstudies
since1993.

Wedonotwishtodwellontheprosandconsofmetaanalysis,butratherwefindourselvesquestioning
theimplicitunderstandingthatmetaanalysisisalwayscapableofrevealingtheexpectedpopulation
correlationbetweenattributes.ThepaperbyLeLorier,J.,Gregoire,G.,Benhaddad,A.,Lapierre,J.,&
Derderian,F.(1997)Discrepanciesbetweenmetaanalysesandsubsequentlargerandomized,controlled
trials.TheNewEnglandJournalofMedicine,337,8,536542isperhapsthemostfamousstudyshowing
thatmetaanalysisdoesnotalwaysproduceaccurateestimatesofpopulationparameters,andthe
recentstudybySchonemann,P.H.,&Scargle,J.D.(2008)AGeneralizedPublicationBiasModel.Chinese
JournalofPsychology,50,1,2129,helpstoexplainwhy.

Ofspecificinterestherethougharethevariousmetaanalyticestimatesofpopulationcorrelations
betweentwospecificBigFivepersonalitytestscales,EmotionalStabilityandConscientiousness.These
arethetwomostimportantbroadpersonalityfactorsassociatedmetaanalyticallywithjob
performance.


Anaside[Feb,2010]:arecentpaperfromDeRaad,B.,Barelds,D.P.H.,Levert,E.,Ostendorf,F.,Mlacic,
B.,DiBlas,L.,Hrebckov,M.,Szirmk,Z.,Szarota,P.,Perugini,M.,Church,A.T.,&Katigbak,M.S.(2010)
Onlythreefactorsofpersonalitydescriptionarefullyreplicableacrosslanguages:Acomparisonof14
traittaxonomies.JournalofPersonalityandSocialPsychology,98,1,160173.Thethreefactorsthey
findreplicableare:

Extraversion
(+)active,chatty,cheerful,dynamic,energetic,enthusiastic,extraverted,exuberant,lively,open,
outgoing,sociable,talkative,vigorous,vivacious
()bashful,closed,introverted,lonely,passive,quiet,reserved,shy,silent,solitary,taciturn,timid,
unsociable,untalkative,withdrawn

Agreeableness
(+)accommodating,agreeable,benevolent,conciliatory,friendly,gentle,goodnatured,kindhearted,
lenient,meek,mild,patient,peaceful,sympathetic,tolerant
()aggressive,bossy,brusque,choleric,coldhearted,despotic,domineering,fierce,hottempered,
intolerant,irritable,overbearing,quarrelsome,shorttempered,stubborn

Conscientiousness
(+)careful,conscientious,diligent,disciplined,dutiful,hardworking,industrious,methodical,
meticulous,orderly,organized,precise,scrupulous,thorough,tidy
()absentminded,careless,chaotic,disorderly,disorganized,frivolous,imprudent,inaccurate,
irresponsible,lax,lazy,negligent,rash,undisciplined,untidy

InterestinglytheyfindEmotionalStability/Anxietyisnotreplicableacrosscultures.

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 3 of16

TheEvidence
Therelevantcorrelationappears(albeitnotreferencedinthepaper)inTable6,p.669,of:
Ones,D.S.,Viswesvaran,C.,&Reiss,A.D.(1996)Roleofsocialdesirabilityinpersonalitytestingfor
personnelselection:Theredherring.JournalofAppliedPsychology,81,6,660679.

Ones(1993)isnotreferencedinthepaper,butthecitationmayrefertoOnes,D.S.(1993).Theconstruct
validityofintegritytests.Unpublisheddoctoraldissertation,UniversityofIowa,IowaCity,IA.Whats
importanthereisthattheseareestimatedpopulationcorrelations.

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 4 of16

Mount,M.K.,Barrick,M.R.,Scullen,S.M.,&Rounds,J.(2005)HigherorderdimensionsoftheBig
FivepersonalitytraitsandtheBigSixvocationalinteresttypes.PersonnelPsychology,58,2,447478.
Table3,p.463ofthispapernowshowsatruecorrelationof0.52insteadof0.26.

LookingatTable2onpage462,wesee..

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 5 of16

The0.26correlationreportedinOnesetal(1993)liesjustoutsidethe90%credibilityvaluereportedin
Mountetal(2005).Itwouldliewithina95%interval,soarareresultperhapsbutnotimpossible.But,
therangeofthatcredibilityintervalinMount,etal.isverywide.

Itshouldbenotedthatthisistheestimatedtruescorecorrelationbetweenthesetwoattributesusedin
Table4,p.845intherecentpaperbySchmidt,F.L.,Shaffer,J.A.,&Oh,IS.(2008)Increasedaccuracyfor
rangerestrictioncorrections:implicationsfortheroleofpersonalityandgeneralmentalabilityinjob
andtrainingperformance.PersonnelPsychology,61,4,827868,andnottheonereportedbyOnes
(1993).

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 6 of16

Steel,P.,Schmidt,J.,&Shultz,J.(2008)Refiningtherelationshipbetweenpersonalityand
subjectivewellbeing.PsychologicalBulletin,134,1,138161.

FromTable6,p.148

Thiscorrelationiscomputedover37studies,utilizing17,464cases(anaverageof472perstudyas
notedonpage14,para2,column1).

GiventheBigFiveattributeEmotionalStabilityisnowcalledNeuroticism,wethinkitislikelythe
scoringwasreversed(highscoresequatetohigherEmotionalInstability),whichwouldexplainthe
reversalinsignofthecorrelationbetweenEmotionalStability(Neuroticism)andConscientiousness
attributes.

Sofarwehavecorrelationsof0.26,0.52,andnow0.33allpopulation/truevalueestimates.

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 7 of16

Meriac,J.P.,Hoffman,B.J.,Woehr,D.J.,&Fleisher,M.S.(2008)Furtherevidenceforthevalidityof
assessmentcenterdimensions:Ametaanalysisoftheincrementalcriterionrelatedvalidityof
dimensionratings.JournalofAppliedPsychology,93,5,10421052.

FromTable3,p.1047,wesee

ThecorrelationbetweenNeuroticismandConscientiousnessiscomputedusing1009casesincludedin6
independentstudies(fromTable1,p.1046).

Sonowwehavecorrelationsof0.24,0.26,0.33,and0.52;allpopulation/truevalueestimates.

NoticehowfarthecorrelationbetweenExtraversionandNeuroticismhasdroppedinthisstudyfrom
thevalueshownonthepreviouspage,0.33downto0.02.

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 8 of16

Hogan,J.,Barrett,P.,&Hogan,R.(2007)Personalitymeasurement,faking,andemployment
selection.JournalofAppliedPsychology,92,5,12701285.

Dataarefrom5,266individualswhowereadministeredtheHoganPersonalityInventoryontwo
occasions(T1andT2),whoseHPIscalescoreswereconvertedintotheirBigFiveequivalentsusinga
formuladevelopedbySmith,B.&Ellingson,J.E.(2002)Substancevsstyle:anewlookatSocial
Desirabilityinmotivatingcontexts.JournalofAppliedPsychology,87,2,211219.FromTable2,p.1276

Note,thisisnotmetaanalyticdata,justaverylargesampleof5,266cases.

Ifwecorrecteachcorrelationfortheunreliabilityofeachscale,theT1correlationbecomes0.67,that
forT2becomes0.70.


Table1:Summaryofresultsfromthevariousevidencebases to

Study SampleSize EstimatedPopulationCorrelation
Ones(1993) ? 0.26
Mountetal(2005) 4,000 0.52
Steeletal(2008) 17,464 0.33
Meriacetal(2008) 1,009 0.24
Hoganetal(2007)T1 5,266 0.67
Hoganetal(2007)T2 5,266(samecasesasT1) 0.70

Itspossiblethattheestimatedpopulationcorrelationsallliewithina95%credibilityintervalextending
from0.20to0.80.Butisthisofanypracticalvalueexcepttoindicatethatsometruecorrelationis
positive,andliessomewherebetweennearzeroandverylarge?

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 9 of16

Ultimately,wethinkwehavethedefinitiveanswerusingthe2007HoganNormativeSample
datasetof156,614cases.ScoringthedataintotheBigFivescalesusingtheSmithandEllingson
formula,wecancomputewhatarelikelytobethenearestrealisticpopulationestimatesoftheattribute
correlationsforaUSpopulation.Further,ifwewishtoassumeeachindividualpossessesatruescore
onanattribute,wecancorrectthesecorrelationsfortheunreliabilityofeachscaletocreatethetrue
scorecorrelationswhichcanbecompareddirectlytothosepresentedinTable1.

Inordertocorrecteachrawscorecorrelationforscaleunreliability,weneedthecompositealphasfor
eachoftheBigFivescales(compositebecauseeachBigFivescaleisconstructedfromfourHPIHICs).
Table2presentsthesecorrelationalresults:

Table2:RawScoreandTrueScoredisattenuatedcorrelationsbetweenEmotionalStabilityand
ConscientiousnessusingtheBigFiverescoredHPIUSNormativedatafortheHoganPersonality
Inventory

Correlations(Big5FactorscoresfromNormativeHPIN=156614.sta)
Markedcorrelationsaresignificantatp<.05000
N=156614(Casewisedeletionofmissingdata)
Variable Emotional Stability Extraversion Openness Agreeableness Conscientiousness
Emotional Stability 0.75 0.16 0.28 0.25 0.37
Extraversion 0.20 0.83 0.27 0.34 0.06
Openness 0.36 0.33 0.80 0.22 0.28
Agreeableness 0.38 0.49 0.32 0.58 0.32
Conscientiousness 0.53 0.08 0.39 0.52 0.66

Therawscorecorrelationsareabovethediagonal,thetruescoredisattenuatedvaluesaregiveninthe
lowerhalfofthematrix(theshadedcells),thealphareliabilityforeachscaleisgiveninthemain
diagonal.

AddingtherelevantvaluetoTable1:

Table3:Summaryofresultsfromthevariousevidencebases to

Study SampleSize EstimatedPopulationCorrelation
Ones(1993) ? 0.26
Mountetal(2005) 4,000 0.52
Steeletal(2008) 17,464 0.33
Meriacetal(2008) 1,009 0.24
Hoganetal(2007)T1 5,266 0.67
Hoganetal(2007)T2 5,266 0.70
HoganHPINormativeUSdataset 156,614 0.53

Ofthefourmetaanalyses(rows14),onlyone(Mount2005)cameclose,usingaquarterofthesample
sizeoftheonethatyieldedapopulationestimateof0.33.

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 10 of16

TheEvidence:TwoMoreStudies

Judge,T.A.,vanVianen,A.E.M.,&DePater,I.E.(2004)Emotionalstability,coreselfevaluations,
andjoboutcomes:Areviewoftheevidenceandanagendaforfutureresearch.HumanPerformance,
17,3,325346.

FromTable2,p.330)wesee

Thesedatawerecomputedfromthesummary(uncorrected)dataprovidedinJudge,T.A.,Erez,A.,
Bono,J.E.,&Thoresen,C.J.(2002)Aremeasuresofselfesteem,neuroticism,locusofcontrol,and
generalizedselfefficacyindicatorsofacommoncoreconstruct?JournalofPersonalityandSocial
Psychology,83,3,693710,Table8,p.704

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 11 of16

Revelle,W.,Wilt,J.,&Rosenthal,A.(inpress)Individualdifferencesincognition:Newmethods
forexaminingthepersonalitycognitionlink.ToappearinAleksandraGruszka,GeraldMatthews,and
BlazejSzymura(editors):HandbookofIndividualDifferencesinCognition:Attention,Memoryand
ExecutiveControl.Draft,April2008,downloadedfrom:
http://personalityresearch.net/revelle/publications/rwr.08.pdf

Frompage22,Table12wesee:

Thesamplesizeforthiscorrelationwas>50,000usingtheinnovativeSyntheticAperturePersonality
Assessmentsamplingprocedure(see:
http://www.personalityproject.org/revelle/publications/sapa.mpa.key.pdfformoredetailsaboutthis
studyandthesamplingtechnique;thetabledataabovearesimilartothedatareportedinslide#74).

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 12 of16

TheFinalPicture


Incorporatingthesestudieswiththosefrom to ,wecansummarizetheeightstudiesresultsin
Table4.

Table4:Summaryofresultsfromthevariousevidencebases to

Study SampleSize EstimatedPopulationCorrelation
Ones(1993) ? 0.26
Mountetal(2005) 4,000 0.52
Steeletal(2008) 17,464 0.33
Meriacetal(2008) 1,009 0.24
Hoganetal(2007)T1 5,266 0.67
Hoganetal(2007)T2 5,266 0.70
HoganHPINormativeUSdataset 156,614 0.53
Judgeetal(2004) 1,885 0.49
Revelleetal >50,000 0.17


So,ourestimatesfortherelationshipbetweenEmotionalStabilityandConscientiousnessrangefrom
0.17throughto0.70.Clearly,thisrangeofpopulationestimatesindicatesthatsomethingisnotright
here.

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 13 of16

InPsychology,attributeswiththesamenamearenotnecessarilythesameatall

Thecluetowhatmightbecausalforthiswidelyvaryingrangeofestimatesisgivenintwopapers:

Rich,G.A.,Bommer,W.H.,MacKenzie,S.B.,Podsakoff,P.M.,&Johnson,J.L.(1999)Applesand
applesorapplesandoranges?AMetaAnalysisofobjectiveandsubjectivemeasuresofsalesperson
performance.JournalofPersonalSelling&SalesManagement,XIX,4,4152.

Abstract
Thegoalofthisstudywastoexaminetherelationshipbetweenobjectiveandsubjectivemeasuresof
salespersonperformance.Theresultsofametaanalysisof21studieswithatotalsamplesizeof4,092
participantsindicatedanoverallmeancorrectedcorrelationof.447,indicatingthatthetwomeasures
sharedonlyabout20%ofvariance.Althoughamoderatorsubgroupanalysisfoundthatthecorrected
meancorrelationwassomewhathigherincertainsituations,thefindingsgenerallysuggestthat
objectiveandsubjectivemeasuresofsalespersonperformancearenotinterchangeable,andthatthe
choiceofthemostappropriatemeasuremayrequireatradeoffbetweenaccuratelytappingthedomain
oftheperformanceconstructandminimizingmeasurementerror.

Thisisaclearindicationthatmetaanalyseswhichutilizestudiescomposedofasamenamecriterion
(hereSalesPerformance)arelikelytoyieldquitedifferentpopulationestimates)ifthecriterionis
notexactlythesame.

Frompp.4142ofthispaper:
Inthesalesmanagementliterature,performancehasbeenmeasuredinavarietyofdifferentways.For
example,roughlyhalfofthestudies(53.3%ofthereportedcorrelations)includedinChurchill,Ford,
Hartley,andWalker's(1985)metaanalysisofthedeterminantsofsalespersonperformancemeasured
performanceusingsubjectiveevaluationsobtainedfrommanagers,peers,orselfreports.Theotherhalf
(46.7%ofthereportedcorrelations)measuredvolume,salescommissions,orpercentofquota.
However,inmostinstances,noattemptwasmadetoexplainwhyonetypeofmeasurewasusedas
opposedtoanother.Theimplicitassumptionappearstobethatobjectiveandsubjectivemeasuresof
performanceareinterchangeable,andthatthedomainofsalesperformanceisadequatelycapturedby
eithersubjectiveratingsorobjectiveresults.Thus,practitionersandresearchersalikeareassumingthat
thesetwoclassesofmeasuresare"applesandapples"andthattheymaybetreatedinterchangeably.

Thesameargumentmighteasilybemadeforthatglobalcriterion,jobperformance.

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 14 of16

Woods,S.A.(2009).Thecomparativevaliditiesofsixpersonalityinventories.Proceedingsofthe
DivisionofOccupationalPsychology,BritishPsychologicalSociety,AnnualConference2009reportedin
detailinWoods,S.A.(2009).TheStructuresandValiditiesofFiveWorkrelatedPersonalityInventories.
UnpublishedWorkingPaper(Jan30th,2009).TheworkingpapermayberequestedfromS.A.Woods,
AstonBusinessSchoolhttp://www.abs.aston.ac.uk/newweb/staff/detail.asp/sfldStaffID=A0000731).

Abstract
Thisstudyexaminedthestructures,convergentvalidities,andcriterionvaliditiesoffiveworkrelated
personalityinventories(theHoganPersonalityInventory.theOccupationalPersonalityQuestionnaire.
theSixteenPersonalityFactorQuestionnaire.thePersonalityandPreferencesInventory,ProfileMatch).
Asampleof371individualsfromtheUKworkingpopulationcompletedvariouscombinationsofthefive
inventories,plusameasureofthelexicalBigFiveandseveralcriterionscales.Overall,theresults
indicatedsensibleandinterpretablefactorstructuresfortheinventories(withtheexceptionofthe
PersonalityandPreferencesInventory),theoreticallymeaningfulconvergentvalidities,andsimilar
criterionvaliditymagnitudesandpatterns.Itwasconcludedthatthesedatagiveconfidenceintheuse
oftheseinventoriesforresearchandpractice,andrevealimportantsimilaritiesandconsistenciesin
theirvalidities.

AsubselectionofTable23onp.30presentsasummaryofthemeancorrelationsforeachparticularBig
Fivescale.Thetableshowsthemeancorrelationbetweenthe5differenttestsmeasuresofeachofthe
BigFiveattributes.So,given5personalitytests,eachmeasuringExtraversion,Woodscorrelateseach
testsscoresonExtraversionwiththosefromtheothertests(pairwise);producing((5x5)5)/2possible
correlations.Theaverageisthentakenofthesecorrelations.

Whatthesedataindicateisthatthereisonlyabroad
levelofagreementbetweenfivemajorworkrelated
personalitytestswhentheirscalesarereexpressed
(whereappropriate)asBigFivepersonality
constructs.

Whereastheevidencebasessupportingeach
questionnairemightwellbesubstantiveand
excellent,anymetaanalysiswhichincorporated
thesetestsaspartofastudylookingatthe
correlationbetweenpersonalityconstructs,oreven
thepredictiveaccuracyofaspecifiedoutcome,is
goingtobeinfluencedbythelackofagreement
betweentestscales.

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 15 of16

Anaside
Maraun,M.D.(1998)MeasurementasaNormativePractice:ImplicationsofWittgenstein'sPhilosophy
forMeasurementinPsychology.Theory&Psychology,8,4,435461...
B.Thedifficultyfacedbypsychologistsinmeasuringisnotmathematicalorempiricalinnature,butis
insteadthattheconceptstheywishtohaveenterintotheirmeasurementoperationsaretypicallyofthe
commonorgardenvariety.Theseconceptshavenotoriouslycomplicatedgrammars.Inlightof(5),this
generatesseriousdifficulties.

C.Therelativelackofsuccessofmeasurementinthesocialsciencesascomparedtothephysical
sciencesisattributabletotheirsharplydifferentconceptualfoundations.Inparticular,thephysical
sciencesrestonabedrockoftechnicalconcepts,whilepsychologyrestsonawebofcommonorgarden
psychologicalconcepts.Thefactof(B)completestheexplanation."

andonpp452453andonwards..
"ConstraintsonMeasurementinPsychology
IfWittgenstein'sviewsarecorrect,thendifficultiesinpsychologicalmeasurementmaybeexplainedby
thefactthatpsychologyhasmischaracterizedmeasurement.However,whatifonedidconsiderthe
grammarsofpsychologicalconcepts.Whatwouldgrammarsayabouthowtomeasurepsychological
concepts?Ibelievethatagrammaticalinvestigationofpsychologicalconcepts(e.g.asinTerHark,1990)
reveals(a)theobviousfactthat,asitstands,commonorgardenpsychologicalconceptsarenot
measurable,and(b)theexistenceofgrammaticalconstraintsonthepossibilityofmeasurement
involvingcommonorgardenpsychologicalconcepts.Theseconstraintsmay,inpart,explainthe
enduringdifficultyfacedbypsychologyinattemptingtomeasure,and,inparticular,ascomparedtothe
physicalsciences(forrelateddiscussionsofthelatterpoint,seeCampbell,1921;Schonemann,1994).

While(a)and(b)mayhavethetoneofoverstatement,Ibelievethatitisneverthelessinterestingto
brieflyconsidereach.Todosorequiresthattwotypesofconceptbedistinguished:(1)technical
conceptsand(2)commonorgardenconcepts.Atechnicalconceptisaconceptdefinedbyaspecialized
orexpertcommunity,andemployedwithinanarrow,technicalfieldofapplication.Acommonor
gardenconcept,ontheotherhand,isaconceptwithacommonemploymentineverydaylife(Baker&
Hacker,1982).Commonorgardenconceptsaretaught,learnedandunderstoodbythepersononthe
street,andhavemeaningsthataremanifestinbroad,normativelinguisticpractices.They,inaddition,
aremoreapttobethesourceofconfusioninthecontextofscientificinvestigation...."


AnObviousConclusion?
MetaAnalyticpopulationestimateswillbeunreliable(unstable)unlessidenticalattributesand
predictors/outcomesareusedineverycontributingstudy.Thismakessensewhenyouthinkaboutwhat
metaanalysiswasreallydesignedforcorrectingsamplingerrorsduetosmall,individualstudy
attemptsatexaminingconstructrelationsandeffectsizes.

Asyoudepartfromthisrule,independentstudymetaanalyticestimatesofsupposedlythesame
phenomenawillshowdegreesofvariationinroughproportiontothevariationinmeaningamongst
samenameattributeswhichcompriseconstituentstudies.

Somethingisdefinitelynotquiterightinthewoodshed.

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009


http://www.pbarrett.net/stratpapers/metacorr.pdf 16 of16

Appendix1:

TheabstracttothepaperbyLeLorier,J.,Gregoire,G.,Benhaddad,A.,Lapierre,J.,&Derderian,F.(1997)
Discrepanciesbetweenmetaanalysesandsubsequentlargerandomized,controlledtrials.TheNew
EnglandJournalofMedicine,337,8,536542.


Background
Metaanalysesarenowwidelyusedtoprovideevidencetosupportclinicalstrategies.However,large
randomized,controlledtrialsareconsideredthegoldstandardinevaluatingtheefficacyofclinical
interventions.

Methods
Wecomparedtheresultsoflargerandomized,controlledtrials(involving1000patientsormore)that
werepublishedinfourjournals(theNewEnglandJournalofMedicine,theLancet,theAnnalsofInternal
Medicine,andtheJournaloftheAmericanMedicalAssociation)withtheresultsofmetaanalyses
publishedearlieronthesametopics.Regardingtheprincipalandsecondaryoutcomes,wejudged
whetherthefindingsoftherandomizedtrialsagreedwiththoseofthecorrespondingmetaanalyses,
andwedeterminedwhetherthestudyresultswerepositive(indicatingthattreatmentimprovedthe
outcome)ornegative(indicatingthattheoutcomewithtreatmentwasthesameorworsethanwithout
it)attheconventionallevelofstatisticalsignificance(P<0.05).

Results
Weidentified12largerandomized,controlledtrialsand19metaanalysesaddressingthesame
questions.Foratotalof40primaryandsecondaryoutcomes,agreementbetweenthemetaanalyses
andthelargeclinicaltrialswasonlyfair(kappa=0.35;95percentconfidenceinterval,0.06to0.64).The
positivepredictivevalueofthemetaanalyseswas68percent,andthenegativepredictivevalue67
percent.However,thedifferenceinpointestimatesbetweentherandomizedtrialsandthemeta
analyseswasstatisticallysignificantforonly5ofthe40comparisons(12percent).Furthermore,ineach
caseofdisagreementastatisticallysignificanteffectoftreatmentwasfoundbyonemethod,whereas
nostatisticallysignificanteffectwasfoundbytheother.

Conclusions
Theoutcomesofthe12largerandomized,controlledtrialsthatwestudiedwerenotpredicted
accurately35percentofthetimebythemetaanalysespublishedpreviouslyonthesametopics.

StrategicWhitepaper#3:Metaanalyticcorrelationwithinpersonalitypsychology April,2009