FAT TAILS RESEARCH PROGRAM

The Meta-Distribution of Standard P-Values

Nassim Nicholas Taleb
Tandon School of Engineering, New York University and Real World Risk Institute, LLC.

PDF
This discussion is extracted from the chapter in Silent 10
Risk on meta-probability.
AbstractWe present an exact probability distribution (meta-
distribution) for p-values across ensembles of statistically iden- 8 n=5
tical phenomena, as well as the distribution of the minimum n=10
p-value among m independents tests. We derive the distribution n=15
6
arXiv:1603.07532v3 [stat.AP] 8 Apr 2016

for small samples 2 < n n 30 as well as the limiting one as

the sample size n becomes large. We also look at the properties n=20
of the "power" of a test through the distribution of its inverse n=25
4
for a given p-value and parametrization.
P-values are shown to be extremely skewed and volatile,

FT
regardless of the sample size n, and vary greatly across repetitions 2
of exactly same protocols under identical stochastic copies of the
phenomenon; such volatility makes the minimum p value diverge
significantly from the "true" one. Setting the power is shown to p
offer little remedy unless sample size is increased markedly or 0.00 0.05 0.10 0.15 0.20
the p-value is lowered by at least one order of magnitude.
The formulas allow the investigation of the stability of the Fig. 1. The different values for Equ. 1 showing convergence to the limiting
reproduction of results and "p-hacking" and other aspects of distribution.
meta-analysis.
From a probabilistic standpoint, neither a p-value of .05 nor
a "power" at .9 appear to make the slightest sense.
an in-sample .01 p-value has a significant probability of
RA
SSUME that we know the "true" p-value, ps , what having a true value > .3.
A would its realizations look like across various attempts on
statistically identical copies of the phenomena? By true value
So clearly we dont know what we are talking about
when we talk about p-values.
ps , we mean its expected value by the law of large numbers Earlier attempts for an explicit meta-distribution in the
across an m ensemble of possible samples for the phenomenon literature were found in [1] and [2], though for situations of
1
P P P
under scrutiny, that is m m pi ps (where denotes Gaussian subordination and less parsimonious parametrization.
convergence in probability). A similar convergence argument The severity of the problem of significance of the so-called
can be also made for the corresponding "true median" pM . "statistically significant" has been discussed in [3] and offered
The main result of the paper is that the the distribution of n a remedy via Bayesian methods in [4], which in fact recom-
small samples can be made explicit (albeit with special inverse mends the same tightening of standards to p-values .01.
functions), as well as its parsimonious limiting one for n large, But the gravity of the extreme skewness of the distribution
with no other parameter than the median value pM . We were
D

of p-values is only apparent when one looks at the meta-

unable to get an explicit form for ps but we go around it with distribution.
the use of the median. Finally, the distribution of the minimum For notation, we use n for the sample size of a given study
p-value under can be made explicit, in a parsimonious formula and m the number of trials leading to a p-value.
allowing for the understanding of biases in scientific studies.
It turned out, as we can see in Fig. 2 the distribution is
extremely asymmetric (right-skewed), to the point where 75%
of the realizations of a "true" p-value of .05 will be <.05 (a I. P ROOFS AND DERIVATIONS
borderline situation is 3 as likely to pass than fail a given
protocol), and, what is worse, 60% of the true p-value of .12
will be below .05. Proposition 1. Let P be a random variable [0, 1]) corre-
Although with compact support, the distribution exhibits sponding to the sample-derived one-tailed p-value from the
the attributes of extreme fat-tailedness. For an observed paired T-test statistic (unknown variance) with median value
p-value of, say, .02, the "true" p-value is likely to be >.1 M(P ) = pM [0, 1] derived from a sample of n size.
(and very possibly close to .2), with a standard deviation The distribution across the ensemble of statistically identical
>.2 (sic) and a mean deviation of around .35 (sic, sic). copies of the sample has for PDF
Because of the excessive skewness, measures of disper- (
sion in L1 and L2 (and higher norms) vary hardly with (p; pM )L for p < 1
(p; pM ) = 2
ps , so the standard deviation is not proportional, meaning (p; pM )H for p > 1
2

N. N. Taleb 1
 FAT TAILS RESEARCH PROGRAM

(pM , .) = 12 . Hence we end up having { : 12 I 2n+n n2 , 12 =

 
1
(n1)
pM } (positive case) and { : 21 I 2 12 , n2 + 1 = pM }


(p; pM )L = p2
s 2 +n

( 1) (negative case). Replacing we get Eq.1 and Proposition 1 is

p p pM p done.
(p 1) pM 2 (1 p ) p (1 pM ) pM + 1
n/2
We note that n does not increase significance, since p-
1
values are computed from normalized variables (hence the
1 2 1p pM 1

+ 1 universality of the meta-distribution); a high n corresponds
p p 1pM 1pM
to an increased convergence to the Gaussian. For large n, we
can prove the following proposition:

1
(p; pM )H = 1 0p 2 (n1)
Proposition 2. Under the same assumptions as above, the
0


1 (pM 1) n+1 limiting distribution for (.):
qp
p
2
0 0 0
p (pM ) + 2 1 p p (1 pM ) pM + 1
1
(2pM )(erfc1 (2pM )2erfc1 (2p))
(1) lim (p; pM ) = eerfc (2)
n

FT
1 n 1 1

1 n

0
where p
= I2p 2 , 2 , pM = I12pM 2 , 2 , p =
1 1 n 1 where erfc(.) is the complementary error function and
I2p1 2 , 2 , and I(.) (., .) is the inverse beta regularized
function. erf c(.)1 its inverse.
Remark 1. For p= 12 the distribution doesnt exist in theory, The limiting CDF (.)
but does in practice and we can work around it with the
sequence pmk = 12 k1 , as in the graph showing a convergence 1
erfc erf1 (1 2k) erf1 (1 2pM ) (3)


(k; pM ) =
to the Uniform distribution on [0, 1] in Figure 3. Also note 2
that what is called the "null" hypothesis is effectively a set of
measure 0. mv
Proof: For large n, the distribution of Z = becomes
RA
s
v
n
Proof: Let Z be a random normalized variable with that ofa Gaussian, and the one-tailed survival function g(.) =
realizations , from a vector ~v of n realizations, with sample

1
erfc , (p) 2erfc1 (p).
mean mv , and sample standard deviation sv , = mvsm v
h 2 2
n
(where mh is the level it is tested against), hence assumed
to Student T with n degrees of freedom, and, crucially, PDF/Frequ.
supposed to deliver a mean of ,
53% of realizations <.05
  n+1
2 25% of realizations <.01
n 0.15

() 2 +n
=
f (; ) n 1


nB 2, 2
5% p-value
where B(.,.) is the standard beta function. Let g(.) be the one- 0.10
cutpoint
D

tailed survival function of the Student T distribution with zero (true mean)

mean and n degrees of freedom: Median

0.05
12 I 2n+n n2 , 12


0
g() = P(Z > ) = 1
 
1 n


2 I 2 2 , 2 + 1
<0
0.00 p
2 +n
0.05 0.10 0.15 0.20
where I(.,.) is the incomplete Beta function.
We now look for the distribution of g f (). Given that Fig. 2. The probability distribution of a one-tailed p-value with expected
value .11 generated by Monte Carlo (histogram) as well as analytically with
g(.) is a legit Borel function, and naming p the probability (.) (the solid line). We draw all possible subsamples from an ensemble with
as a random variable, we have by a standard result for the given properties. The excessive skewness of the distribution makes the average
transformation: value considerably higher than most observations, hence causing illusions of
"statistical significance".


f g (1) (p)
(p, ) = 0 (1)

|g g (p) | This limiting distribution applies for paired tests with known
or assumed sample variance since the test becomes a Gaussian
We can convert into the corresponding median survival variable, equivalent to the convergence of the T-test (Student
probability because of symmetry of Z. Since one half the T) to the Gaussian when n is large.
we can ascertain that
observations fall on either side of ,
= 1 , hence
the transformation is median preserving: g() Remark 2. For values of p close to 0, in Equ. 2 can be
2

N. N. Taleb 2
 FAT TAILS RESEARCH PROGRAM

Expected min p-val

5

0.10

4 .025
0.08
.1

3 .15 0.06

0.5
0.04
2
0.02

1 m trials
2 4 6 8 10 12 14

p Fig. 4. The "p-hacking" value across m trials for pM = .15 and ps = .22.
0.0 0.2 0.4 0.6 0.8 1.0

Fig. 3. The probability distribution of p at different values of pM . We

observe how pM = 12 leads to a uniform distribution.
size of n. To gauge the reliability of as a true measure of
power, we perform an inverse problem:

FT
X,p,n
usefully calculated as:

s

 
1 1 (X)
(p; pM ) = 2pM log
2p2M
s
r  
1
  
1

Proposition 4. Let c be the projection of the power of the
log 2 log 2p 2 2 log(p) log 2 log 2 log(pM )
2p2
e M test from the realizations assumed to be student T distributed
and evaluated under the parameter . We have
+ O(p2 ). (4) (
(c )L for c < 21
RA
The approximation works more precisely for the band of (c ) =
1
relevant values 0 < p < 2 . (c )H for c > 21

From this we can get numerical results for convolutions of where

using the Fourier Transform or similar methods. p n
We can and get the distribution of the minimum p-value per (c )L = 1 1 1 2
!
m trials across statistically identical situations thus get an idea n+1
q1 1 2
of "p-hacking", defined as attempts by researchers to get the
 q 
2 1 (1 1)1 2 (1 1)1 +1 2 1 1 1 1
3 3 3
lowest p-values of many experiments, or try until one of the p
tests produces statistical significance. (1 1) 1
(6)
Proposition 3. The distribution of the minimum of m obser-
vations of statistically identical p-values becomes (under the

1 n

n
D

limiting distribution of proposition 2): (c )H = 2 (1 2 ) 2

B ,
2 2
1 1 1

m (p; pM ) = m eerfc (2pM )(2erfc (2p)erfc (2pM )) n+1
1
2

m1

( ) 13 1+2 13 1+2 (2 1)2 1 1
 
1 1 1
2 (2 1)2 +2
1 erfc erfc (2p) erfc (2pM ) (5) 2 1 + 3
2 p
(2 1) 2 B n2 , 12


Proof: Tn (7)
P (p1 > p, p2 > p, . . . , pm > p) = i=1 (pi ) = (p) m.
1 n 1 1 1 n



Taking the first derivative we get the result. where 1 = I2 2, 2 , 2 = I2 c 1 2, 2 , and 3 =
1 n 1

c
Outside the limiting distribution: we integrate numerically I(1,2ps 1) 2 , 2 .
for different values of m as shown in figure 4. So, more
precisely, for m trials, the expectation is calculated as: III. A PPLICATION AND C ONCLUSION
Z 1 Z p m1 One can safely see that under such stochasticity for
E(pmin ) = m (p; pM ) (u, .) du dp the realizations of p-values and the distribution of its
0 0
minimum, to get what people mean by 5% confidence
(and the inferences they get from it), they need a p-value
II. I NVERSE P OWER OF T EST of at least one order of magnitude smaller.
Let be the power of a test for a given p-value p, for Attempts at replicating papers, such as the open science
random draws X from unobserved parameter and a sample project [5], should consider a margin of error in its

N. N. Taleb 3
 FAT TAILS RESEARCH PROGRAM

own procedure and a pronounced bias towards favorable

results (Type-I error). There should be no surprise that a
previously deemed significant test fails during replication
in fact it is the replication of results deemed significant
at a close margin that should be surprising.
The "power" of a test has the same problem unless one
either lowers p-values or sets the test at higher levels,
such at .99.

ACKNOWLEDGMENT
Marco Avellaneda, Pasquale Cirillo, Yaneer Bar-Yam,
friendly people on twitter ...

R EFERENCES
[1] H. J. Hung, R. T. ONeill, P. Bauer, and K. Kohne, The behavior of the
p-value when the alternative hypothesis is true, Biometrics, pp. 1122,
1997.
[2] H. Sackrowitz and E. Samuel-Cahn, P values as random variables

FT
expected p values, The American Statistician, vol. 53, no. 4, pp. 326
331, 1999.
[3] A. Gelman and H. Stern, The difference between significant and
not significant is not itself statistically significant, The American
Statistician, vol. 60, no. 4, pp. 328331, 2006.
[4] V. E. Johnson, Revised standards for statistical evidence, Proceedings
of the National Academy of Sciences, vol. 110, no. 48, pp. 19 31319 317,
2013.
[5] O. S. Collaboration et al., Estimating the reproducibility of psychological
science, Science, vol. 349, no. 6251, p. aac4716, 2015.
RA
D

N. N. Taleb 4