Permutational Non-parametric Tests

“There are two kinds of statistics, the kind you look up and the
kind you make up.”
Rex Stout (Writer)
Permutational Non-parametric tests
Advantages/Disadvantages

• PNPT make NO Assumptions therefore any data can be used

• However the assumptions of independence (spatial &

temporal) and design considerations (randomization,
sufficient replicates, no pseudoreplication) should still be
upheld – good statistical practice!
Permutational Non-parametric tests
Calculating D (delta) & its distribution

• PNPT work with absolute differences a.k.a distances

– Smaller values indicate similarity
• Makes the calculations equivalent to sum-of-squares

𝑠𝑖𝑔𝑛𝑎𝑙 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑔𝑟𝑜𝑢𝑝𝑠

𝐷= 𝐷=
𝑛𝑜𝑖𝑠𝑒 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑤𝑖𝑡ℎ𝑖𝑛 𝑔𝑟𝑜𝑢𝑝𝑠

• For our test we can compare D to an expected distribution of

D the same way we do when we calculate an F-value
• Use permutations (iterations) to generate the distribution of D
from our raw data
• Therefore shape of D distribution is dependent on your data
Permutational Non-parametric tests
Determining the distribution of D

• After you permute this process 5000 times (your choice) a

distribution of D will emerge
• Shape depends on your data – may be normal or not (doesn’t matter)

2 3
10
Permutational Non-parametric tests
Determining the distribution of D

• After you permute this process 5000 times (your choice) a

distribution of D will emerge
• Shape depends on your data – may be normal or not (doesn’t matter)

4921 D calculations < 10 79 D calculations ≥ 10

from permutations from permutations

P-value:
79/5000 = 0.0158

2 3
10
Permutational Non-parametric tests
Permutational ANOVA

Permutational ANOVA in R: The option seqs=T calculates

library(lmPerm)
sequential sum of squares
summary(aovp(YIELD~FARM*VARIETY, seqs=T)) (similar to regular ANOVA)

Pairwise Permutational ANOVA in R: Good choice for balanced

out1=aovp(YIELD~FARM*VARIETY, seqs=T) designs
TukeyHSD(out1)

You can change the maximum

number of iterations with the
maxIter= option
Permutational Non-parametric tests
• For parametric tests we know Normal, T-distribution, F-distribution
look like
– Therefore we can use the standard calculations (t-value) to calculate statistics

• When we violate the known distribution we need some other curve

to work with
– Hard to estimate a theoretical distribution that fits your data

• Best solution is to permute your data to generate a distribution

• Permutational non-parametric statistics are just as powerful as

parametric tests

• This technique is similar to bootstrapping

– But bootstrap samples data rather than changing all observation classes
If permutational techniques are so good why
not always use them?
• You say permutational non-parametric tests are as powerful
as parametric statistics – YES They Are!

• But they are still fairly new to statistical practices

• Still unknown or not understood among many uses

• Best practice is to stick with parametric statistics when you

can, but when you can’t permutational tests are great
options!