Similarity between Models

and the
Measurement of Model Risk
P. Hnaff
IAE Paris
Universit Paris 1 Panthon-Sorbonne

26 Nov 2012
IAE-Logo

Model Risk
Model Risk Defined

Model Risk
I

Risk related to the use of the wrong model (i.e. a model

that does not account for important risk factors)

Risk related to poor or unstable calibration of the model

Both issues translate into :

I

Inaccurate initial pricing

Inaccurate risk indicators, leading to ineffective hedging

strategy

All of this while the model correctly prices benchmark

instruments.
IAE-Logo

Model Risk
Model Risk Defined

Some References

An inventory of model risks : Derman (1996) Model Risk

Goldman Sachs

Dynamic hedging simulations and empirical assessment of

model risk : Rubinstein (1985), Bakshi, Cao and Chen
(1997), Madan & Eberlein (2007).
Overall, incorporating stochastic volatility and
jumps is important for pricing and internal
consistency. But for hedging, modeling stochastic volatility alone yields the best performance.
IAE-Logo

Model Risk
Model Risk Defined

Some References

N. El Karoui, et al. Robustness of the Black and Scholes

formula. Mathematical Finance (1998).

T.J. Lyons. Uncertain volatility and the risk-free synthesis

of derivatives. Applied Mathematical Finance (1995).

R. Cont. Model uncertainty and its impact on the pricing of

derivative instruments, Mathematical Finance (2006).

PH and C. Martini. Model validation : theory, practice and

perspectives, The Journal of Risk Model Validation (2011).
IAE-Logo

Model Risk
Model Risk Defined

From Observation to Measurement

Basel Committee directive of July 2009 on model risk :

I

Quantify model risk

Provision the risk with Tiers I capital

Needed : A measure of model risk

I

expressed in currency terms

standardized across asset classes and models, to allow for

meaningful comparisons.
IAE-Logo

Model Risk
Problem Statement

A definition of model risk (R. Cont [2006])

I Set of benchmark instruments,
HiI payoffs,
the mid-market prices, C [C bid , C ask ].
CiI
i
i
i
Q Set of n models, consistent with the market prices
of benchmark instruments :
Q Q E Q [Hi ] [Cibid , Ciask ], i I

(1)

Upper and lower price bounds over the family of models :

(X ) = sup E Qj [X ] (X ) =
j=1,...,n

inf

j=1,...,n

E Qj [X ]

Risk measure :
IAE-Logo

Q = (X ) (X )

Model Risk
Problem Statement

Focus of talk

The set of models Q, consistent with the market prices of

benchmark instruments :
Q Q E Q [Hi ] [Cibid , Ciask ], i I

(2)

How to normalize Q in order to make meaningful comparisons

between various models and asset classes ?

IAE-Logo

Model Risk
Problem Statement

Focus of talk

Q Q E Q [Hi ] [Cibid , Ciask ], i I

(3)

Define a notion of size for Q, in order to normalize the risk

measure.
n
o
Q = Q|E Q [Hi ] [Cibid , Ciask ], i I, | D(Q, P) <
(4)
Where P is the reference model, and D(Q, P) a measure of
similarity between Q and P.
IAE-Logo

Similarity between models

Similarity between models

Observe n samples Xi , i = 1, . . . , n from a multivariate
distribution. ci : number of occurrences of Xi .

Definition (Multivariate Likelihood)

Probability of observing an empirical distribution Q : qi =
model P is true.
n! Y ci
L(Q|P) = Q
pi
ci !

ci
n

if

0 L(Q|P) 1
Average log-likelihood :



L(Q|P)
= log L(Q|P)1/n

IAE-Logo

Similarity between models

Kullback-Leibler Divergence from Q to P

lim L(Q|P)
=


qi log

qi
pi

= DKL (Q|P)
eDKL (Q|P) : average likelihood of observing distribution Q when
P is the actual process.

IAE-Logo

Similarity between models

Illustration : Bivariate Normal Density

Avg. likelihood of P
if Q is true

0 2 4
X

0.8
0.4

Avg. Likelihood

0.0

Bivariate normal densities

0 8 43 8 2

F IGURE: KL Divergence of 2 bivariate normal densities, as a function

IAE-Logo
of angle between PC axis of covariance matrix.

Similarity between models

Illustration : Bivariate Normal Density

Avg likelihood of P
if Q is true

0 2 4
X

0.8
0.4
0.0

=0

= 0.8

Avg. Likelihood

Bivariate normal densities

1.0

0.0

1.0

F IGURE: KL Divergence of 2 bivariate normal densities, as a function

IAE-Logo
of correlation.

Similarity between models

Similarity of model features

Model Features
Definition (Model Features)
The model features X are the set of observations on the risk
factors that are needed to determine the payoff of a financial
instrument.
Example :
I

European option on a futures at expiry (T ) :

X = {F (T , T )}

Option expiring at T1 , on a spread between two futures

contract expiring at T1 and T2 :
X = {F (T1 , T1 ), F (T1 , T2 )}

IAE-Logo

Similarity between models

Similarity of model features

Why Focus on Model Features ?

For some payoffs, different models have similar features.
dF (t, T )
F (t, T )

= dWt

Var(ln(F (T , T ))) = 2 T

dF (t, T )
F (t, T )

= e(T t) dWt

Var(ln(F (T , T ))) =


2 
1 e2T
2

(5)
(6)

(7)
(8)
IAE-Logo

Feature F (T , T ) is log-normal in both cases.

Similarity between models

Similarity of model features

Why Focus on Model Features ?

... but the feature F (t, T ), t < T does not have the same
variance.

0.10
0.05
0.00

Integrated variance

0.15

BS
Onefactor

0.0

0.5

1.0

1.5

2.0

Time

F IGURE: Integrated variance of log(F (t, T ))

IAE-Logo

Similarity between models

Computation of KL Divergence

Calculation of KL-Divergence
Kullback-Leibler divergence between two equivalent probability
distributions, with observed features in R d .
I

Discrete density
D(Q, P) =


qi log

i
I

qi
pi

Continuous density


Z
D(Q|P) =
Rd

fQ (x) log

fQ (x)
fP (x)


dx
IAE-Logo

Similarity between models

Computation of KL Divergence

Kernel-based density estimation

Let Xi , i = 1, . . . , n be a sample of random vectors in U R d ,
drawn from a distribution with density function f . The kernel
density estimate of f , noted f is defined by :
n

X
f (X ) = 1
KH (X Xi )
n
i=1

where :
X is a feature vector in R d
H is a symmetric, positive definite matrix of rank d
K is the kernel function : a symmetric multivariate
density :
1

KH (X ) = kHk 2 K (H 2 X )

IAE-Logo

Similarity between models

Computation of KL Divergence

Kernel function
Use a uniform density over a domain which is function of the
local density of sample points.

IAE-Logo

F IGURE: Uniform density over domains of varying size

Similarity between models

Computation of KL Divergence

Uniform kernel density

K (x) =

1kxxi kRx
Rxd cd

with :
Rx radius of uniform density
d dimension of features space
cd volume of unit sphere in R d
Define Rx as the distance to the k -th nearest neighbor of x.
IAE-Logo

Similarity between models

Computation of KL Divergence

KL Divergence
Using the expression for density in terms of k-nn statistics one
gets :




d X
k (U, X )
|U|
+
log
(9)
D(Q, P) = log
|V |
|V |
k (V , X )
X V

With :
d dimension of feature space
U, V sample sdrawn from Q and P
k (U, X ) distance to k -th nearest neighbor of X in sample
U.
See : Boltz, Debreuve and Barlaud. High-dimensional statistical
measure for region-of-interest tracking. IEEE Transactions on IAE-Logo
Image Processing (2009).

Similarity between models

Determining the optimal k

Determining the optimal k

Total Calculation time T is proportional to nk

T = C n (1 + k )

Accuracy, measured by the variance of the D(Q|P)

estimator, increases with k .

IAE-Logo

Similarity between models

Determining the optimal k

Variance of kNN density estimator

(x) be a kNN density
Assume a sample of size n, and let p
estimator with k neighbors. Let V (k , x) be the volume of the
sphere containing k neighors of x.
(x) =
p

k 1
nV (k , x)

Proposition

(x)) =
VAR (p


(k 1)(n 1)
(x)2
1 p
(k 2)n)

See : D.G. Luenberger and P. Woehrmann. On kNN Density

Estimation. Working Paper Series, FINRISK.

IAE-Logo

Similarity between models

Determining the optimal k

Variance of D(Q, P)

D(Q, P) = C +

d X
[log (k (U, X )) log (k (V , X ))] (10)
|V |
X V

We need to estimate (ignoring correlation between k (U, X )

and k (V , X )) :
VAR log (k (U, X ))

IAE-Logo

Similarity between models

Determining the optimal k

Variance of D(Q, P)

U (x) =
p

k 1
nk (U, x)cd

IAE-Logo

Similarity between models

Determining the optimal k

Variance of D(Q, P)

U (x) =
p

k 1
nk (U, x)cd

Use :
k (U, x) =

k 1
U (x)cd
np

and
Var (f (X )) f 0 (E(X ))2 Var(X )

IAE-Logo

Similarity between models

Determining the optimal k

Variance of D(Q, P)

U (x))
Use Luenbergers expression for Var(p
U (x) = 1/n)
The samples points are iid (p

V (x) = 1/n
Approximate also p

to get :

Var(k (U, x)) =


(k 1)(n 1)
1
(k 2)n

IAE-Logo

Similarity between models

Determining the optimal k

Variance of D(Q, P)

Finally :
Var(DL(Q, P)) =

2n
|U|


(k 1)(n 1)
1
(k 2)n

IAE-Logo

Similarity between models

Determining the optimal k

Optimization of k for a given computation time

Following Luenbergers method : Compute D(Q|P) by
repeating the calculation N times for fixed n and k , with n = k .
1X

D(Q, P)i
D(Q,
P) =
N
i

Total calculation time is :

T = Nn(1 + k )
and



Var D(Q,
P) =

1
N(k 2)

IAE-Logo

Similarity between models

Determining the optimal k

Optimization of k for a given computation time

min
k

1
N(k 2)

such that :
Nn(1 + k ) = T
= n

k
which has the solution :
s
k =2+

4+

IAE-Logo

Similarity between models

Determining the optimal k

Estimation of T = C(1 + k ). = 0.091

1.1
1.0
0.9
0.8
0.7
0.6
0.5
0.4

10

12

14

16

F IGURE: D(Q, P) computation time as a function of k

IAE-Logo

Similarity between models

Determining the optimal k

Empirical validation
0.017
0.016
0.015
0.014
0.013
0.012
0.011
0.010
0.0092

10

12

14

F IGURE: D(Q, P) computation time Variance as a function of k .

Empirical minimum is k = 8 vs. predicted k = 7

IAE-Logo

Similarity between models

Determining the optimal k

Summary : Construction of Q
1. Simulate a sample V of feature vectors X for reference
model P.
2. Define a list {Qj } of candidate models. This list may be
built by perturbation of the parameters of P, or by
postulating alternate stochastic processes. There are no
constraints on the method used.
3. Simulate samples Uj of feature vectors X for each model
Qj .
4. Retain models Qj such that :


E Qj (Hi ) C bidi , C aski

i I

IAE-Logo

Similarity between models

A Toxicity Index

Toxicity Index at level

When using model P to price payoff X with features F . Define

model risk as :
Q = (X ) (X )
Over normalized set
Q :{Q|eD(Q|P) > , E Q [Hi ] [Cibid , Ciask ]}
0<1

IAE-Logo

Similarity between models

A Toxicity Index

Calculation Steps for Toxicity Index

Calculation of model risk associated with the use of model P for

pricing derivative X .
1. Calibrate model P on a set of benchmark instruments.
2. Compute the minimum and maximum value of the exotic
derivative according to each model in set Q .
3. Model risk is finally computed by
Q = (X ) (X )

IAE-Logo

Numerical Illustration
data

Illustration

I
I

Benchmark :SPX options

Three families of models :
1. Double Exponential Jump Diffusion (Kou & Wang 2001)
2. Stochastic Volatility (Heston 1993)
3. Stochastic Volatility with Jumps (Bates 1996)

Model risk of exotic options on SPX

IAE-Logo

Numerical Illustration
data

Data

IAE-Logo

F IGURE: SPX implied volatility by log moneyness and time to expiry

Numerical Illustration
Model Universe

Double Exponential Jump Diffusion [Kou & Wang,

2001]

dSt
=
St


Nt
X
1 2
dt + Wt +
Yi
2
i=1

(11)

fY (y ) = p1 e1 y 1y 0 + (1 p)2 e2 y 1y <0
Nt is a Poisson process with rate .
1
16.15

2
1.85

.206

p
.225

.118
IAE-Logo

Numerical Illustration
Model Universe

Calibration of Double Exponential Jump Diffusion

Model

F IGURE: Bid/Ask implied volatility and fitted volatility.

IAE-Logo

Numerical Illustration
Model Universe

Calibration of Double Exponential Jump Diffusion

Model

F IGURE: Fit by quartile. red : worst, green : best, yellow : middle

IAE-Logo

Numerical Illustration
Model Universe

Calibration of Double Exponential Jump Diffusion

Model

F IGURE: Best quartile. Blue dot : best fit

IAE-Logo

Numerical Illustration
Model Universe

Calibration of Hestons Model

dSt
= dt + t dWts
St

dt = ( t )dt + t dWt

.1104

.5045

.3591

-.74

t
.0272

IAE-Logo

Numerical Illustration
Model Universe

Calibration of Hestons Model

F IGURE: Bid/Ask implied volatility and fitted volatility.

IAE-Logo

Numerical Illustration
Model Universe

Calibration of Hestons Model

F IGURE: Fit by quartile. red : worst, green : best, yellow : middle

IAE-Logo

Numerical Illustration
Model Universe

Calibration of Hestons Model

F IGURE: Best quartile. Blue dot : best fit

IAE-Logo

Numerical Illustration
Model Universe

Model Risk : Asian option in Hestons model

25

Asian option 2Yr ATM - Heston

Price range (P=103.40)

20
15
10
5
0
1.00
0.95 0.90 0.85 0.80 0.75 0.70 0.65 0.60 0.55
Average Likelyhood

F IGURE: Model risk as a function of average likelihood of observing

market data when the alternate model is true

IAE-Logo

Numerical Illustration
Model Universe

Model Risk : Asian option in Bates model

Asian option 2Yr ATM - Bates

25

Price range (P=103.40)

20
15
10
5
00.9

0.8

0.7

0.6
0.5
Average Likelyhood

0.4

0.3

0.2

F IGURE: Model risk as a function of average likelihood of observing

market data when the alternate model is true

IAE-Logo

Numerical Illustration
Model Universe

Model Risk (Heston+Bates)

80

60

40

Model Risk

Asian
Lookback
Range

20

0.0

0.2

0.4

0.6

0.8

1.0

KL Divergence

F IGURE: Model risk as a function of average likelihood of observing

market data when the alternate model is true (inverted x axis)

IAE-Logo

Conclusion

Conclusion
I

We have described a procedure, inspired from the field of

image processing, for computing the degree of similarity
between models, independently of the definition of these
processes.

We then use this measure to compute a normalized

measure of model risk.

The definition of the set of alternate models Q is crucial to

the measure.

We have presented an implementations that is

computationally intensive but does not put any restrictions
on the composition of Q.

IAE-Logo

Technical Details

Implementation details

model calibration and asset pricing : QuantLib/python

I
I

I
I

QuantLib : www.quantlib.org
Python wrapper : pyql www.github.com

size of Q : 2000 parameter perturbations per model.

calculation of D(Q|P) :
I
I

20,000 scenarios per Qj

KL-Divergence by k -nn algorithm : FNN R package
(www.r-project.org)

IAE-Logo