ISE Introduction to Robustness and Robust

Statistics
Abstract
The meaning of robustness and its meaning in different settings are
explained. The importance of robust statistics is shown through
comparison of the classical statistical methods and basic parallel robust
methods for the same data. A few examples using such methods are
mentioned to highlight their use in current research in various fields of
study.
Keywords: Robustness, Robust Statistics

1. Definition and Meaning in General Settings:

According to Merriam-Webster, Robust is defined as being capable of
performing without failure under a wide range of conditions.[1] Generally
speaking, it would mean that anything that is described as robust can be
expected to work (presumably not at the same level or efficiency) even
after being subjected to environments or situations which would be
considered outliers. It can be seen being used in different fields of study
and industries to define various processes and systems.
When we discuss about robustness in a general setting, we also talk about
resilience, as they are both considered different ways to tackle a problem.
In every organization, the manager tries to assess and predict the future
events which might affect the outcome of the said process/system he is
responsible for in an adverse manner and create procedures to be
followed in the case such an event does take place. These procedures are
meant to reduce or nullify the effects of the event on the inner workings of
the organization, so as to not affect their ability to reach their goals. This
goal could be anything ranging from offering of services to customers to
manufacturing a product. In most cases they are able to predict such
events and create appropriate procedures. Such events would be the high
probability events. The most damaging events are the low probability high
impact events. Usually they are not planned for as the probability of the
event happening is extremely low. An organization, process or products
ability to cope with such an event depends on how robust it is.[2]
According to the Institute for Telecommunications Sciences, System
Robustness is defined as The measure or extent of the ability of a
system, such as a computer, communications, data processing, or
weapons system, to continue to function despite the existence of faults in
its component subsystems or parts. Note: System performance may be
diminished or otherwise altered until the faults are corrected[3].

When a robust approach is used to handle the event, we try to keep

operating or functioning using the same fundamental process and
resources. The more robust the process is, the more hits it can take until
its impending failure. In a resilient approach, the threat is analysed and
the required changes to the system are made to cope with the shock.
Here, the emphasis is on the ability to adapt, recover and change the
system for a longer sustainment period. A robust strategy is considered
for shorter period of time to survive small shocks, whereas, resilient
strategy is usually considered for a longer period. In a robust strategy, we
deal with trade-offs to keep the system alive. As a result, the output might
not be optimal as we are still relying on the same amount of resources.
Consider a machine in a manufacturing plant which is on the verge of a
breakdown. Due to its robustness, despite being close to failure we can
say that it is still working. The machine operator believing the machine is
till good for production keeps using it until its eventual breakdown .We can
say that the is operational but is on verge of breakdown. This is the result
of a phenomenon called Drift. Drift is the slow movement of a system,
process or machine towards failure while still under the assumption that
everything is alright. It can be seen that drift is a by-product of a robust
strategy. Instead if the plant used a resilient strategy for all its machines,
the defective machine would have been identified and replaced or
repaired saving the lost downtime[2].
Now having understood the meaning of robustness and resilience up to a
certain extent in a general setting we move onto the robustness in
statistics and statistical processes.

2. Robustness in Statistics
Robustness in statistics, as described by Huber signifies insensitivity to
small deviations from the assumptions [4]. When using classical
statistical methods, we take a few assumptions. These assumptions
include: 1) Normality Assumption (All the populations being analysed are
considered to normally distributed.) 2) The normal distributions have the
same variances and standard deviation. The problem with these
assumptions is that they are frequently violated when considering real
world data. So, when using robust statistics we try to reduce the influence
of these outliers on the estimators representing the location and the scale
of the data points. The data could be skewed, contain outliers and have
thick tails and when such data sets are studied using classical statistical
methods, false results are obtained which are highly influenced by the
presence of these departures from the assumptions of normality and
symmetry. The other areas of violations have not been study as deeply as
the presence of outliers.

3. Comparison of Classical methods to Robust Methods

When representing a data set, we use two primary statistical values i.e.
the Mean and the variance. The mean is used to show the central
tendency or the location and the variance or standard deviation which are
used to represent the distribution, dispersion, spread or the scale of the
data points about the location.
=

1
x
n ;

( xx )2

S=
2

n1

Lets start by comparing the behaviour of these normal statistics in the

presence of outliers. For example, consider the following dataset:
(5.9, 3.9, 3.03, 4.34, 4.30, 3.30, 3.49, 3.46, 4.37, 58.9, 6.9, 5.34, 3.5, 4.96,
6.7, 5.34, 3.9, 6.45, 4.2, 4.85, 2.09, 8.20, 4.6, 3.72, 5.2, 4.28, 7.92, 1.6,
5.71, 6.66)
The mean for the data lies at 6.570 and it can be clearly seen as an
outlier and that it affects the value of the mean, dragging it to a higher
value. Calculating the mean value for the same dataset by trimming the
outlier gives us a value of 4.766. Another statistic that can be used to
represent the central tendency is the median which is the exact middle
value when the dataset is arranged in a descending or an ascending order.
Median is a much more robust statistic, which does not undergo as much
fluctuation as the mean. For the same dataset, the median is 4.485 and
for the trimmed dataset, median jumps just to the next value i.e. 4.370.
This is visually represented using R studio in Fig. 1, where the red lines
represent the means for the respective datasets.[5]

Figure 1: Boxplots of dataset with and without the outlier

Clearly, the median is much more robust against such deviations. If the
dataset has a symmetric distribution, the median and the mean will have
values which are similar or are close to each other than in a situation
where the distribution is skewed. In such a distribution the mean value,
being influenced by the outliers will most likely not be close to the
median. The median t of a data set containing data points (x 1, x2,,xn)
can be represented by the following[6]:
t= Med ( x ) , if {x i >t }= { x i <t } ,

Here we can see, a single value can have a massive impact on the mean
or any other normal statistic values as a matter of fact. If the value of the
last point was or -, then the mean also would have been infinity or
minus infinity. Hence the mean can be called an unbounded statistic and
even a single value can result in aggressive fluctuation of the value itself.
Again the median is impervious to such effects of bad data [6].

Figure 2: Inter Quartile range [7]

Standard deviation or Variance are the estimators used for describing the
spread of data. The robust counterpart for them are the Inter Quartile
Range (IQR) and Median Absolute Deviation (MAD). IQR is obtained by
dividing the dataset into quartiles and finding the difference of the first
and the third quartile. Comparing different estimators, using the example
from before. The standard deviation and the variance of the dataset with
the outlier are 10.0058 and 100.16; 1.5872 and 2.51 respectively for the
dataset without the outlier. The IQR before and after are 2.0875 and 1.99.
We can see the drastic difference between the two estimators. Inter
Quartile Range is mathematically represented by the following:
IQR = Q3 Q1; where Q3 and Q1 are the third and the first quantile values.
Median Absolute Deviation is another robust estimator to define the scale
of the data. It is similar to the standard deviation as both use the residual
or the error to determine scale. The Median Absolute Deviation is the
median of the values obtained after finding the absolute difference
between the values of the dataset and the median of the data set or the
median of the absolute values of the errors or residuals based on the
median. It is mathematically represented by the following equation [6]:
MAD ( X )=MAD ( x 1 , x 2 , , x n )=Med {xMed (x)}
The MAD value for a standard normal random variable is 0.6745 and
hence when comparing it to the standard deviation of the dataset we use
a normalized median absolute deviation represented by [6]:
MADN ( x)=

MAD( x )
0.6745

Again comparing the variations of the scale parameters, MAD value for
the dataset is 1.47 and 1.35 after removal of the big data point.

The problem arises when we try to use these robust methods for an actual
standard normal distribution. The efficiency of the robust methods is low
when compared to the classical methods. Robust methods behave in an
optimal manner when the distribution has thicker tails or when outliers
exist in the distribution and the classical methods are the better option for
an optimal result for a normal distribution and even a small deviation from
it can cause sub optimality[6].
Next we discuss an important part of statistics, regression. Regression
analysis is used to understand the relation between two variables using
various methods and fit a curve to the data to predict future values. The
most ordinary method is the Ordinary Least Squares (OLS) method which
minimizes the value of the residual from the data points to the fitted curve
or line. The most simple is linear fitting of data to a line. Again the
problem arises due to the outliers. The model is based on finding the
minimum value for sum of square of residuals of the data point form the
mean, hence in turn the model is highly sensitive e to outliers.
For an example, we take a data set containing the weights and the heights
of
10 children [8] and create a linear regression model.
Heights -> (65.78, 71.52, 69.40, 68.22, 67.79, 68.70, 69.80, 70.01, 67.90,
66.78)
Weights>(112.99,136.49,153.03,142.34,144.30,123.30,141.49,136.46,112.37,120
.67)

Figure 3: Linear Square Regression Model a) Without Outlier b) With

Outlier
When change the value of the last set to (Height=90, Weight=109), the
slope and intercept of the model change drastically from 59.6 and 0.068
to 93.40 and -0.1715. As we compare these changes in the estimators we
understand why robust estimators and robust statistics are important.
Using robust regression package in R:

Figure 5: Robust Regression models a) Without outliers

Figure 5: Robust Regression models b) With outliers

The robust fitting in R is done using M-estimators. The rlm command
uses Hubers model for regression by default. We can also use Tukeys
Bisquare method or Hampels Method. There are 3 different types of
estimators[3][8]:
a) M-estimators
b) L-estimators
c) R-estimators
M-estimators are maximum likelihood estimators and Mean is a part of
this class of estimator. L-estimators use the linear combination of order
statistics and the Median are a part of this class. R-estimators are related
to ranking residuals[9]. We mainly go over M estimators for our purposes.
In least square method, the objective is to minimize the value of sum of

residual squared. Huber in 1964 suggested M-estimators for regression

modelling[4]. There are multiple M-estimators present such as the Tukeys
Bisquare M-estimator, Hubers M-estimator and. The objective here was
the minimization of the following equation[10]:
n

(ri )
i=1

Hubers M-estimator[9]:
1
1
( z i) = z i2 , if | z i |<c (or) ( z i) =c |z i| c i2 ,if | z i | c
2
2
Where

z i = r /s. Hubers m-estimator allows for behaves well for both

i

data with and without outliers.

Tukeys Bisquare M-Estimator[9]:
( z i) =
Where

1 6
( c ( c 2z 2i ) ) , if |z i|<c
6

(Or)

( z i) =0, if zi c

z i = r /s. In Tukeys Bisquare M-Estimator any value beyond a

i

certain value is deleted. Least Absolute Deviation (LAD) is the OLS

equivalent for median[6].
Before actually deciding to use robust techniques, we must be able
determine whether a set of values has outliers. The most basic methods
to finding them is to visually represent the data by using frequency
distribution or a Boxplot. We can also use statistical methods. Using the Zscore based on the mean, outliers can be found. A modified Z-score based
on MAD can also be used. For the modified Z-score, it is recommended to
consider values greater than 3.5 as outliers[11].
Z i=

M i=

Y iY
s

0.6745( x i x )
MAD

A few basic formal outlier tests have been given below[11]. All the
following tests assume that the data approximately follows the Normal
distribution.
1) Grubbs Test:
This test is used when we are trying to find a single outlier. It is
through Hypothesis testing, with the Null hypothesis representing
No Outliers and the alternative hypothesis being the presence of

outlets in the dataset. The test statistic is G=

Y Y min
. or
s

Y maxY
s

depending on whether the lowest or the highest value is being

tested.
2) Tietjen-Moore
Test:
This is done for the detection of multiple outliers when the number
of outliers is known. This is similar to the Grubbs test w.r.t, that
both go through Hypothesis testing. If the number of outliers is
unknown, then we resort to using the Generalized Extreme Deviate
Test.
3) Generalized Extreme Studentized Deviate Test: N number of tests
are performed to test N outliers. Unlike the previous tests, ESD only
requires the upper bound to the number of outliers present.
Once we have detected outliers, they can be treated in different manners.
Trimming or truncation of these values is one method. It means
completely dropping these points from our dataset. R.R.Wilcox suggested
removal of 20% of the data on both extremes to get a dataset that is fairly
normal[12]. Another method is the Winsorization of the values lying above
a certain value. Certain upper and lower limits are set, which is usually
about three standard deviation s away from the mean. Winsorization
involves the converting of such values which lie above or below the limits
to the extreme acceptable values present at either ends or replacing the
extreme values of a dataset to the nearest acceptable neighbors[13].
Before using these techniques we must see whether the values are
actually valid or were a result of an error and how they were obtained. If
they are legitimate values then the outliers mean something. But as these
values are outliers and do not represent the mass of the data, they cannot
be used to for prediction of future values.

4. Implementation of Robust Statistics:

1) Implementation of Robust statistics In Image Processing.
Robust statistics is used here to separate local areas of images into
smaller segments along with K-means clustering. This is difficult to
do because of the noise present in the images taken in the real
world. This noise acts as outliers in this case[14]. Following is an
image between different techniques used to segment an image. The
fourth column was generated using the model developed in this
paper and performs much better than the other two methods
mentioned.

Figure 7:Comparison results for a T-shaped image. The first column:

original image with red initial contours. The second column:
segmentation results by the RSF model. Third column: segmentation
results by the MRSFBRS model. The fourth column: segmentation
results by our model.[14]
2) Robust Statistics in Ecological modeling. Robust statistics were
used to analyze environmental data. These datasets did not confirm
to a normal distribution. The data was Winsorized to get better
models from the data[15]. Following is graph using the classical and
robust methods:

Figure 6: Scatter plots of the modelling results and observations of total phosphorus load in
classical analysis and in robust statistics for the Middle Warta river in the validation process. [15]

3) Robust Statistics in Neutrinos detection software[16]:

Detection of neutrons involves the detection of muons. These
muons are a result of interaction between the neutrinos and the ice.
Reconstructing the track of these muons is crucial to identifying
whether the muons were produced by the neutrinos or not. The
IceCube project utilizes an algorithm (Single-Photo-Electron-Fit) for
the reconstruction. This algorithm implements Least Squares Model,

but it was preferred to reduce the weight of the outliers on the

reconstruction. Hence the Hubers fit replaced the least square
method in the algorithm. As Hubers fit behaves like Least Square
Method when the data values are not outliers and reduces the
weight of the outliers when required[16].

References:
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]

[16]

Robust, Merriam-Webster. [Online]. Available: http://www.merriamwebster.com/dictionary/robust. [Accessed: 30-Apr-2016].

Resilience Engineering #1: Robust Vs. Resilient. [Online]. Available:
http://www.activegarage.com/resilience-engineering-1-robust-resilient. [Accessed: 30Apr-2016].
Definition: system robustness. [Online]. Available: http://www.its.bldrdoc.gov/fs1037/dir-036/_5274.htm. [Accessed: 06-May-2016].
P. J. Huber and E. M. Ronchetti, Robust Statistics, 2nd Editio., vol. 523, no. 3. 2009.
R Statistics Essential Training Lynda.com. [Online]. Available:
http://www.lynda.com/R-tutorials/Examining-robust-statistics-univariateanalyses/142447/149952-4.html? [Accessed: 30-Apr-2016].
R. A. Maronna, R. D. Martin, and V. J. Yohai, Robust Statistics. Chichester, UK: John
Wiley & Sons, Ltd, 2006.
Box Plot Diagram to Identify Outliers. [Online]. Available:
http://www.whatissixsigma.net/box-plot-diagram-to-identify-outliers/. [Accessed: 02May-2016].
SOCR Data Dinov 020108 HeightsWeights - Socr. [Online]. Available:
http://wiki.stat.ucla.edu/socr/index.php/SOCR_Data_Dinov_020108_HeightsWeights.
[Accessed: 03-May-2016].
M-estimators. [Online]. Available: http://www.statisticalconsultants.co.nz/blog/mestimators.html. [Accessed: 03-May-2016].
M-Estimators in Regression Models - Open Access Library. [Online]. Available:
http://www.jourlib.org/paper/2828448#.VywKDzArKUk. [Accessed: 06-May-2016].
Engineering Statistics Handbook,1.3.5.17. Detection of Outliers. [Online]. Available:
http://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm. [Accessed: 02May-2016].
R. R. Wilcox, A Foundation for Robust Methods, Introd. to Robust Estim.
Hypothesis Test., pp. 2342, 2012.
W. J. Dixon and K. K. Yuen, Trimming and winsorization: A review, Stat. Hefte, vol.
15, no. 23, pp. 157170, Jun. 1974.
C. Huang and L. Zeng, Robust image segmentation using local robust statistics and
correntropy-based K-means clustering, Opt. Lasers Eng., vol. 66, pp. 187203, Mar.
2015.
M. S. Ostojski, J. Gbala, P. Orliska-Woniak, and P. Wilk, Implementation of robust
statistics in the calibration, verification and validation step of model evaluation to
better reflect processes concerning total phosphorus load occurring in the catchment,
Ecol. Modell., vol. 332, pp. 8393, Jul. 2016.
M. G. Aartsen, R. Abbasi, Y. Abdou, M. Ackermann, J. Adams, J. A. Aguilar, M.
Ahlers, D. Altmann, J. Auffenberg, X. Bai, M. Baker, S. W. Barwick, V. Baum, R. Bay,
J. J. Beatty, S. Bechet, J. Becker Tjus, K.-H. Becker, M. L. Benabderrahmane, S.

BenZvi, P. Berghaus, D. Berley, E. Bernardini, A. Bernhard, D. Z. Besson, G. Binder,

D. Bindig, M. Bissok, E. Blaufuss, J. Blumenthal, D. J. Boersma, S. Bohaichuk, C.
Bohm, D. Bose, S. Bser, O. Botner, L. Brayeur, H.-P. Bretz, A. M. Brown, R. Bruijn,
J. Brunner, M. Carson, J. Casey, M. Casier, D. Chirkin, A. Christov, B. Christy, K.
Clark, F. Clevermann, S. Coenders, S. Cohen, D. F. Cowen, A. H. Cruz Silva, M.
Danninger, J. Daughhetee, J. C. Davis, M. Day, C. De Clercq, S. De Ridder, P. Desiati,
K. D. de Vries, M. de With, T. DeYoung, J. C. Daz-Vlez, M. Dunkman, R. Eagan, B.
Eberhardt, J. Eisch, S. Euler, P. A. Evenson, O. Fadiran, A. R. Fazely, A. Fedynitch, J.
Feintzeig, T. Feusels, K. Filimonov, C. Finley, T. Fischer-Wasels, S. Flis, A.
Franckowiak, K. Frantzen, T. Fuchs, T. K. Gaisser, J. Gallagher, L. Gerhardt, L.
Gladstone, T. Glsenkamp, A. Goldschmidt, G. Golup, J. G. Gonzalez, J. A. Goodman,
D. Gra, D. T. Grandmont, D. Grant, A. Gro, C. Ha, A. Haj Ismail, P. Hallen, A.
Hallgren, F. Halzen, K. Hanson, D. Heereman, D. Heinen, K. Helbing, R. Hellauer, S.
Hickford, G. C. Hill, K. D. Hoffman, R. Hoffmann, A. Homeier, K. Hoshina, W.
Huelsnitz, P. O. Hulth, K. Hultqvist, S. Hussain, A. Ishihara, E. Jacobi, J. Jacobsen, K.
Jagielski, G. S. Japaridze, K. Jero, O. Jlelati, B. Kaminsky, A. Kappes, T. Karg, A.
Karle, J. L. Kelley, J. Kiryluk, J. Kls, S. R. Klein, J.-H. Khne, G. Kohnen, H.
Kolanoski, L. Kpke, C. Kopper, S. Kopper, D. J. Koskinen, M. Kowalski, M.
Krasberg, K. Krings, G. Kroll, J. Kunnen, N. Kurahashi, T. Kuwabara, M. Labare, H.
Landsman, M. J. Larson, M. Lesiak-Bzdak, M. Leuermann, J. Leute, J. Lnemann, O.
Macas, J. Madsen, G. Maggi, R. Maruyama, K. Mase, H. S. Matis, F. McNally, K.
Meagher, M. Merck, T. Meures, S. Miarecki, E. Middell, N. Milke, J. Miller, L.
Mohrmann, T. Montaruli, R. Morse, R. Nahnhauer, U. Naumann, H. Niederhausen, S.
C. Nowicki, D. R. Nygren, A. Obertacke, S. Odrowski, A. Olivas, A. Omairat, A.
OMurchadha, L. Paul, J. A. Pepper, C. Prez de los Heros, C. Pfendner, D. Pieloth, E.
Pinat, J. Posselt, P. B. Price, G. T. Przybylski, L. Rdel, M. Rameez, K. Rawlins, P.
Redl, R. Reimann, E. Resconi, W. Rhode, M. Ribordy, M. Richman, B. Riedel, J. P.
Rodrigues, C. Rott, T. Ruhe, B. Ruzybayev, D. Ryckbosch, S. M. Saba, T. Salameh,
H.-G. Sander, M. Santander, S. Sarkar, K. Schatto, F. Scheriau, T. Schmidt, M.
Schmitz, S. Schoenen, S. Schneberg, A. Schnwald, A. Schukraft, L. Schulte, O.
Schulz, D. Seckel, Y. Sestayo, S. Seunarine, R. Shanidze, C. Sheremata, M. W. . Smith,
D. Soldin, G. M. Spiczak, C. Spiering, M. Stamatikos, T. Stanev, A. Stasik, T.
Stezelberger, R. G. Stokstad, A. Stl, E. A. Strahler, R. Strm, G. W. Sullivan, H.
Taavola, I. Taboada, A. Tamburro, A. Tepe, S. Ter-Antonyan, G. Tei, S. Tilav, P. A.
Toale, S. Toscano, E. Unger, M. Usner, S. Vallecorsa, N. van Eijndhoven, A. Van
Overloop, J. van Santen, M. Vehring, M. Voge, M. Vraeghe, C. Walck, T. Waldenmaier,
M. Wallraff, C. Weaver, M. Wellons, C. Wendt, S. Westerhoff, N. Whitehorn, K.
Wiebe, C. H. Wiebusch, D. R. Williams, H. Wissing, M. Wolf, T. R. Wood, K.
Woschnagg, D. L. Xu, X. W. Xu, J. P. Yanez, G. Yodh, S. Yoshida, P. Zarzhitsky, J.
Ziemann, S. Zierke, M. Zoll, B. Recht, and C. R, Improvement in fast particle track
reconstruction with robust statistics, Nucl. Instruments Methods Phys. Res. Sect. A
Accel. Spectrometers, Detect. Assoc. Equip., vol. 736, pp. 143149, Feb. 2014.