c o m p u t e r l a w & s e c u r i t y r e v i e w 2 7 ( 2 0 1 1 ) 4 5 e5 2

available at www.sciencedirect.com

www.compseconline.com/publications/prodclaw.htm

The limits of privacy in automated profiling and data mining

Bart W. Schermer 1

abstract

Keywords: Automated profiling of groups and individuals is a common practice in our information
Automated profiling society. The increasing possibilities of data mining significantly enhance the abilities to
Data mining carry out such profiling. Depending on its application, profiling and data mining may cause
Privacy particular risks such as discrimination, de-individualisation and information asymmetries.
In this article we provide an overview of the risks associated with data mining and the
strategies that have been proposed over the years to mitigate these risks. From there we
shall examine whether current safeguards that are mainly based on privacy and data
protection law (such as data minimisation and data exclusion) are sufficient. Based on
these findings we shall suggest alternative policy options and regulatory instruments for
dealing with the risks of data mining, integrating ideas from the field of computer science
and that of law and ethics.
ª 2011 Dr. Bart W. Schermer. Published by Elsevier Ltd. All rights reserved.

1. Introduction private companies and institutions to help overcome the

Profiling is the process of discovering correlations between
data in databases that can be used to identify and represent
a human or nonhuman subject (individual or group) and/or the
application of profiles (sets of correlated data) to individuate
and represent a subject or to identify a subject as a member of
a group or category.2 Profiling is used for purposes ranging
from anti-terrorism to direct marketing. Profiling relies heavily
on data mining for its effectiveness. Data mining, or knowl-
edge discovery in databases (KDD), is the nontrivial extraction
of implicit, previously unknown, and potentially useful infor-
mation from data.3 Over the past decades it has evolved from
an experimental technology into an important instrument for
problem of information overload.
While the benefits of profiling and data mining for public
and private sector institutions are significant, misuse or abuse
of profiling may have negative consequences for individuals
and society as a whole. Much has been written over the years
about the possible negative effects of profiling and data
mining.4 Often, the negative effects of profiling and data
mining are framed as threats to (informational) privacy.
Subsequently, proposed regulatory solutions are predomi-
nantly aimed at protecting privacy. While this approach might
provide some measure of protection, it is questionable
whether it is the most effective regulatory approach. It might
be that a more targeted (regulatory) strategy is necessary to

1
Bart W. Schermer is an assistant professor at eLaw@Leiden, the centre for law in the information society of Leiden University. This
article was written as part of the research project ‘Data mining without discrimination’. Bart wishes to thank the project team (Bart
Custers, Toon Calders, Sicco Verwer and Tal Zarsky) for their valuable comments on the original draft.
2
Hildebrandt, M. (2008), Defining Profiling: A New Type of Knowledge? In: Profiling the European Citizen, Cross-Disciplinary Perspectives
(Hildebrandt, M., Gutwirth, S., eds.), Springer Science, p. 17.
3
Frawley, W.J., Piatetsky-Shapiro, G., Matheus, C.J. (1992), Knowledge Discovery in Databases: An Overview. In: AAAI Magazine, Fall
1992, p. 58.
4
See for instance: Marx, G.T. (1985), The surveillance society: the threat of 1984-style techniques. In: The Futurist, June 21-6; Vedder, A.
(1999), KDD: The challenge to individualism. In: Ethics and Information Technology, Volume 1, Number 4/December, 1999; Lyon, D. (2003),
Surveillance as social sorting, privacy risk and digital discrimination, New York: Routledge; Custers, B. (2003), Effects of Unreliable Group
Profiling by means of Data Mining. In: Lecture Notes in Computer Science, Volume 2843/2003, Springer; and Hildebrandt, M., Gutwirth, S.
(2008), Profiling the European Citizen, New York: Springer Verlag.
0267-3649/$ e see front matter ª 2011 Dr. Bart W. Schermer. Published by Elsevier Ltd. All rights reserved.
doi:10.1016/j.clsr.2010.11.009
46 c o m p u t e r l a w & s e c u r i t y r e v i e w 2 7 ( 2 0 1 1 ) 4 5 e5 2
address these issues. In this article we shall explore the risks
associated with automated profiling and data mining and
examine whether privacy is still a viable regulatory approach
to mitigate these risks, or that additional safeguards are
necessary. The twofold problem statement to be explored in
this article is therefore:
Is privacy an effective approach when it comes to dealing with
possible negative effects of data mining?
Are there other strategies available that might prove more effective?
In this article we shall approach these questions from
a multi-disciplinary perspective, since this article was written
as part of a scientific project that focuses specifically on the
cooperation between the fields of computer science and law.5
2. Data mining approaches in profiling
When we study the use of data mining for profiling purposes,
we can distinguish among two distinct data mining approaches,
viz., “descriptive data mining” and “predictive data mining”.
2.1. Descriptive data mining
The goal of descriptive data mining is to discover unknown
relations between different data objects in a database.
Descriptive data mining algorithms try to discover knowledge
about a certain domain by determining commonalities
between different objects and attributes. By discovering
correlations between data objects in a dataset that is repre-
sentative of a certain domain, we can gain insight to it.6
Descriptive data mining is interesting because more insight
into a particular domain allows for better planning and allo-
cation of resources in this realm. For instance, descriptive data
mining on tax declaration forms may find out a natural divi-
sion of people into a limited number of groups based on their
economical activity, financial situation and occupation. Also,
descriptive data mining could expose relations between the
different fields on the forms. It is important to notice here that
in descriptive data mining no “target” has been given. There-
fore, descriptive data mining is often also called “unsuper-
vised data mining”. Moreover, descriptive data mining
describes relations rather than explains them. The existence
of a correlation in a dataset does not necessarily mean that
this relation will always occur in the real world, nor does it
explain why the correlation is there. In other words, it is vital
not to mix up correlation with causation when it comes to
descriptive data mining.
2.2. Predictive data mining
As the name implies, the goal of predictive data mining is to
make a prediction about events based on patterns that were
5
NWO Project Data mining without discrimination. For infor-
mation see: http://www.onderzoekinformatie.nl/en/oi/nod/
onderzoek/OND1337064/
6
Cocx (2009), Algorithmic tools for data-oriented law enforcement,
PhD. Thesis, University of Leiden, p. 3.
determined using known information. When it comes to
profiling, this means that information about an individual is
mined in order to determine whether he/she fits the previously
established profile. The training data in predictive mining
therefore usually consists of a collection of annotated objects or
individuals; e.g., demographics of a set of individuals together
with the annotation which are known terrorists. As such,
predictive data mining is often called “supervised data mining”.
An often-used method in predictive data mining is classi-
fication. By using data mining algorithms called classifiers, we
can establish whether a new object fits a previously estab-
lished class. Classes are based on input fields that contain
different attributes associated with the class. When an object
shares attributes with the data objects belonging to the class,
it is likely that it also belongs to this class. For instance, the
class label “canary” would be made up of attributes such as
{yellow, beak, wings, tail}. When an object has a certain
amount of these attributes, it becomes increasingly likely that
it is indeed a canary.
It is important to note that classification only determines
the likelihood of something belonging to a certain class. When
classification is based on few determining factors (e.g.,
something is either 1, or it is not 1) classification can be quite
accurate. However, when a certain event or occurrence is
dependent on a multitude of factors, accurate classification
becomes more difficult, if not impossible. Also, often many of
the factors that determine the class of the object are not
present in the dataset. For instance, whether or not someone
will grow up to be a criminal is dependent on so many factors,
that accurate classification (i.e., criminal or not criminal) is
impossible. As such, the outcome of a classifier on a new
object will always be of a probabilistic nature.
3. Ethical and legal issues associated with
profiling and data mining
While profiling and data mining have proven to be very useful
tools in dealing with the information overload, they are not
without controversy. The reason for this is that there are
a number of ethical and legal issues associated with profiling
and data mining, some of which I shall describe in this Section
I shall distinguish between the risks associated with profiling,
and issues that may arise due to an incorrect application of
data mining in the context of profiling.
3.1. Risks associated with profiling
While data mining and profiling are for the most part
conceptually framed as threats to informational privacy, it is
our opinion that the strong association with privacy obfus-
cates the actual risks that profiling and data mining may pose
to groups and individuals. In our view the most significant
risks associated with profiling and data mining are discrimi-
nation, de-individualisation and information asymmetries.
While Article 15 of the European Data protection directive
(1995/46/EC) addresses the issue of automated profiling and
the risks associated with it, it does not feature prominently in
our current privacy discourse.
 c o m p u t e r l a w & s e c u r i t y r e v i e w 2 7 ( 2 0 1 1 ) 4 5 e5 2
3.1.1. Discrimination
Classification and division are at the heart of (predictive) data
mining. As such, discrimination is part and parcel of profiling
and data mining. However, there are situations where
discrimination is considered unethical and even illegal. This
can occur for instance when a data mining exercise is
focussed on characteristics such as ethnicity, gender, religion
or sexual preference. But even without a prior desire to judge
people on the basis of particular characteristics, there is the
risk of inadvertently discriminating against particular groups
or individuals. The reason for this is that predictive data
mining algorithms may “learn” to discriminate on the basis of
biased data used to train the algorithm.
A classifier must be trained in order to classify data. When
the training data is contaminated (for instance because it
discriminates against a particular group), the classifier will
learn to classify in a biased way, strengthening discriminatory
effects. Such cases occur naturally when the decision process
leading to the labels was biased. An example in the area of law
enforcement may help to explain this. When police officers
have targeted an ethnic minority disproportionately in the past
based on their own bias, it is likely that these minorities will
feature more prominently in crime statistics. If these crime
statistics are then used as training data for a classifier, chances
are high that the classifier will learn that that there is a strong
correlation between ethnicity and crime. This in turn will lead
to discriminating results that can constitute the basis for future
discrimination. This effect is further strengthened by the fact
that a classifier will most likely not have access to all the
important factors on which to base a prediction because, e.g.,
they are missing in the data. Therefore, the importance of those
factors that are present in the data grows and will be even more
important in prediction than they were in the input data.
3.1.2. De-individualisation
In many cases data mining is in large parts concerned with
classification and thus there is the risk that persons are judged
on the basis of group characteristics rather than on their own
individual characteristics and merits.7 Group profiles usually
contain statistics and therefore the characteristics of group
profiles may be valid for the group and for individuals as
members of that group, though not for individuals as such. An
example may illustrate this. For instance when people in
Amsterdam are 3% more criminal than people in the rest of
the Netherlands, this characteristic goes for the group (i.e.,
people in Amsterdam), for the individuals as members of that
group (i.e., randomly chosen people living in Amsterdam), but
not for the individuals as such (i.e., for John, Mary and William
who all live in Amsterdam). When individuals are judged by
group characteristics they do not possess as individuals, this
may strongly influence the advantages and disadvantages of
using group profiles.8 Apart from the negative effects group
7
Vedder, A. (1999), KDD: The challenge to individualism. In:
Ethics and Information Technology, Volume 1, Number 4/December,
1999.
8
Custers, B.H.M. (2010), Data Mining with Discrimination Sensitive
and Privacy Sensitive Attributes. Proceedings of ISP 2010, Interna-
tional Conference on Information Security and Privacy, 12e14
July 2010, Orlando, Florida (forthcoming).
47
profiling may have on individuals, group profiling can also
lead to stigmatisation of group members. Moreover, divisions
into groups can damage societal cohesion. When group
profiles, whether correct or not, become public knowledge,
people may start treating each other accordingly. For instance,
when people start believing that citizens of Amsterdam are
more criminal, people may start to react and communicate
with more suspicion towards citizens of Amsterdam, regard-
less of the correctness of such a profile.
3.1.3. Information asymmetries
Data mining can lead to valuable insights for those parties
employing it. When data mining is aimed at gaining more
insight into individuals or groups, we encounter the problem
of information asymmetry. Information asymmetries may
influence the level playing field between government and
citizens, and between businesses and consumers, upsetting
the current balance of power between different parties.
In the context of the relation between government and
citizens information asymmetries can affect individual
autonomy. If data mining indeed yields actionable knowledge,
the government will have more power. Moreover, the fear of
strong data mining capabilities on the part of the government
may “chill” the willingness of people to engage in for instance
political activities, out of fear of being watched. For this
panoptic fear to materialise, a data mining application does not
even have to be effective.9
Information asymmetries may upset the level economic
playing field between consumers and businesses. Further-
more, there are instances where data mining can aid in
making decisions about consumers that are considered
unwanted, unethical or illegal. An example of this would be
excluding particular individuals or groups from goods and
services based on their ethnicity.10
The issue of information asymmetry is exacerbated by the
limited transparency of data mining. Since data mining (in
particular predictive data mining) is used to make decisions
about groups and individuals, people will be affected by data
mining exercises. However, for the most part it will be unclear
to persons why a particular decision has been made and on
what grounds. This could lead to a sense of helplessness. A
problem that is compounded by the fact that it is difficult to
seek redress from automated decision processes. Professor of
law at the George Washington University Law School, Daniel
Solove has likened this situation with that of Josef K. in Kaf-
ka’s Der Prozess.11
3.2. Application issues
The risks described above may manifest themselves regard-
less of the fact that the data mining was applied correctly in
9
Schermer, B.W. (2007), Software Agents, Surveillance, and the
Right to Privacy: a Legislative Framework for Agent-enabled
Surveillance, PhD. Thesis, Leiden University, p. 42.
10
Zarsky, T.Z. (2006), Online Privacy, Tailoring, and Persuasion.
In: Privacy and technologies of Identity, a cross disciplinary conversa-
tion, Strandburg, K., Stan Raicu, D. (eds), Chapter 12, pp. 209e224,
Springer, 2006.
11
Solove, D.J. (2004), The Digital Person: Technology and Privacy in
the Information Age. New York: New York University Press, p. 47.
48 c o m p u t e r l a w & s e c u r i t y r e v i e w 2 7 ( 2 0 1 1 ) 4 5 e5 2
a technical sense. But as we have seen in the example of
discrimination, when data mining is applied incorrectly, the
risks associated with profiling and data mining may be
strengthened. Therefore we shall discuss several common
application issues associated with data mining that may pose
additional risks to groups and individuals.
3.2.1. Accuracy and reliability
The success of a data mining exercise is dependent on the
quality of the raw data being mined. If the data is inaccurate,
the results will also be inaccurate. This is true for both
descriptive and predictive data mining. In the area of predic-
tive data mining issues with accuracy and reliability are
particularly problematic, given the fact that the results of
a predictive data mining exercise are oftentimes used to make
(automated) decisions about individuals and/or groups. But
even if the raw data is free of errors, accuracy and reliability
remain an issue.
In particular there is the problem of “false positives” and
“false negatives.” This means that people that in fact do not fit
the class are fitted in the class (a false positive), or people that fit
the class are left out (false negative). False positives and false
negatives occur for various reasons, one being that not all
information is available. For instance, in the example of the
“canary classification”, the presence of attributes such as
{yellow, beak, wings, tail} were strong indications that an object
could be classified as a canary. However, a duckling is also
yellow and has a beak, wings and a tail, making it a false positive
in our scenario.
3.2.2. Causation versus correlation
The goal of data mining is to find implicit and previously
unknown relations between data. As such, data mining yields
new knowledge about a given problem space. In descriptive
data mining, this knowledge is based on the correlation
between certain objects and attributes. However, while data
mining can establish that there is a relationship between
certain objects and attributes, it does not explain why this
relationship exists. As such, it is important that we do not
mistake correlation for causation.12 For instance, data mining
may reveal that burglars use cannabis more often than other
people. While it is tempting to point to cannabis as a cause for
burglary, the data does not support such a conclusion.
Data mining experts warn for the fact that a correlation
between certain facts does not imply a causal relation, nor
does it explain why there is a correlation between facts.13 Such
a warning needs to be heeded, given the fact that data mining
efforts and statistics might provide input for policymaking.
Since the goal of policymaking is addressing the causes of an
issue rather than fighting symptoms, it is important to know
more about the background, emergence and causation of
certain events. In particular unguided descriptive data mining
is less suited, if not unfit for the discovery of this information.13
12
See for an in depth analysis of causation: Pearl, D. (2009),
Causality: Models, Reasoning, and Inference, Cambridge: Cambridge
University Press, (2nd edition).
13
Cocx, (2009), Algorithmic tools for data-oriented law enforcement,
PhD. Thesis, University of Leiden, p. 143.
3.2.3. Data dredging
In unguided descriptive data mining we look for correlations in
the data without using a pre-defined working hypothesis.
Dependent on the size of the dataset and the “confidence
interval” used to determine correlations, our data mining
exercise will yield certain results. While these results might
indeed be significant, there is also a chance that they are
completely random. So, the results we find (and the hypothesis
we formulate on the basis of these results) need to be validated
to exclude the possibility that the correlation is in fact totally
random. It is important to note that the dataset used in con-
structing the hypothesis cannot be used for the validation of
that same hypothesis, since this data is by default “biased”
towards supporting that particular hypothesis. The risk in data
dredging is that we present the results of the initial data
mining exercise as facts, rather than as a hypothesis that
needs to be tested further. An example to illustrate this point is
the following: suppose we want to test if some people have
special mental abilities and are able to predict the future. We
set up an experiment in which 1000 volunteers need to predict
which sequence of heads and tails will emerge from flipping an
unbiased coin 10 times. Statistics teach us that if a participant
makes random guesses, he or she still has a chance of 1 out of
1024 to get the sequence fully correct. As such, we can expect
that on average one participant will predict the outcome of all
10 tosses correctly. Of course, this outcome does not support at
all the claim that this participant has special mental abilities.
First the participant needs to confirm his perfect scores in new
controlled experiments. To link this example to data mining:
one could consider the outcome of the prediction task as the
input data. There are 1000 hypotheses being tested: “partici-
pant 1 can predict everything correctly” till “participant 1000
can predict everything correctly”. As such the outcome of the
experiment is used to generate the “pattern”; e.g., “participant
174 can predict everything correctly” that will be presented to
the user. It is instructive to see that even a statistical test for
significance will confirm the validity of this pattern.
4. The problem with privacy
In the previous section we have discussed several risks asso-
ciated with data mining in the context of profiling. The current
approach to mitigating these risks is by invoking the right to
(informational) privacy. At the core of informational privacy is
the notion that data subjects have the right and the ability to
shield personal data from third parties. Given the societal
importance of the free flow of information, this right to
informational privacy is balanced by the legitimate interests
of third parties to process personal data. In Europe, the Data
Protection Directive sets forth the rules under which data
controllers are allowed to process personal data on individ-
uals. A data controller is not allowed to process personal data
unless he has a legitimate purpose (see article 7 of the Direc-
tive). Furthermore, the Directive sets forth some core data
protection principles such as data quality, security safeguards
and data minimisation.
Personal data protection is important and has provided
individuals with protection from misuse and abuse of their
 c o m p u t e r l a w & s e c u r i t y r e v i e w 2 7 ( 2 0 1 1 ) 4 5 e5 2
personal data. There are however a number of closely linked
issues of a technological, legal and societal nature with this
approach.
From a data mining perspective the primary issue with
informational privacy is that by limiting the use of (particular)
personal data, we run the risk of reducing the accuracy of the
data mining exercise.10 So while privacy may be protected, the
utility of the data mining exercise is reduced. Informational
privacy does not only decrease the usefulness of data mining,
it also increases the probabilities of false positives and false
negatives. Moreover, data exclusion and data minimisation do
not necessarily provide sufficient protection from the risks of
profiling, because it is often still possible to uniquely identify
particular persons from the data, even after key attributes
such as name, address and social security number have been
removed. This indirect identification is troublesome from
a legal perspective, since personal data protection law is so
dependent on the notion of personal data. Also, it is difficult to
determine which pieces of data can lead to the identification
of a person once combined. But even in those instances where
identification is impossible, there is no guarantee that indi-
viduals are not adversely influenced. Take for instance the
practice of behavioural targeting in online advertising. Actual
identification is not necessary for targeting, merely being able
to individualise users on the basis of their IP-address or their
browser cookies is enough for the targeting to be effective.
Privacy and data protection law provide little protection
against this issue, since they are primarily focussed on a priori
privacy protection through the minimisation of data.14
It may even be so that invoking the right to informational
privacy is counterproductive in some cases. New research
suggests that this is the case for anti-discrimination.15 In order
to counter discrimination, the Data Protection Directive
prohibits the processing of sensitive attributes such as
ethnicity or religion. However, it is questionable whether this
approach is effective. In general it is to be expected that other
demographic attributes in the dataset still allow for an accu-
rate identification of the ethnicity of a person. For example, the
ethnicity of a person might be strongly linked with the postal
code of his residential area, leading to indirect racial discrim-
inatory based on postal code. This effect and its exploitation is
often referred to as redlining, stemming from the practice of
denying or increasing services such as, e.g., mortgages or
health care to residents in certain often racially determined
areas.8 Research shows that even after removing the sensitive
attribute from the dataset discrimination persists.16 Since
discrimination may also be based on underlying “neutral”
14
Zarsky, T.Z. (2002), Mine Your Own Business! Making the Case
for the Implications of the Data Mining of Personal Information in
the Forum of Public Opinion. In: Yale Journal of Law and Technology,
pp. 1e57.
15
Verwer and Calders (2010), Three Naive Bayes Approaches for
Discrimination-Free Classification. In: Data Mining: special issue
with selected papers from ECMLePKDD 2010; Springer.
16
Pedreschi, D., Ruggieri, R., and Turini, F. (2008), Discrimination-
aware Data Mining. In: Proceedings of the 14th ACM SIGKDD Confer-
ence on Knowledge Discovery and Data Mining; Kamiran, F., Calders,
T. (2009), Classification without discrimination, IEEE International
Conference on Computer, Control & Communication (IEEE-IC4);
IEEE press.
49
attributes such as area code and income, we need a mecha-
nism to determine whether this is actually the case. Removing
sensitive attributes from the dataset may deny us the means to
check ex post whether an algorithm has indeed discriminated
against a certain group. The reason for this is that by removing
sensitive attributes from the data, we merely remove the key
indicators of discriminatory results, but not other, more indi-
rect indicators that may lead to similarly discriminating
results.
A final issue concerns user consent in relation to the
transparency of automated profiling. While the data subject
may give his or her consent for the processing of personal data
and the use of automated profiling, it will likely be unclear to
the data subject what the actual extent and impact of profiling
will be on his person. As such, data subjects may underestimate
the risks of automated profiling. Moreover, consenting with
automated profiling may yield direct benefits for consumers
(such as free goods or services), whereas the risks of automated
profiling may be less clear and tangible. This raises the question
to what extent the data subject can freely give his or her
informed consent.
From the above considerations we may conclude that the
concept of privacy and its application in data protection law
does not provide adequate protection from the risks associated
with automated profiling. Moreover, privacy law does not deal
adequately with the application issues mentioned in this
article. While the principle of data quality (article 6 of the Data
Protection Directive) and the right not be subjected to a deci-
sion solely based on the automated processing of data (article
15 of the Data Protection Directive) should provide some
measure of protection against application issues, in practice
their effect is limited. In part this can be attributed to the fact
that the data mining community (adept at spotting application
issues) and the legal community do not co-operate enough.
These technological and legal-technical issues highlight
a key weakness in the current approach to data protection and
informational privacy. Privacy and data protection law is
based primarily on ex ante protection, but has little in the way
of ex post protection mechanisms. This has everything to do
with the nature of privacy as a mechanism for hiding. We can
say that privacy in many cases is a means rather than an end
in itself.17 The actual underlying goals of privacy protection
(autonomy, equal treatment, economic equality) are protected
through the right to privacy. However, once this layer of ex
ante protection (i.e., the possibility to hide or shield informa-
tion from observation) has been breached, there are few
mechanisms for protecting the underlying interests. Because
of the nature of information processing in today’s hyper-
connected network society, this layer of ex ante protection is
becoming weaker and weaker.
Moreover, a priori privacy protection also carries with it
particular political issues. While privacy is a powerful meme
in the political debate about fair information processing, it is
also oftentimes seen as the antithesis of values that benefit
from openness such as security, innovation and efficiency.
Furthermore, given the fact that privacy is construed as an
17
Schermer, B.W. (2007), Software Agents, Surveillance, and the
Right to Privacy: a Legislative Framework for Agent-enabled
Surveillance, PhD. Thesis, Leiden University, p. 198.
50 c o m p u t e r l a w & s e c u r i t y r e v i e w 2 7 ( 2 0 1 1 ) 4 5 e5 2
individual right it often loses out in the public debate, because
it is positioned against the interests of society as a whole.18
5. The way forward
We have established that while a useful mechanism for
reducing potential risks of profiling and data mining, infor-
mational privacy by itself does not provide adequate protec-
tion. The current starting point of a priori privacy protection,
based on access restrictions on personal information, is
increasingly inadequate in a world of automated and inter-
linked databases and information networks. In most cases, the
output is much more relevant, since this is the information
that is being used for decision making. It thus seems useful to
complement a priori access controls with a posteriori account-
ability.19 Currently there is very little accountability and over-
sight when it comes to data mining. For the most part this is
because the data mining process is a black box for outsiders.
The black box nature of data mining is compounded by the fact
that the persons traditionally involved in assessing the legality
of (business) processes (policymakers and legal professionals)
are in large parts unable to assess effectively the legality of
data mining algorithms. In part this is because organisations
are not open and transparent about their decision-making
processes, but the fact that policymakers and legal profes-
sionals often lack the required knowledge of computer science
and mathematics also plays a significant role.
5.1. Descriptive data mining
Since descriptive data mining is primarily used for decision
support tasks and yields input for policymaking, it may
(indirectly) affect the (legal) position of groups and individ-
uals.20 While data mining professionals caution that data
mining results are not always directly suited as input for
policymaking, statistics and automated profiles created using
descriptive data mining play an important role in policy-
making and business decisions nonetheless. Measures should
therefore be taken to identify and avoid errors that result from
an incorrect application or interpretation of descriptive data
mining. Problems with accuracy and reliability, mistaking
correlation for causation, and data dredging are examples of
application issues that need to be taken into account when it
comes to descriptive data mining.
Above all, an increased awareness with data subjects and
data controllers about the possibilities, limits and risks of data
mining is necessary. One of the reasons for introducing article
15 in the Data Protection Directive is that human decision
makers are inclined to attach too much weight to the results of
automated decision-making tools, seeing their results as
18
Schermer, B.W. (2007), Software Agents, Surveillance, and the
Right to Privacy: a Legislative Framework for Agent-enabled
Surveillance, PhD. Thesis, Leiden University, p. 115.
19
Weitzner et al. (2006), Transparent Accountable Data Mining: New
Strategies for Privacy Protection, MIT Technical Report.
20
The recent retrial and subsequent acquittal of Dutch murder
suspect Lucia de Berk after serving six years in prison highlights
the impact faulty (interpretation of) statistics may have on
a person’s life.
apparently objective and incontrovertible.21 The same pattern
can be seen in politics; much, if not too much, weight is
attached to data mining results and statistics. Therefore, the
need for awareness also extends to policymakers.
While it would be beneficial to have clear cut procedural
mechanisms to verify and validate the results of data mining
exercises used to support the political decision-making
process, such mechanisms are hard if not impossible to design.
Rather, we should look towards educating policymakers about
the limits of data mining and the interpretation of results.
Another approach might be to get a “second opinion” from data
mining experts in collaboration with legal experts for data
mining results that play a significant role in decision- or
policymaking.
5.2. Predictive data mining
Predictive data mining is particularly suited for making deci-
sions in concrete cases. In particular the practice of classifi-
cation is used to make decisions about groups and individuals
(for instance flagging someone as a potential terrorist, or
denying someone a loan). Protection against the risks of
predictive data mining should be primarily aimed at making
the decision-making process more transparent, removing any
possibilities for the inaccurate or incorrect application of
predictive data mining algorithms, and detecting any illegal or
unethical decisions (most likely a posteriori).
Consider, e.g., a bank that wants to use historical informa-
tion on personal loans to learn models for predicting for new
loan applicants the probability that they will default their loan.
It could very well be that this data shows that members of
certain ethnic groups are more likely to default their loan.
Nevertheless, from an ethical and legal point of view it is
unacceptable to use the ethnicity of a person to deny the loan to
him or her, as this would constitute an infringement of the
discrimination laws. In such cases, the ethnicity of a person is
likely to be an information carrier rather than a distinguishing
factor; people from a certain ethnic group are more likely to
default their loan because, e.g., the average level of education in
this group is lower. In such a situation it is in general perfectly
acceptable to use level of education for selecting loan candi-
dates, even though this would lead to favouring one ethnic
group over another. The bank could legally decide to split up the
group of loan applicants according to their education level, and
learn more fine-grained models for each of these groups sepa-
rately. A prerequisite for this grouping or stratification approach
is of course that the attribute education level is present in the
dataset. When this attribute is not present in the dataset,
because, e.g., it was not recorded, we are in a situation that using
ethnicity, directly or indirectly, will improve predictive accu-
racy. Currently, data mining algorithms for mining predictive
models use accuracy as single most important quality measure
and criterion to be optimised. In cases as described above, this
behaviour, however may lead to discriminative models.
5.2.1. Accountability
It must be noted that in individual cases it is almost impossible
to prove discrimination; given the abundance of data available,
21
COM (92) 422 finaleSYN 287, 15 October 1992, p. 26.
 c o m p u t e r l a w & s e c u r i t y r e v i e w 2 7 ( 2 0 1 1 ) 4 5 e5 2
almost always it is possible to find ex post a neutral explanation
why the person was denied loan, marked as a terrorist, et
cetera. Unless striking evidence such as internal memos
explicating the discriminative procedure that was followed,
data only is insufficient. Therefore, the decision procedure
itself should be part of the discrimination assessment.
There are two (complementary) approaches to enhance
accountability in predictive data mining.22 The first is to
improve detection of privacy and ethical sensitivities in data
mining results ex ante. Fule and Roddick for instance have
proposed a system that weighs the sensitivity of data processing
and flags those algorithms that are most likely to contain ethical
and privacy sensitivities.23 Schreuders has proposed a more
generic approach that focuses on reviewing data mining and
decision-making processes using a set of targeted questions.24
The second approach is to determine whether an algorithm
has discriminated ex post. This is an area that is still in large
parts experimental. Ruggieri et al. suggest an approach tailored
specifically towards detecting possible discrimination in data
mining.25 In their approach, Ruggieri et al. use data of individ-
uals enriched by the decision that was taken for these individ-
uals to detect rules that specify subgroups in the data for which
the decisions significantly differ from the overall decisions.
5.2.2. Transparency
Closely related to the issue of accountability is that of trans-
parency. One of the key concerns regarding profiling and data
mining is the lack of transparency and the subsequent limits
of redress from automated decision making. Greater trans-
parency is therefore important when it comes to the legiti-
mate use of profiling and application of data mining. The
European Data Protection Directive already contains a provi-
sion (article 15) that states that data subjects may not be
subjected to automated decision-making processes that have
legal effect or otherwise significantly influence their position.
In the current privacy discourse however, article 15 plays but
a minor role. Norwegian Data Protection expert Bygrave has
argued that article 15 should be regarded as one of the core
data protection principles.26 We agree with Bygrave on this
point and argue further that more attention should be given to
the information notice requirements of articles 10, 11 and 12
of the Directive. Apart from informing the data subject about
the purpose of the information processing, data subjects
should have more knowledge about the actual decision-
making process underlying an automated decision.27 Article
12, paragraph a, third dash, of the Directive already states that
22
I use the term ‘accountability’ here in the sense of answerability.
23
Fule, P., Roddick, J.F. (2006), Detecting privacy and ethical
sensitivity in data mining results. In: Proceedings of the 27th Aus-
tralasian conference on Computer science, Volume 26, pp. 159e166.
24
Schreuders, E. (2001), Data mining, de Toetsing van Beslisregels en
Privacy, ITeR reeks 48 (in Dutch).
25
Ruggieri, S., Pedreschi, D., and Turini, F. (2010), Integrating
induction and deduction for finding evidence of discrimination.
In: Artificial Intelligence & Law, 18:1e43; Springer.
26
Bygrave, L. (2001), Minding the machine: article 15 of the EC
data protection directive and automated profiling. In: Computer
Law & Security Report, 2001, Volume 17, pp. 17e24.
27
Groothuis, M. (2003), Beschikken en digitaliseren: over normering
van de elektronische overheid, PhD Thesis, Leiden University, p. 69
(in Dutch).
51
data subjects should be informed about the logic and rationale
behind the decision-making process, but as of yet the appli-
cation of Article 12 in practice seems very limited. The legal
duty to be transparent should carry over into the design of
data mining algorithms and accompanying IT-infrastructures,
i.e., “transparency by design”. In this area Weitzner et al. for
instance, propose a system in which the user can verify
certain data about him- or herself and check whether auto-
mated decision processes are indeed correct.19 Such a system
of transparency supplemented by user control could, in large
parts, take away privacy fears that stem from the “Kafka
metaphor”. Furthermore, it could help in reducing false posi-
tives. However, it must be noted that it does not take away any
“Big Brother” effects (rather it could strengthen and enforce
these effects as users are more aware ethrough con-
frontatione about the power of data mining), nor does trans-
parency in itself legitimise the use of personal data.
5.2.3. Risk aware data mining algorithms
Apart from detecting of data mining algorithms that might
adversely affect groups and individuals, we must ensure that
these risks are mitigated through the programming code
itself. This application of Lessig’s idea of “Code as Law” can
take various forms.28
The development of privacy-preserving data mining algo-
rithms has been a field of research for quite some time now.
The notion of privacy aware data mining algorithms, suggested
amongst other by Agrawal and Srikant, uses methods such as
the exclusion or masking of sensitive attributes to preserve
privacy.29 However, as described above, what is likely more
effective is to look beyond privacy (privacy being a means
rather than an end in itself) and program algorithms to become
“aware” of the underlying risks such as discrimination. In this
approach we would need to instruct the system on which
information it should (or should not) base its decisions.30 In the
case of discrimination for instance, it is desirable to have
a means to “tell” the algorithm that its predictions should be
independent of sensitive attributes such as sex and ethnicity.
We call such constraints “independency constraints”.31 By
using these constraints, we can structure systems that are
discrimination aware by design.
6. Conclusions
While profiling and data mining are important tools for
dealing with information overload, it is of vital importance
that they are used in a responsible manner as to avoid risks to
groups and individuals. We have established that risks asso-
ciated with descriptive and predictive data mining include
28
Lessig, L. (2006), Code 2.0, New York: Perseus Book Group.
29
Agrawal, R., Srikant, R. (2000), Privacy-Preserving Data Mining.
In: ACM SIGMOD Record, Volume 29, Issue 2 (June 2000), pp.
439e450.
30
Kamiran, F., Calders, T. (2009), Classification without discrimi-
nation, IEEE International Conference on Computer, Control &
Communication (IEEE-IC4); IEEE press.
31
Kamiran, F., Calders, T., Pecehnizkiy, M. (2009), Building
Classifiers with Independency Constraints. In: IEEE ICDM Work-
shop on Domain Driven Data Mining. IEEE press 1.
52 c o m p u t e r l a w & s e c u r i t y r e v i e w 2 7 ( 2 0 1 1 ) 4 5 e5 2
discrimination, de-individualisation and information asym-
metries. Apart from these risks we have observed that appli-
cation issues such as data dredging, issues with accuracy and
reliability, and mistaking correlation for causation, can also
negatively affect groups and individuals.
The traditional approach to mitigating these issues is
invoking the right to privacy. In our opinion, such a strategy is
no longer viable by itself. We have shown that traditional
approaches rooted in the idea of ex ante privacy protection are
by themselves insufficient for dealing with the negative
consequences of automated profiling and data mining, and may
even be counterproductive as they render detecting discrimi-
nation more difficult. In particular the notions of data mini-
misation and data exclusion are troublesome, as they may
reduce the accuracy of data mining and may deny us the data
necessary to detect discrimination in automated profiling.
Furthermore, privacy law will not adequately protect us from
the application issues mentioned in this article (viz., accuracy
and reliability, data dredging and mistaking correlation for
causation).
When it comes to additional and alternative strategies to
mitigate the risks of automated profiling we must look
towards mechanisms that increase the accountability (both
through ex ante screening of data mining applications for
possible risks and ex post checking of results) and trans-
parency of automated profiling. The legal basis for imple-
menting these mechanisms can be found in article 15 of the
Data Protection Directive, which in our opinion should be
featured far more prominently in the data mining and auto-
mated profiling debate.
To avoid the application issues mentioned we must raise the
level of awareness with policymakers about the limits of auto-
mated profiling and data mining. Apart from raising awareness,
mechanisms should be implemented that detect the improper
use of data mining, profiling, and statistics in policymaking.
Such a mechanism could be a “second opinion” or “sanity
check” by a panel of data mining experts and legal professionals.
The last approach is to build algorithms and decision-
making procedures in such a way that they minimise risk.
This means going beyond the idea of ‘privacy by design’ by
looking towards the underlying issues involved in a particular
data mining and automated profiling process. The creation of
independency constraints for instance, is a way to minimise
the risk of discrimination in automated profiling.
What becomes clear from these proposed solutions is that
they require a strong inter-disciplinary mindset and
approach. Within our own project we found that the legal
professionals and computer scientists use a totally different
vocabulary. The most important recommendation is therefore
to strengthen the ties between these different fields of science
(computer science, law, ethics) in order to create necessary
safeguards.
Bart W. Schermer (schermer@considerati.com) is an assistant
professor at eLaw@Leiden, Centre for Law in the Information
Society.