Research Evaluation, 19(5), December 2010, pages 306316

DOI: 10.3152/095820210X12809191250807; http://www.ingentaconnect.com/content/beech/rev

Evaluation of artistic research

Tomas Hellstrm

The gradual inclusion of art colleges and schools into the regular university system has prompted
previously craft-based subjects, such as music, visual arts, design and performance, to develop and
formalize a research component. While such artistic research is debated, many key points of this
debate are actualized when project proposals and results are evaluated by institutional funders. To
provide firmer ground for such evaluations, this article explicates some dimensions of value of artistic
research: its public and private, and its intrinsic and extrinsic values respectively, particularly in
relation to mainstream science. The article describes possible tensions between researcher and
evaluator standards, stemming from the special conditions of this field, using a Swedish case study.
Key dimensions of value and associated epistemic tensions are identified, involving levels of
acceptable serendipity in research processes and outcomes, acceptable forms of communication and the
cross-sectoral nature of artistic research.

C
HANGES IN THE tertiary education system
have, in many places, ushered previously in-
dependent fine, applied and performing arts
colleges/schools, as well as design and crafts colleges,
into the university system. With this inclusion the
arts subjects have come under the formal quality
systems of the universities and their funders, most
prominently under the accreditation requirements for
degree-awarding tertiary education institutions
(Lesage, 2009). As a result arts education is now
often expected to include research for higher de-
grees, teaching is expected to connect to research,
and teaching staff, as far as possible, should have
PhD degrees in their field of teaching. In Sweden,
the UK and elsewhere, the science funding system
has set aside money for artistic research; that is for
research into the artistic means and modes of ex-
pression, and in particular for research training.
However, the standards to be applied to such re-
search are subject to long-standing debate (e.g.
Strand, 1998; UK Council for Graduate Education,
2001). Such standards, as embodied in evaluation of
artistic research, are the focus of this article.
Tomas Hellstrm is at CIRCLE (Center for Innovation, Re-
search and Competence in the Learning Economy), Box 117,
Lund University, 221 00 Lund, Sweden; Email: tomas.
hellstrom@circle.lu.se; Tel: 0046 706566600.
Arguably the notion of an artistic research cate-
gory is mainly an outcome of the more or less volun-
tary inclusion of a research component in what were
previously craft-centered art and design schools. The
emergence of this new type of research, which is
supposed to combine artistic and traditional research
elements, has been circumscribed by debate and un-
certainty as to what standards to apply and to what
ends. Some of these discussions are captured in
higher education conferences such as the biennial
Research into Practice conference in Hertfordshire,
UK, and the Artistic Research: Evaluation and
Canon Formation meeting in Zurich 2010. This ar-
ticle aims to elucidate and explicate a number of es-
sential theoretical and methodological issues bearing
on evaluation of artistic research. A central concern
of the article is the challenge of assigning value to
artistic research, given its place between art and sci-
ence. This is essentially a problem of the relation-
ship between two spheres of inquiry which have
been institutionally separate for a long time. The
article illustrates the implications of these issues for
evaluation praxis through an analysis of the Swedish
Research Councils evaluation of its artistic research
program in 2008.
In what follows we will first look more closely
at the institutional and epistemic conditions for artis-
tic research and its evaluation, by (1) addressing
the question of what artistic research is, and (2) out-
lining some key tensions between art praxis and

306 0958-2029/10/05306-11 US$12.00 Beech Tree Publishing 2010 Research Evaluation December 2010

scientific research which are highly relevant to the
evaluation of artistic research. Second, a longer dis-
cussion will be undertaken on the issue of how to
assign value to artistic research. The arguments out-
lined here will focus on two lines of reasoning,
namely the public vs. private, and the intrinsic vs.
extrinsic qualities of artistic research. Third, an em-
pirical study of evaluation of artistic research will be
presented, which focuses on the tension between
evaluators assessments of project results compared
to that of researchers. Finally a discussion and some
suggestions for evaluation theory and praxis in this
emerging field will be presented.
What is artistic research?
The evaluation of artistic research must be connected
to its ambitions and its territory, or its problem
space. In this regard the relation of artistic research
to science and art respectively is important, since art
and science have developed distinctly different
modes of critique and evaluation, and artistic re-
search seems to fall somewhere in between.
Lagerbielke and Johansson (1993), in a relatively
early contribution to policy in this area, pointed out
that artistic research could encompass everything
from basic, curiosity-driven, to applied research, as
long as it fulfilled the criteria of contributing to artis-
tic understanding and development. It could therefore
also be located closer to the artistic process as such,
as for example in the setting up of an experimental
20th-century opera, or connect closer to technical de-
velopment for art, for example, in improving system-
atic, craft-based knowledge for some creative artistic
process. The authors thereby cover the territory be-
tween artistic creation and research on artistic pro-
cesses in general. This broad definition relates artistic
research to science, however placing it somewhere
between art praxis and academic research on art.
Mirroring the artscience continuum suggested by
Lagerbielke and Johansson, yet adding another layer
of complexity, Sheikh (2009) suggests three differ-
ent meanings of artistic research:
1. Research into artistic practices and materials;
2. Artistic practice as research (i.e. where the
production of art is seen as a form of research in-
quiry); and
3. Research that is artistic (i.e. an aesthetic approach
to scientific research and the objects of science).
What is suggested here is that art praxis may be
viewed as a form of research (under certain condi-
tions), that systematic inquiry may be purposefully
aimed at the improvement of art praxis, and that, in a
twist, scientific inquiry may itself be conducted in an
aesthetic mode.
When considering allowing such definitions it is
important to remind oneself that art is already, and
has been for some time, an established object of
Research Evaluation December 2010
Evaluation of artistic research
study in the humanities. Andersson (2009) mentions
a few possible approaches or focus points for artistic
research which enter into this territory to some de-
gree, yet also suggest specific differences. Artistic
research and traditional humanities arts research
may pose both ontological questions such as what
art is, and epistemological questions relating to how
meaning and knowledge are formed in art praxis.
However, Andersson suggests some normative focus
points typically not dealt with in the humanities, for
example, normative method issues such as what the
practice of art (and of artistic research) should be in
various fields. As we will see later, in actual artistic
research evaluation this turns out to be a highly
valued outcome.
Clearly many of the challenges in establishing cri-
teria for evaluation of artistic research rest on as-
sumptions about its relation to art and science
respectively. In this regard one may talk about an
artscience tension, which is what we will turn to
next. In Barry et als (2009) terminology, artistic
research, or artscience is concerned with what art
is or can be (p. 38), its aim being to investigate the
relation between art and the social as well as be-
tween art and science. However, these authors also
note how the relationship between art and science
has tended to be short-lived, as arts role in the mar-
riage, especially in public understanding of science
projects, often becomes one of superficial decoration
and subservience to science. If artistic research as-
sumes that a merger of science and art is to take
place, then the question is to what extent the epis-
temic agenda should be set by science, and what de-
gree of subservience if any to science should be
accepted by arts-oriented researchers (Barry et al,
2009).
While art and science can be conceived as over-
lapping, they are also, to a large extent, different
epistemic and institutional projects. Andersson
(2009) mentions a number of differences in the prac-
tice and aims of art and science, which artistic re-
search must resolve in its own praxis. These are in
brief:
The difference between how far the two fields are
expected to be able to clarify goals and aims be-
fore actually initiating an inquiry process;
Flexibility/constraints with regard to the form im-
posed by standard practices on the final product
and how it can be represented;
The extent to which art and science respectively
are expected to position themselves in a theoreti-
cal landscape (including referencing to a canon);
The level of methodological transparency ex-
pected; and
The propositional and textual nature of the aca-
demic report vis--vis the concrete, representa-
tional nature of the artistic product.
These dimensions of difference are mainly epistem-
ic, in that they represent ideals and values guiding
307
Evaluation of artistic research
Science expects a higher level of
formalization and documentation than
is normally expected from artistic
development, where instead tacit
understanding and indeterminacy of
outcomes are accepted and expected
evaluation of the products and creative processes in
the two fields. The common thread running through
these differences is that science expects a higher lev-
el of formalization and documentation than is nor-
mally expected from artistic development, where
instead tacit understanding and indeterminacy of
outcomes are accepted and expected.
Traditional forms of academic evaluation are
closely related to such epistemic differences. For
example, peer review in science, where one practi-
tioner evaluates the work of another, is difficult to
translate to art. For the purpose of evaluation the
notion of a peer in science is that of an academic
colleague in the same or a closely related field. In
art, on the other hand, the praxis is that institutional-
ized evaluation is conducted by a critic, who usually
represents a different professional identity and skill
set from that of the artist.
Furthermore, in science, two projects (albeit not
any two) can be compared on the basis of their rela-
tion to the research front, and their promise in terms
of solving certain scientific problems. Artistic re-
search, insofar as it is related to furthering the de-
velopment of art praxis, does not necessitate the idea
of a frontier in the same way. Instead artistic pro-
gress is often taken to imply an avant-garde which
will habitually undermine the forefront or paradigm
each time one such starts to emerge (Brger, 1984;
MacGregor, 2003). The evaluation of a project com-
pared to another in terms of an art frontier would
be at odds with the art side of the artistic research
equation.
One argument related to the above is that all artistic
processes are based in some type of research. In art
and science alike, problems are identified, options are
generated and eliminated and experiments are being
constructed and tried. For the sake of argument, let us
suppose that art is already a form of research; what
then is at stake when we attempt to merge art with sci-
ence? Epistemic goals and assumptions, including
methodology, will certainly differ on key dimensions,
and we will return to these later. One set of chal-
lenges, however, is likely to be found in the institu-
tional sphere (Jewesbury, 2009). As arts colleges
merge with universities, the administrative and fund-
ing system of the university is brought to bear on their
central academic practices. This typically, and
308
increasingly, includes an incentive to run large clas-
ses, and to bring in external projects which generate
overheads for the university. Local university policy
becomes tied to national policy and, in the UK for ex-
ample, this has meant that research activity, central-
ly defined in terms of outputs of peer-reviewed
publications, acquired research grants, etc., is pro-
moted locally through personnel policy.
A likely effect, which has been observed in other
settings, is a move away in focus from the academic
work process itself to certain types of outcome
measures, namely publication and citation counts
and external funding (Glser and Laudel, 2007).
Therefore, perhaps more now than ever, the shift
from artist to artistic researcher is likely to re-
quire a shift in skill set, new goals, and new forms of
professional role fulfillment/satisfaction. This shift
is likely to have consequences on the institutional
level quite in contrast with the domains traditional
values, for example, an increase in scale, formaliza-
tion and comparability (see Feller, 2002).
Some argue that by moving art into a formal re-
search context, such as it is increasingly represented
in todays university system, its praxis becomes in-
strumentalized and commodified in the same way
science is being instrumentalized today (cf. Jew-
esbury, 2009), namely by having its natural flow cut
up into projects (with fake beginnings, middles and
ends), by being reduced to countable output units such
as publications and citations (which in the first in-
stance may not represent authentic outputs of art), and
other deliverables (Ziman, 1994). The art process it-
self may end up becoming instrumentalized to
achieve and maintain these artificially imposed struc-
tures so that funding can be secured. It is therefore
imperative that evaluation models are not unreflec-
tively applied to artistic research, but that the fields
own value set is as accurately assessed as possible.
This is where we will turn in the next section.
Assigning value to artistic research
The notion of value is central to the field of cultural
and art production, just as it is to science (Connor,
1992), and it is the notion of value that connects the
field of art and artistic research to the practice and
theory of evaluation. Evaluation is often taken to be
the assessment of an observed value compared to
some standard (Stake, 1983). A central assumption
in artistic production, which is also increasingly ac-
cepted in the study and evaluation of science, is that
this standard should not be conceived as static, but
rather as negotiated and transactional, that is it
should be viewed as predicated on the interactions,
divergence and agreements among a group of actors
(Boix-Mansilla, 2006; Dickie, 1988). Context and
relevant stakeholder expectations matter for how
value is conceived in all spheres of inquiry where
social interaction matters to outcomes. Value is also
performatively significant. The way practitioners
Research Evaluation December 2010

formulate their own view of the value of their
art/science will be closely connected to the construc-
tion of a self-image put forward to achieve certain
social outcomes (cf. Butler, 1997). Value formula-
tions are therefore not only about the objective
outcome activities but also about the desire to have
these appreciated in a certain way.
Following this reasoning, we recognize quality as
a relative concept which, to a large extent, is predi-
cated on the goals of a group operating in a particular
environment. In hybrid forms of knowledge creation
such as artistic research, the aims of creative activity
may differ radically across research collectives.
There are several reasons for this. Artistic research is
emerging as a rather eclectic field in terms of methods
and theories (Sullivan, 2010). Apart from being inter-
or transdisciplinary in nature it is also trans-sector in
that it stretches from the sciences to the arts (cf. Klein,
2006, 2008). The trans-sectoral aspect goes beyond
pure epistemic issues and includes concerns for diffu-
sion and use of new knowledge by completely new
audiences, with different knowledge interests, as well
as the relationship between historically separated pro-
fessional identities. The question of stakeholder-
driven evaluation who are the beneficiaries? is
central to this problem. Even so over time goals are
variable because contexts change (Boix-Mansilla and
Gardner, 2003), as do intellectual sentiments, and
this is the case for both science and art.
Ideally, according to Boix-Mansilla and Gardner
(2003), variability of goals should drive variability
of evaluative indicators. However, it is also praxis in
science that methodological (including evaluative)
principles represent a more stable set of conditionals
than the research problems themselves. Theories and
problems change, but methodological principles re-
main (roughly) the same. This can be observed
throughout the history of science (Losee, 2004), and
is likely also to hold for artistic research as it
matures.
This represent a basic tension in formulating
evaluation frameworks for publically supported or
other practically relevant research: on the one hand,
evaluation criteria should represent useful cogni-
tive/methodological ways of organizing inquiry in a
collective and, on the other, it should be responsive
to and capture ideas of value among external stake-
holders, who are not part of this insider group. Of-
tentimes the two universes of value come into
conflict. Many of the value dimensions of artistic
research can be conceived as such, and here we will
attempt to understand some of these in quasi-
economic terms, as relating, first, to a public/private
tension and, second, to an intrinsic/extrinsic tension
in artistic research production and consumption.
Public/private dimensions
Artistic research, just as academic research in gen-
eral, has both private and public good dimensions.
Practitioners will value the activity of production as
Research Evaluation December 2010
Evaluation of artistic research
it is being performed, and accumulate skills which
attach a possible future market value to them as in-
dividuals. At the same time outputs can be commod-
ified and consumed by a larger public, and
institutional forms in both art and science have been
constructed to promote the public character of these
outputs. In addition, art and science also display club
good qualities, namely that part of their value is only
appropriated by a limited group of practitioners due
to the esoteric knowledge and skill needed for its
consumption. The evaluation of publically funded
artistic research, as well as public science in general,
should therefore involve an assessment of the
amount of restriction this club good quality puts on
the wider dissemination of knowledge developed.
The club good quality of artistic production and
appropriation (conceived here as markets for artistic
research) makes evaluation criteria problematic.
First, formalization of criteria would entail the ex-
plication of value, as well as an explanation for these
values. Formal evaluation would in effect have the
latent function of broadening the community of
practitioners able to identify these goods as valuable
(or not). Such a broadening of the market for artistic
research may be appealing to funders; however, if
one assumes that the outputs of artistic research, just
as art, take the form of positional goods, that is
goods which receive their value on the basis of con-
ferring status on the holder (cf. Hirsch, 1976), then
the strategy of explicating evaluation criteria may
well turn out to be self-defeating.
Following Bourdieus (1984) notion of the status
function in cultural production captured by his con-
cept of distinction, as well as positional goods argu-
ments from economics, suggests that the value of
artistic research may be more dependent on its club
good qualities. This is obviously not the case for
science, where principles derived from research are
continuously applied in a variety of fields and com-
munities outside their original discovery. The idea
that, even in the long run, some research may be
valuable only within a practitioner group seems
counterintuitive from an evaluation perspective. Yet
it is a characteristic of much scientific research that
when a contributory activity becomes part of and
sustains a tradition or school, the usefulness of that
work is defined within that tradition or school and
consequently receives its meaning in a universe of
discourse which is neither fully individual nor en-
tirely public (Weinberg, 1967) . Individuals develop
skill through artistic research which is valued by a
limited collective, and that same collective derives
social benefits by controlling and legislating the skill
set that matters for the discipline.
However, assuming that artistic research confers
public as well as private and club benefits, we need to
establish the nature of the public values of relevance.
At least two dimensions of public value seem appli-
cable in the case of artistic research: those values aris-
ing from the work itself, which are contained within
the work, and which may be appreciated by a public;
309
Evaluation of artistic research
Whether value is conceived as a
quality inhering primarily in the
artifact, or in certain outcomes or
epiphenomenal effects brought about
by the activity, is as much an issue for
artistic research as for science in
general
and those values arising from an institutional setting
associated with art and artistic research, for example,
from a museum or art school which in turn may have
various other functions than displaying/producing
specific pieces of art and artistic research. A third in-
termediate form of value, located somewhere in be-
tween these two, is embodied in the public function of
intellectual commentary by artistic research practi-
tioners. In artistic research terms, these distinctions
would pertain to the process and product of the re-
search and to its extended institutional context re-
spectively (Throsby, 2001).
This discussion may be read as related to the issue
of whether artistic research should support elitist or
egalitarian ideals: should outcomes (and indeed pro-
cesses leading up to outcomes) be valued for their
ability to contribute to a core-set of researchers and
connoisseurs (cf. Collins and Evans, 2007), or
should they be evaluated on the basis of, for exam-
ple, creating shared values and engagement among
groups usually peripheral to the artistic world?
Intrinsic/extrinsic dimensions
As we will see exemplified later, the issue of where
the value of artistic products and processes is located
whether value is conceived as a quality inhering
primarily in the artifact/text/play, etc., or in certain
outcomes or epiphenomenal effects brought about
by the activity is as much an issue for artistic re-
search as for science in general (cf. the question of
science for its own sake and science for profit).
The intrinsic value of art is easy to understand since
it has, historically, tended to dominate discourse on
the subject. Ruskins (1859/2004) theory of art, for
example, located the value of an art object in the
pleasure or richness of experience that went into cre-
ating it. Similarly, labor theory of value focuses on
the work put into producing the output (Moseley,
2005). Related to this concept is the notion of intrin-
sic or absolute value, that is a worth which is not
dictated by a market or the use value of a specific
group or actor. The humanist tradition emphasizes
the intrinsic aesthetic and broader cultural value of
artistic work, and that the intrinsic properties of a
work will, if deserved, eventually gain acceptance in
310
a tradition or culture regardless of how it is valued at
the time of its creation (Dickie, 1988).
However, one does not have to assume any specif-
ic intrinsic values either in art or science to make
intrinsic evaluative assessments of such activities.
One evaluative strategy agnostic to specific values
involves the externalization of value by having it
designated by a reference group of some kind. Peer
review and citation counts represent the premier ex-
ample for science. Another such strategy is the prax-
is from program evaluation of assessing the
organizational capacities tied to or enabled by a re-
search activity. In artistic research the application of
unbiased peer review and bibliometric proxies is
naturally difficult to achieve. And while program
evaluation tends to combine such tools with various
forms of organizational assessment, it is important to
recognize that what is perceived as a productive or-
ganizational arrangement depends very much on
what part of the academic landscape the evaluator is
coming from (cf. Whitley, 2003). The level of or-
ganization, its size, the amount of hierarchy, division
of labor or other mechanisms of formalization, will
be assessed differently depending on research ideol-
ogy and disciplinary affiliation.
Such externalizing approaches may have addi-
tional confounding effects on artistic research. At
the center of the problem lies the issue of the relative
separability between the cognitive process of
knowledge production and its proxy indicators, such
as for example bibliometric outcomes and organiza-
tional and financial processes. If, as is now common,
such external processes are taken as proxies for epis-
temic quality in a project, its legitimacy rests on an
assumption that others, peers, have deemed the pro-
ject to be of high academic quality and awarded re-
sources for that reason. This logic is recognized in
the credibility cycle forwarded by Latour and
Woolgar (1979). The inference of quality from pre-
vious judgments and allocations by others assumes a
consensus about what the goals and outcomes of a
research process ought to be. While such consensus
is norm rather than exception in paradigmatic sci-
ence, for aesthetic objects and processes it may be,
in principle, impossible. This is part of the bigger
problem of the essentially contestable nature of qual-
ity judgments in art, and its implications for the in-
stitutional forms of research evaluation. As pointed
out by, among others, Jewesbury (2009), this may
lead to a situation where good process receives
funding and ends up being promoted as good art.
Because of attempts to adapt art production to insti-
tutional forms of science, artistic research may fall
between two chairs and end up being perceived as
either poor research or poor art.
In what follows a case study will be presented, of
an evaluation of artistic research carried out by the
Swedish Research Council in 2005. The case is in-
tended to illustrate how some of the tensions in the
aims and consequences of artistic evaluation are
realized in evaluation practice.
Research Evaluation December 2010
 Evaluation of artistic research

The Swedish Research Councils support to
artistic research, 20012005
An illustrative case of artistic research evaluation
will be presented here, and analyzed in light of the
previous discussion. The case will focus on critical
quality/value dimensions for artistic research as
identified and elaborated within the framework of an
evaluation initiated by the Swedish Research Coun-
cil, of their support for artistic research and devel-
opment between 2001 and 2005 (Swedish Research
Council, 2007). A total of seven program networks
(research networks) and seven projects received
funding during these years. The funding for these
networks and projects amounted to about 680,000
from 2001 to 2007 (some projects ran up until
2007). Table 1 provides a brief summary of the main
academic orientation and funding of the projects
dealt with in this study. These five networks/projects
can be said to be representative of the activities
funded within the program as a whole.
An expert committee was appointed by the coun-
cil, which consisted of professors from the Nordic
countries in areas overlapping with those that re-
ceived support, namely architecture/design, visual
arts, musicology and drama. All of the committee
members had substantial evaluation experience. The
process of evaluation was carried out in the tradi-
tional program evaluation style of self-evaluation by
the recipients, followed by output assessment by the
committee and site visits with interviews. Instead of
going further into the details of this process, the fo-
cus here will be on the criteria applied by evaluators
as well as by those evaluated (reported through
self-evaluation), and the implications of these crite-
ria in terms of the conditions for conducting artistic
research funded by the council. This will serve to
focus the previous discussion on evaluation of artis-
tic research into a few conceptually and practically
relevant dimensions.
Funding and evaluation criteria
The criteria for funding were elaborated by a council
program committee in 2001, and used for the ex ante
evaluation and selection of proposals for networks
and projects. The same criteria, complemented with
a few additional items, formed the basis for the final
evaluation in 2007. The evaluation framework natu-
rally groups into two interrelated sets of criteria:
Academic environment and networking
The presence of continuous and integrated seminar
activities and education, at least on the bachelor/
masters level;
Involvement in an international network (or plans
for developing such a network) with other institu-
tions working in the area;
An organizationally well-defined cooperative en-
gagement with at least one more university/
college; alternatively inter-faculty cooperation,
where at least one of the partners is research-
active and is conducting PhD training.
The issue of inter-faculty cooperation and PhD train-
ing relates to a second set of criteria, more closely
associated to research production.

Table 1. Research orientation and funding for of
networks/projects analyzed in this study
Research focus
(thousand
Network A Explores the
artistic/creative
processes in the
borderland between
written music and
improvisation
Network B Investigates the forms of
formal and tacit
expression employed in
guiding dramatic practice
Project A Studies the interpretative
processes of the Western
musical tradition
Project B Combines artistic and
scientific insights to
understand visual
perception
Project C Attempts to develop a
form of operatic (dance)
representation of
scientific knowledge
Research production
At least part of the program should (aim to)
contribute to documenting theoretical reflection
Funding Duration on artistic production and development; and in
addition
euros)
Contribute to a discussion on the relation between
580 20012004 art and science.
Connected to these two criteria, the evaluation direc-
tives also suggest that networks/projects should be
assessed on the basis of their contribution to the de-
1,114 20012005 velopment of methods for artistic research, as well
as on how they have furthered the cooperation be-
tween artists and researchers.
429 20032005
The first set of criteria is fairly straightforward and
adheres to received notions of value in terms of how
research environments should be organized, and how
445 20032005 they ought to connect to the wider research communi-
ty. However, already here, with regard to inter-faculty
cooperation, and specifically in the second set of cri-
teria, there is a distinct move from general, acknowl-
520 20032006 edged quality factors to very specific epistemic
directives. For example, ideas about contributions to
the relationship between art and science, and con-
cretely towards that between artists and researchers,

Research Evaluation December 2010 311

Evaluation of artistic research
certainly make considerable assumptions in terms of
epistemic viability/desirability, as well as profession-
al/institutional capacity. The expectation that there
will be method development for art and artistic re-
search similarly begs a number of critical questions
regarding the relationship between art and possible
forms of research. In the next section we will see
how these issues are exemplified in the evaluation
committees statements.
The account below is organized as follows. For a
selection of networks and projects, analytical sum-
maries of the criteria applied by the evaluation team
are presented, each followed by a summary of the
self-evaluation provided by the respective network/
project leaders. The aim here is to identify a number
of normative threads running through the evalua-
tions, and possible conflicts and tensions in the eval-
uation criteria and outcomes intended by the
projects. The account is followed by a summary in
the form of a table.
Network A
Evaluators Results from the network are not ade-
quately fed back into the educational process. Learn-
ing during the period seems only to have been
relevant to the participants themselves, and in terms
of their personal development. A workshop series
that was organized was conducted without a clear
methodology, and there has been no attempt at sys-
tematic accumulation of insights from the improvi-
sational activities arranged within the network. The
final report connects poorly to the original purpose
behind the network. There has been a general lack of
leadership.
Researchers The network has moved closer to-
wards an understanding of art (and improvisation)
through art praxis and observation. Most important
has been the cooperation achieved in the project
and the self-knowledge achieved by the practition-
ers. With regard to the latter, the increased artistic
awareness of the participating musicians and the
contribution to their professional careers in this
regard have been very important. Another central
outcome has been to complicate and question the
participants understanding of the relationship be-
tween performance and written culture, and be-
tween improvised and written music. An important
outcome has also been to validate older pieces
through performance.
Network B
Evaluators Critical reflection is lacking in the contri-
butions, and in addition there needs to be a clearer
relationship to other work in the field. There is a need
to problematize work methodologies, and to docu-
ment work methods so that knowledge can be accu-
mulated and transferred to others. Theoretical insights
are not translated into education. Contributions are
312
too abstract to connect to artistic praxis. There is an
individual bias and lack of generality and coordina-
tion between sub-projects. Need for more meta-level
discourse, and general need for written text.
Researchers The network has developed new forms
of cooperation for artistic research and in fact a new
research field. It has identified a need to transfer
knowledge between actors on the level of research,
and not just within basic education. The resulting
cooperation has generated new concepts for re-
search, which have affected disciplinary identity (for
the art school), and will stimulate future reorienta-
tion/development. This involves the establishment
and awareness of a shared professional language.
The network has facilitated long-term personal de-
velopment for its participants and created a new in-
tellectual milieu.
Project A
Evaluators One positive aspect of this project is that
it is divided up into sub-projects, well documented
through reports and papers, and has had its results
presented at several international conferences.
Method development for learning and interpretative
processes is substantial as is the theoretical anchor-
ing of the project. The prospect for translating re-
search into education is deemed to be good.
Researchers The project has offered professional de-
velopment in terms of understanding the conditions
for research in the field, for example, the relation be-
tween originality and skill. Flexible method sets were
developed which were adapted to the individual needs
of the participants. With regard to learning, the most
important contribution was the establishment of a
meeting place/forum for exchange between artistic
researchers, teachers and educational researchers.
Another important outcome was learning with re-
gard to leading a project of this type.
Project B
Evaluators This project is well documented. It has
also had several concrete effects on educational of-
ferings, among others it has resulted in an obligatory
course in one of the arts colleges. There has been
constructive cooperation between the disciplines, as
well as across faculties. Missing is a deepened ac-
count of the interplay between theory and praxis, as
well as thorough methodological accounts. There is
a need for more meta-reflection on the implications
of the interdisciplinary work.
Researchers The most important result is the realiza-
tion that dialogue is possible between art and the
sciences and that many problems in art are also
addressed in, for example, the sciences of neuro-
physiology and psychology. The experience of a
space is the foundation for understanding its theory.
Research Evaluation December 2010

The evaluations and self-evaluations
may be grouped into two general
themes concerning the values
processes/outcomes of the respective
networks and projects, namely those
relating to knowledge and learning
and those relating to knowledge
transfer and communication
The meeting of art and science has stimulated new
artistic experiments. Many of the results have been
applied in education in art as well as psychology.
Project C
Evaluators The project has delivered a publically
appreciated art performance, with didactic elements,
Table 2. Key phrases from evaluators and researchers ordered according to outcome categories
Knowledge development and learning
Evaluators
Feedback of results into education is lacking (NA)
Learning tied to individual persons (NA)
No systematic accumulation of insights (NA)
Connects poorly to original purpose (NA)
Lack of project leadership(NA)
Critical reflection lacking (NB)
No clear relation to other works in the field (NB)
Individual bias, lack of generality (NB)
Need for more written text (NB)
+ Theoretical anchoring (PA)
No thorough account of the theorypraxis link (PB)
Methodological documentation missing (PB)
Meta-reflection on the consequences of interdisciplinary work
(PB)
Theoretical and meta-level contribution unclear (PC)
Knowledge transfer and communication
Evaluators
Need for documentation (NB)
Need for transfer of work methods (NB)
Poor transfer of theory into education (NB)
+ Documented through paper writing (PA)
+ Conference presentations (PA)
+ Work methods documented (PA)
+ Good prospects for transfer into education (PA)
+ Well documented (PB)
+ Spill-over into education (PB)
+ Cooperation across disciplines and faculties (PB)
+ Connection between art and science (PC)
+ Successful outreach to a wider audience (PC)
Unclear about level of documentation (PC)
Notes: + indicates positive and indicates negative
evaluation
Capital letters within parentheses indicate which network/project
Research Evaluation December 2010
Evaluation of artistic research
relating to the connection between science and art. It
has represented a personal learning process for the
project leader; however, it is unclear if the results
reach an adequate level of documentation, or if the
theoretical contribution will develop the meta-level
understanding enough. The main contribution of the
project is that of outreach and popular communica-
tion of research to a (young) audience.
Researchers Fruitful cooperation has been estab-
lished across disciplines. The project has been great-
ly appreciated by a wider social community than that
of academe. In addition it has resulted in personal
learning with regard to project planning and the ap-
plication of new technology. It has contributed un-
derstanding of similarities and differences between
artistic and scientific work processes, and of possi-
ble forms of communication between these fields.
Media reception of the project has also been very
positive.
The evaluations and self-evaluations may be
grouped into two general themes concerning the val-
ues processes/outcomes of the respective networks
Researchers
+ Understanding of art praxis through observation (NA)
+ Increased artistic self-knowledge/awareness among participants
(NA)
+ More complex understanding of the relation between written and
performed culture (NA)
+ Validation of older pieces (NA)
+ Stimulated new disciplinary insights and research concepts (NB)
+ Shared professional language and academic awareness (NB)
+ Long-term personal development (NB)
+ Insights about the connection between art and science in terms of
problems and methods (PB)
+ New artistic experiments stimulated by the meeting with science
(PB)
+ Personal learning with regard to project planning and technology
(PC)
+ Understanding of relationship between disciplines (PC)
Researchers
+ Developed new forms of artscience cooperation (NB)
+ New intellectual milieu (NB)
+ Establishing of a meeting place for this type of research (PA)
+ Educational applications (PB)
+ Established fruitful cooperation between disciplines PC)
+ Interdisciplinary communication (PC)
+ Positive media reception (PC)
313
Evaluation of artistic research
and projects, namely those relating to knowledge
and learning and those relating to knowledge trans-
fer and communication. In order to provide an ana-
lytical summary of the results, Table 2 presents key
phrases from evaluators and researchers respective-
ly, ordered according to these two categories. These
categories will later be used in the discussion to
reflect on the implications for artistic research
evaluation.
Discussion and conclusions
In the above, several tensions related to the evalua-
tion of artistic research were identified, namely
those between science and art, private and public
value dimensions, and qualities of an intrinsic and
extrinsic nature. In many ways the divergence of
opinion between evaluators and researchers exempli-
fies these tensions and assigns them practical mean-
ing. Following the division of value suggested in
Table 2, we see how, with regard to knowledge crea-
tion and learning, evaluators emphasized clear
methodologies and method development, versus the
researchers emphasis on individual method and
skill development. This may be traced back to An-
derssons (2009) suggestion that a lower level of
methodological transparency is typically expected in
artistic inquiry compared to that of mainstream
research.
It may be argued that, given the higher level of
acceptable serendipity present in artistic research,
the demand for such methodological explication at
the expense of intuitively guided skill application
and development may not be acceptable. In this re-
gard cultural perhaps more than scientific production
(and consumption) represents a cumulative learning
process, and should be evaluated as such. The result
of such cultural learning may be significant un-
planned (non-calculated) externalities accruing to
the producer of the work or the funder (Throsby,
2001). This is why goal-oriented outcome evaluation
is likely to be non-exhaustive for artistic production,
and perhaps putting the emphasis on personal and
group learning as actual project outcomes would
prove a more useful approach, at least in this early
stage of the fields development.
It is also a matter of acknowledging the natural
methodological and theoretical variety of emerging
fields. Radical methodological variation is in fact
present in many disciplinary fields, for example,
biology, psychology and physics, to the point where
a large part of a traditional discipline may already be
interdisciplinary in its practices if not in its basic
training (Dupr, 1993). In addition, long-running
programs of inquiry often change so much over time
as to preclude any notion of a stable theoretical or
methodological panacea (Vickers, 1997). This being
said, there is nothing in such arguments to preclude
critical meta-reflection on process and results, such
as is requested by the evaluators. Via Kuhn (1962)
314
we have learnt that engaging in critical meta-
reflection on the basis for inquiry is necessary in the
formative phases of the emergence of a field. This
evaluator requirement will, at least in so far that one
assumes that no avenues for inquiry are prematurely
closed off, certainly satisfy the researchers call for
an increased complexity of understanding, their con-
cern with a disciplinary identity, and need to im-
prove personal understanding.
The key tension regarding knowledge creation
and learning may not be about which specific
knowledge should be sought, but rather how it
should be socially validated. The artistic researchers
clearly favor an individualistic or personal valida-
tion of their work, while the evaluators ask for at-
tendance to documentation, synthetization and broad
(international) scholarly communication of results,
as well as connection to other works in the field. The
notion that art, and by extension artistic research,
represents a distinctive club good is echoed here.
The academic journey represented by many of these
projects, much to the evaluators dismay, is still one
of concern mainly to the researcher him/herself or to
the immediate community/group to which they be-
long.
This brings us to the second category of value
suggested in Table 2, namely that of knowledge
transfer and communication. It is clear from the
summary that evaluators expect more tangible forms
of transfer of documented results and educational
integration. This may be rightfully expected from a
subject/program that has achieved a certain level of
maturity. However, the artistic researchers are rather
emphasizing the creation of an environment for
building a field in the first place; that is, they value
the enabling of communication and knowledge
transfer internal to the projects and networks.
The creation of cross-disciplinary connections
promoted by the evaluators were also valued by the
researchers. However, where evaluators stressed
international linkages, outreach and communication
of research, researchers focused on issues in rela-
tion to the creation of platforms for future research
relationships, in particular those that suggested the
possibility of problem transfer between art and
science. Again we see how, rather than being in
epistemic opposition, evaluators and researchers
seem to disagree more on the stage of maturity of
the field at the moment, and thereby also on how it
should be evaluated. Knowledge integration is fa-
vored by both; however, researchers seems to per-
ceive a longer-term challenge in the merging of
relevant fields into a functioning whole, as well as
in forging links between the emerging field and
other related ones, than do evaluators. Indeed this
is a key challenge for any interdisciplinary field
(Klein, 2008) which needs to be recognized by
evaluators.
Spaapen et al (2007) have pointed out how context
often matters more in interdisciplinary research than
does disciplinary affiliation. In the case of artistic
Research Evaluation December 2010

research, literal notions of interdisciplinarity (read
disciplinary interaction) is therefore likely to be un-
justified due to an extended stakeholder set, which
may consist of, for example, disciplinary humanist
researchers who research art, end-consumers of art,
art practitioners, art students, the artistic researchers
themselves, curators, as well as funders/councils.
Still even in research explicitly attempting to bridge
epistemic boundaries, and where separation from
disciplinary mores is explicitly promoted by re-
search councils, evaluators may end up asking for
communication and knowledge transfer in more tra-
ditional terms of disciplinary interaction, where
they should really look for the extended stakeholder
set, or what Funtowicz and Ravetz (1993) referred to
as an extended peer community.
In conclusion, some observations with regard to
evaluation may be drawn: First, there is a need for
an extended stakeholder set in artistic research eval-
uation. With regard to peer review, expertise may
not be readily available, but rather will have to be
approximated or created. Experts from related fields,
including practitioners and other users, will have to
negotiate and calibrate their understanding of quali-
ty, between each other, as well as between them-
selves as a group and the evaluated party. In
evaluation terms this may take the form of repeated
discussions between reviewers and project parties to
ensure learning in the evaluation process (cf. Laudel,
2006).
Second, as an emerging field there is a need for
new indicators of intrinsic quality (associated with
the first category in Table 2). Traditional intrinsic
indicators would amount to, for example, fit between
framework and data, aesthetic qualities of theories,
explanatory reach; that is, epistemic indicators. But
artistic research, due to its cross-sector quality, and
its often rather practical research goals, suggests that
such intrinsic values are closely connected to the
transformative effect of new knowledge on an ex-
tended stakeholder set. In this sense the stakeholder
set and the intrinsic quality indicators are closely
tied together. The following is a condensed list of
insight relevant to artistic research evaluation based
on the above discussion.
Higher levels of acceptable serendipity in artistic
research implies less reliance on methodological
explication and planned outcomes.
Communication should not be reified to mean
publication and conference attendance, but should
also recognize the need to form institutional plat-
forms for inquiry where such did not previously
exist.
It is important to assess correctly the level of
maturity of a field since this dictates what kind of
outputs can be expected.
Due to the cross-sectoral character of artistic re-
search, extra attention must be given to identify-
ing the extended stakeholder set of relevance for
validation and use of knowledge generated.
Research Evaluation December 2010
Evaluation of artistic research
References
Andersson, E 2009. Fine science and social arts on common
grounds and necessary boundaries of two ways to produce
meaning. Art and Research, 2(2), available at: <http://www.
artandresearch.org.uk/v2n2/andersson.html>, last accessed 2
June 2010.
Barry, A, G Born and G Weszkalnys 2008. Logics of interdiscipli-
narity. Economy and Society, 37(1), 2049.
Boix-Mansilla, V 2006. Assessing expert interdisciplinary work at
the frontier: an empirical exploration. Research Evaluation,
15(1), April, 1729.
Boix-Mansilla, V and H Gardner 2006. Assessing Interdisciplinary
Work at the Frontier. An Empirical Exploration of Symptoms of
Quality. Interdisciplines: Harvard Interdisciplinary Studies Pro-
ject. Available at: <http://www.interdisciplines.org/
interdisciplinarity/papers/6/2>, last accessed 2 June 2010.
Bourdieu, P 1984. Distinction: a Social Critique of the Judgement
of Taste. London: Routledge.
Brger, P 1984. Theory of the Avant-Garde. Minneapolis, MN:
University of Minnesota Press.
Butler, J 1997. Excitable Speech: a Politics of the Performative.
London: Routledge.
Collins, H M and R Evans 2007. Rethinking Expertise. Chicago,
IL: University of Chicago Press.
Connor, S 1992. Theory and Cultural Value. Oxford: Blackwell.
Dickie, G 1988. Evaluating Art. Philadelphia: Temple University
Press.
Dupr, J 1993. The Disorder of Things. Harvard, MA: Harvard
University Press.
Feller, I 2002. Performance measurement redux. American Jour-
nal of Evaluation, 23(4), 435452.
Funtowicz, S and J Ravetz 1993. Science for the post-normal
age. Futures, 25(7), 739755.
Glser, J and G Laudel 2007. Evaluation without evaluators. In
The Changing Governance of the Sciences, eds R Whitley and
J Glser, pp. 127151. Dordrecht: Springer.
Hirsch, F 1976. Social Limits to Growth. Harvard, MA: Harvard
University Press.
Jewesbury, D 2009. Some problems with research in UK fine
arts institutions. Art and Research, 2(2), available at:
<http://www.artandresearch.org.uk/v2n2/jewesbury.html>, last
accessed 2 June 2010.
Klein, J T 2006. Afterword: the emergent literature on interdisci-
plinary and transdisciplinary research. Research Evaluation,
15(1), April, 7580.
Klein, J T 2008. Evaluation of interdisciplinary and transdiscipli-
nary research: a literature review. American Journal of Preven-
tive Medicine, 35(2), 116123.
Kuhn, T 1962. The Structure of Scientific Revolutions. Chicago,
IL: Chicago University Press.
Lagerbielke, G and N Johansson 1993. Konstnrligt
utvecklingsarbete [Artistic development work]. Ds 1993:3.
Stockholm: Ministry of Education and Research.
Latour, B and S Woolgar 1979. Laboratory Life: the Social Con-
struction of Scientific Facts. Los Angeles: Sage Publications.
Laudel, G 2006. Conclave in the Tower of Babel: how peers re-
view interdisciplinary research proposals. Research Evalua-
tion, 15(1), 5768.
Lesage, D 2009. Whos afraid of artistic research? On measuring
artistic research output. Art and Research, 2(2), available at:
<http://www.artandresearch.org.uk/v2n2/lesage.html>, last ac-
cessed 2 June 2010.
Losee, J 2004. Theories of Scientific Progress. London:
Routledge.
MacGregor, C 2003. Bodies on the boundaries: subjectification
and objectification in contemporary performance. In Cultural
Work: Understanding the Cultural Industries, ed. A Beck, pp.
5672. London: Routledge.
Moseley, F ed. 2005. Marxs Theory of Money: Modern Apprais-
als. New York: Palgrave/MacMillan.
Ruskin, J 1859/2004. On Art and Life. London: Penguin Books.
Sheikh, S 2009. Objects of study or commodification of
knowledge? Remarks on artistic research, Art and Research,
2(2), available at: <http://www.artandresearch.org.uk/v2n2/
sheikh.html>, last accessed 2 June 2010.
Spaapen J, H Dijstelbloem and F Wamelink 2007. Evaluating
Research in Context: a Method for Comprehensive Assess-
ment, 2nd edn. The Hague: COS.
Stake, R E 1983. Program evaluation, particularly responsive
315
Evaluation of artistic research
evaluation. In Evaluation Models: Viewpoints on Educational
and Human Services Evaluation, eds G F Madaus, M Scriven
and D L Stufflebeam, pp. 287310. Norwell, MA: Kluwer.
Strand, D 1998. Research in the Creative Arts. Canberra: De-
partment of Employment, Education, Training and Youth
Affairs.
Sullivan, G 2010. Art Practice as Research: Inquiry in Visual Art,
2nd edn. London: Sage.
Swedish Research Council 2007. Kontext Kvalitet Kontinuitet:
Utvrdering av Vetenskapsrdets Anslag till Konstnrlig For-
skning och Utveckling 2001-2005 [Context Quality Conti-
nuity: Evaluation of the Swedish Research Councils Funding
of Artistic Research and Development 2001-2005]. Stockholm:
Swedish Research Councils Report Series.
316
Throsby, D 2001. Economics and Culture. Cambridge: Cambridge
University Press.
UK Council for Graduate Education 2001. Research Training in
the Creative and Performing Arts and Design. Policy paper.
Vickers, J 1997. [U]nframed in open, unmapped fields: teaching
and the practice of interdisciplinarity. Arachn: an Interdiscipli-
nary Journal of the Humanities, 4, 1142.
Weinberg, A 1967. Reflections on Big Science. Cambridge, MA:
MIT Press.
Whitley, R 2003. Competition and pluralism in the public sciences:
the impact of institutional frameworks on the organization of
academic science. Research Policy, 32, 10151029.
Ziman, J 1994. Prometheus Bound. Cambridge: Cambridge
University Press.
Research Evaluation December 2010
Copyright of Research Evaluation is the property of Beech Tree Publishing and its content may not be copied or
emailed to multiple sites or posted to a listserv without the copyright holder's express written permission.
However, users may print, download, or email articles for individual use.