Funded by NSF IIS-1161997 II and NSF IIS 1510741

UCR Time Series Classification Archive

Please reference as:

Yanping Chen, Eamonn Keogh, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah
Mueen and Gustavo Batista (2015). The UCR Time Series Classification Archive. URL
www.cs.ucr.edu/~eamonn/time_series_data/
 Welcome!
Dear Colleague
If you are reading this, you are interested in using the UCR Time Series Classification Archive. This archive is a
superset of, and completely replaces [5]. Both [5], and this current Archive were born out of my frustration
with papers reporting error rates on a single dataset, and claiming (or implicitly suggesting) that the
results would generalize [6]. However, while I think the availability of previous versions of the UCR archive
has mitigated this problem to a great extent, it may have opened up other problems.
1) Several researchers have published papers on showing “we win some, we lose some” on the UCR Archive.
However, there are many trivial ways to get “win some, lose some” type results on these datasets (For
example, just smoothing the data, or generalizing from 1NN to KNN etc.). Using the Archive can therefore
apparently add credence to poor ideas (very sophisticated tests are required to show small but true
improvement effects [3]). In addition Gustavo Batista has pointed out that “win some, lose some” is
worthless unless you know in advance which ones you will win on! [4].
2) It could be argued that the goal of researchers should be to solve real world problems, and that improving
accuracy on the UCR Archive is at best a poor proxy for such real world problems. Bing Hu has written a
beautiful explanation as to why this is the case [2].
In spite of the above, the community generally finds the archive to be a very useful tool, and to date, more
than 1,200 people have downloaded the UCR archive, and it has been referenced several hundred times.
We are therefore are delighted to share this resource with you. The password you need available in this
document, read on to find it.

Best of luck with your research.

Eamonn Keogh
 Data Format
Each of the datasets comes in two parts, a TRAIN partition and a TEST partition.
For example, for the synthetic control dataset we have two files, synthetic_control_TEST and
synthetic_control_TRAIN
The two files will be in the same format, but are generally of different sizes.
The files are in the standard ASCII format that can be read directly by most tools/languages.
For example, to read the two synthetic control dataset s into Matlab, we can type…
>> TRAIN = load('synthetic_control_TRAIN');
>> TEST = load('synthetic_control_TEST' );
…at the command line.

There is one data instance per row. The first value in the row is the class label (an integer between 1 and
the number of classes). The rest of the row are the data values, and individual time series.

This
instance is
in class 1

This
instance is
in class 2
 function UCR_time_series_test %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (C) Eamonn Keogh %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
TRAIN = load('synthetic_control_TRAIN'); % Only these two lines need to be changed to test a different dataset. %

Sanity Check TEST = load('synthetic_control_TEST' ); % Only these two lines need to be changed to test a different dataset. %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

TRAIN_class_labels = TRAIN(:,1); % Pull out the class labels.

TRAIN(:,1) = []; % Remove class labels from training set.
In order to make sure TEST_class_labels = TEST(:,1); % Pull out the class labels.
TEST(:,1) = []; % Remove class labels from testing set.
that you understand correct = 0; % Initialize the number we got correct
for i = 1 : length(TEST_class_labels) % Loop over every instance in the test set
the data format, you classify_this_object = TEST(i,:);
this_objects_actual_class = TEST_class_labels(i);
should run this predicted_class = Classification_Algorithm(TRAIN,TRAIN_class_labels, classify_this_object);
if predicted_class == this_objects_actual_class
simple piece of correct = correct + 1;
end;
matlab code (you can disp([int2str(i), ' out of ', int2str(length(TEST_class_labels)), ' done']) % Report progress
end;
cut and paste it, it is %%%%%%%%%%%%%%%%% Create Report %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
disp(['The dataset you tested has ', int2str(length(unique(TRAIN_class_labels))), ' classes'])
standard Matlab) disp(['The training set is of size ', int2str(size(TRAIN,1)),', and the test set is of size ',int2str(size(TEST,1)),'.'])
disp(['The time series are of length ', int2str(size(TRAIN,2))])
disp(['The error rate was ',num2str((length(TEST_class_labels)-correct )/length(TEST_class_labels))])
%%%%%%%%%%%%%%%%% End Report %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Note that this is slow % Here is a sample classification algorithm, it is the simple (yet very competitive) one -nearest
% neighbor using the Euclidean distance.
“teaching” code. To % If you are advocating a new distance measure you just need to change the line marked "Euclidean distance"
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
consider all the function predicted_class = Classification_Algorithm(TRAIN,TRAIN_class_labels,unknown_object)
best_so_far = inf;
datasets in the for i = 1 : length(TRAIN_class_labels)
compare_to_this_object = TRAIN(i,:);
archive, you will distance = sqrt(sum((compare_to_this_object - unknown_object).^2)); % Euclidean distance
if distance < best_so_far
probably want to do predicted_class = TRAIN_class_labels(i);
best_so_far = distance;
something more end
end;
sophisticated
(indexing, lower
>> UCR_time_series_test
bounding etc) 1 out of 300 done
2 out of 300 done
…
299 out of 300 done
300 out of 300 done
The dataset you tested has 6 classes
The training set is of size 300, and the test set is of size 300.
The time series are of length 60
The error rate was 0.12
In this package we have produced a
Excel file that gives basic
information about the datasets
(number of classes, size of train/test
splits, length of time series etc)

In addition, we have computed the

error rates for:

• Euclidean distance
• DTW, unconstrained
• DTW, after learning the best
constraint in the test set*

*Note that our simple method for

learning the constraint is not
necessary the best (as explained in
the next slide).
 Worked Example
We can use the Archive to answer the following question. Is DTW better than Euclidean distance for
all/most/some/any problems?
As explained in [4], if DTW is only better on some datasets, this is not very useful unless we know ahead of time
that it will be better. To test this we can build a Texas Sharpshooter plot (see [4] for details).
In brief, after computing the baseline (here, the Euclidean distance) we then compute the expected improvement
we would get using DTW (at this stage, learning any parameters and settings), then compute the actual
improvement obtained (using these now hardcoded parameters and settings).

When we create the Texas Sharpshooter plot , each dataset fall into one of four possibilities.

In our worked example, we will try to optimize the performance of DTW,

and predict its improvement (which could be negative), in a very simple
way. Texas Sharpshooter Plot [4]
Expected Improvement: We will search over different warping window
constraints, from 0% to 100%, in 1% increments, looking for the warping
We expected We expected
window size that gives the highest 1NN training accuracy (if there are to do worse, an
ties, we choose the smaller warping window size). but we did improvement

Actual Accuracy Gain

better. and we got it!
Actual Improvement: Using the warping window size we learned in the
last phase, we test the holdout test data on the training set with 1NN.

We expected We expected
Note that there are better ways to do this (learn with increments
to do better,
smaller than 1%, use KNN instead of 1NN, do cross validation within the to do worse,
test set etc). However, as the next slides show, the results are pretty but actually
and we did. did worse.
unambiguous even for this simple effort.

Expected Accuracy Gain

 2.2

78

The results are strongly

2
supportive of the claim 82
“DTW better than
Euclidean distance for most
problems”

We sometimes had 1.8

difficultly in predicting
when DTW would be
better/worse, but many of
the training sets are tiny,
making such tests very 76
Actual Accuracy Gain

1.6
difficult.

For example, 51 is
BeetleFy, with just 20 train
and 20 test instances. Here
1.4
we expected to do a little
better, but we did a little 41 40
worse. 68 10
42 52
In contrast, for 76 15
1.2 709
39
(LargeKitchenAppliances) 27
5 38 7
we had 375 train and 375 14
1385 29 6 7280
test instances, and where 59
60 3 28 79
6411
able to more accurately 198 32 71
3681
44
predict a large 48 6254 174
2584323 37
67 45
improvement. 65 20 1884
1 63 24
16
83
34
56
86
21
30
1725
12
4773
49
26
55
22 31 7550
4 77 35 53 69
61 33 4657
51
66

0.8
0.8 1 1.2 1.4 1.6 1.8 2 2.2
Expected Accuracy Gain
 2.2

e
em
on
Ph
pli
(after plotting in Matlab, nAp
e
the code is in Appendix A, itch
2 allK
m
you can zoom in to avoid S
the visual clutter seen to
the right).

1.8

s
ce
plian
p
nA
che
K it
rge
La
Actual Accuracy Gain

1.6

1.4
et Y t
im r ick r ick e X
letS
e
C C
n
ap a ce ke
S h r
T skset Z 7 C hic
icm g- d
noon
Cnyrym tn in B ir
y n
y g h
s S d sR
S Li
o rdWsoUrnC2 lEC G af s es
1.2 n 1 W 5a
0 e i
ctat get-aroo 2 d U Le ipce ev ic
p t i o F F n
r ) i n
L n
noo c tkrayr oYu2 te p S e
T v
y D
Boseu hswtC l) ion
ro u nta Cfm eaLeepigrgtniaeTcssryoiXrnrieberS ay (al O iceDn ra t
g eG g me Scdeoa)gor(khm
e t
Wu
f i b e l L b
IAI
g lrael orGrrataZxxc1t ac e ic s
l e
Scctrre
g e
a
em u
nWPyxelneIarTm t
aLurveuIW C i
nceeLreyAThrheorre F c rs pt E c e fr i
e A Se TaooF tduhequnLaG rnreaG
tltelisnioLunstxiniutrotbTflrnaiC GoeTrCo
u tlin Toe a sadgshiliogactS
(ErteedSaTheAirew hO
r
rtA
s sC G
auoaeS d t C s C p ute Ha urfa Re
h e o
e r l Oe c
udeatxsruttansatlilnEutl E e i n om S
nxO FgTism bWtaw linleM PGraLGnpaxntvehlstetonasG
anvogSChnerStodaeuaEm hnB eoP eynux O ineds C c t
ot
ha la c tlWa
a noirxnxxW Oi uS i P yd
W
e
C l cebaaC
W
zlG
h
o
rR eD eR
e
ovelblsaam
D sFFO
a
t l e Rob
n
r
laPaHiuiensgrC r F l a ry OP e
udvlwis0iiv0gviloeianlT u e a t a r O
leP shehaaC tiaP abA eM eorsICS riS
aa5BT
vwalena0aaFsO n-lPLthLA dOwH e Cor y AIB
dd
Mi Beef s
nlP
taI lP S trhC
D
Y ltoG
r oaooionaoU
rlw yfcrtnm
yfxrC
a
W
-eoideIm
fP G
n nlem
A
i vaG a a nrro n n m ne
1 D i ta e
t C ESS P
Wt
ICd
DW ra
FoEnP n E - I
OGi M H A C
h u
m eA tl i S o Ha Wi
Dis Mo ANNoP rox x y O u
la n Fl
P haeetle
le B
dd
Mi

0.8
0.8 1 1.2 1.4 1.6 1.8 2 2.2
 Suggested Best Practices/Hints
1. If you modify the data in anyway (add noise, add warping etc), please give the modified data back to the archive
before you submit your paper (we will host it, and that way a diligent reviewer can test your claims while the paper
is under review).
2. Where possible, we strongly advocate testing and publishing results on all datasets (to avoid cherry-picking),
unless of course you are making an explicit claim for only a certain type of data (i.e. classifying short time series).
In the event you don't have space in your paper, we suggest you create an extended tech report online and point
to it. Please see [4] (esp. Fig 14) for some ideas on how to visualize the accuracy results on so many datasets.
3. If you have additional datasets, we ask that you donate them to the archive in our simple format.
4. When you write your paper, please make reproducibility your goal. In particular, explicitly state all parameters. A
good guiding principle is to ask yourself Could a smart grad student get the exact same results as claimed in this
paper with a days effort?. If the answer is no, we believe that something is wrong. Help the imaginary grad student
by rewriting your paper.
5. Where possible, make your code available (as we have done), it will makes the reviewers task easier.
6. If you are advocating a new distance/similarity measure, we strongly recommend you test and report the 1-NN
accuracy (as we have done). Note that this does not preclude the addition of other of tests (we strongly encourage
additional test), however the 1-NN test has the advantage of having no parameters and allowing comparisons
between methods.
7. Note that the data is z-normalized. Paper [7] explains why this is very important.
 Suggested Reading
1. Xiaoyue Wang, Abdullah Mueen, Hui Ding, Goce Trajcevski, Peter Scheuermann, Eamonn J.
Keogh: Experimental comparison of representation methods and distance measures for time
series data. Data Min. Knowl. Discov. 26(2): 275-309 (2013).
2. Bing Hu, Yanping Chen, Eamonn J. Keogh: Time Series Classification under More Realistic
Assumptions. SDM 2013: 578-586.
3. Hills, J., Lines, J, Baranauskas, E., Mapp, J. and Bagnall, A. Classification of time series by
shapelet transformation. Data Mining and Knowledge Discovery Journal. ISSN 1384-5810,
2013.
4. Gustavo E. A. P. A. Batista, Xiaoyue Wang, Eamonn J. Keogh: A Complexity-Invariant Distance
Measure for Time Series. SDM 2011: 699-710
5. Keogh, E., Zhu, Q., Hu, B., Hao. Y., Xi, X., Wei, L. & Ratanamahatana, C. A. (2011). The UCR
Time Series Classification/Clustering Homepage.
6. Eamonn J. Keogh, Shruti Kasetty: On the Need for Time Series Data Mining Benchmarks: A
Survey and Empirical Demonstration. Data Min. Knowl. Discov. 7(4): 349-371 (2003)
7. Thanawin Rakthanmanon, Bilson J. L. Campana, Abdullah Mueen, Gustavo E. A. P. A. Batista,
M. Brandon Westover, Qiang Zhu, Jesin Zakaria, Eamonn J. Keogh: Addressing Big Data Time
Series: Mining Trillions of Time Series Subsequences Under Dynamic Time Warping. TKDD
7(3): 10 (2013)
Shiyuan Liu, Li Lv. awei Han & Manish Gupta Greg Fanslow Inderjit Dhillon and HYUK CHO
Thiago Santos Quirino, Mei-Ling Shyu Iinstry Liang and Qin Lv Benjamin Bustos and Heider Sanchez Enriquez Xiaojin Li Aida Valls
Pierre-Franqois Marteau, Eirik Benum Reksten Thinh Vuong and Thinh Vuong Korkinof, Dimitrios Narayanan Chatapuram Krishnan and Sethuraman
Cosmin Bocaniala, Lancaster University. Lucas Gallindo Martins Soares Yiorgos Adamopoulos Yixin Chen and Yujie He Panchanathan
Yueguo Chen,Anthony K.H. Tung, Beng Chin Ooi, National Sungyoung Lee and La The Vinh Zhao Xiaohui Christina Yassouridis Stephan Gunnemann and Thomas Seidl
University of Singapore. Huidong (Warren) Jin Xue Bai itti laurent and jiaping ZHAO Thirumaran Ekambaram and M. Narasimha Murty
Vernon Rego, Vernon Rego. Purdue University. Santiago Velasco Baker Abdalhaq Ahmad Al-Hasanat Christine Preisach and Lars Schmidt-Thieme
Mislav Malenica, Tomislav Smuc Jairo Cugliari Paria Shirani and Mohammad Abdollahi Azgomi Artur Dubrawski, Mathieu Guillame-Bert Dino Isa and Rajprasad Kumar
Man Hon WONG, ZHOU Mi, The Chinese University of Quazi Abidur Rahman Manuel Oviedo de la Fuente. Rodica Potolea and Victor Ionescu Feibao Zhuo
Hong Kong. Hungyu Henry Lin and James Davis JiRinki Pakshwar JANETH CAROLINA RENDON AGUIRRE Frans van den Bergh
Guillaume Bouchard Xerox Research Centre Europe. Azuraliza Abu Baka and Almahdi Mohammed Almahdi Nikita Mishra and Somesh Kumar Javier Prieto Kfir Glik
Hoa Vo and David Joslin Seattle University. Benjamin Bustos and Victor Sepulveda Mike Jones Gianniotis, Nikos Xiao Yu
Carlotta Orsenigo , University degli Studi di Milano. Krisztian Buza Deák Szilárd Nicola Rebagliati Wesley Chen & Pavlos Protopapas Yingying Zhu
Sahar Torkamani and Volker Lohweg
Dr. Paolo Ciaccia Young Xin Ira Assent and Søren Chrestensen Rosanna Verde and Antonio Balzanella
Najarian, Kayvan
Xiaoqing Weng, Jiaotong University. Shen Wang and Haimonti Dutta Marcela Svarc Pengjiang Qian Paul Baggenstoss
Longin Jan Latecki and Qiang Wang, Temple University. Victor Garcia-Portilla Bankó Zoltán Hudson Fujikawa and Renato Ishii Koichi ASAKURA and Wei Fan
Tony Bagnall Samir Al-Stouhi yasuko matsubara and Yasushi Sakurai Antonio Gattone Lucia Sacchi and Iyad Batal
Alexandra M. Carvalho and Susana Vieira and Lucia Cruz
Michail Vlachos, IBM. Guo Fei Liu, Yueming and Su, Jianzhong Morne Neser
Hanci Lyu and James Kwok
Bernard Hugueney Teodor Costachioiu hongxiong ye and QIAN-LI MA Peihai Zhao Luca Chiaravalloti
Sourav Mukherjee Stephen Pollard Huseyin Kaya Sergey Milanov vikram deshmukh
Marcos M. Campos (Oracle) Eser Kandogan Hezi Halpert and Mark Last Zhibo Zhu Harri M.T. Saarikoski
Ludmila I. Kuncheva lida rashidi Paolo Missier and Tudor Miu Stefan Kramer and Atif Raza Caio Nogara Andreatta and Neusa Grando
Edward Omiecinski and Jun Li Takashi Washio and Satoshi Hara Jiandong Wang and Peng-Cheng ZOU Brijnesh-Johannes Jain Ying Xie
Victor Eruhimov, Intel Meizhu Liu and Baba C. Vemuri Shahriar Shariat Talkhoonche and Vladimir Pavlovic Qian Xiaochao Ulf Grobekathofer
Rob Jasper Jose Principe and Sohan Seth Hideo Bannai Felix Reinhart and Witali Aswolinskiy Christophe Genolini
Andre Coelho Ramanuja Simha and Rahul Tripath Gerard Medion and Dian Gong Kwangho Heo Celine Robardet
Gernot Herbst Rodolphe JENATTON and Francis Bach Shengfa Miao and Arno Knobbe Cun Ji Musa Chemisto
Vit Niennattrakul Siladitya Dey and Ambuj Singh Penugonda Ravikumar Sara Carolina Gouveia Rodrigues and Cláudia Antunes Wangmeng Zuo
Nozomi Matsuda Mahsa Orang and Nematollaah SHIRI V. Nabil Alshurafa and Majid Sarrafzadeh Kumar Vasimalla Paolo Remagnino
Flavio Miguel Varejao and Idilio Drago Saket Saurabh Vladislav Miškovic Jose Alejandro Cordero Soumi Ray and Tim Oates
HAORIANTO COKROWIJOYO TJIOE Wu Wush David Providakes and Jason Wang Liu Xiao Pierre Ganarski and Francois Petitjean
Molnar Miklos Deepti Dohare Wei Ding and Yang Mu Carlos Francisco Soares de Souza Joydeep Ghosh and John Tourish
Steinn Gudmundsson and Thomas Runarsson Subhajit Dutta. Meng-Jung Shih and Shou-de Lin Yurong Luo Alireza-Xaker
Niall Adams and Sai Wing Man Evgeny Pyatkov OSman Gunay Wang, Kaile(Keller) Wilfgang nejdl and ERNESTO DIAZ-AVILES
Isabel Maria Marques da Silva, Maria Eduarda da Rocha Ion George TODORAN Lu Min and Xiaoru Yuan Babak Hosseini Romain Tavenard and Laurent Amsaleg
Pinto Augusto da Silva and Joaquim Fernando Pinto da Qin Zou Hiba Bhamroukhi Dong Han Hahn-Ming Lee, Christos Faloutsos,Hsing-Kuoh Pao, Ching-
Costa K.S.SUBHASHINI Laurence Chu Colin O’Reilly Hao (Eric) Mao
Panagiotis Papapetrou and George Kollios Feng Gu, Julie Greensmith Puneet Singh Aldo Goia Woong-Kee Loh
Huang Tan Pedro Felzenszwalb Joaquim Vinicius C Assuncao Rahim Khan bikesh singh
Sergio Guadarrama Onur Seref B Kalyan Kumar and debarun kar Adam Oliner Marco Grimaldi, Cesare Furlanello and Giuseppe Jurman
Alicia Troncoso Lora Alessia Albanese Jim Austin and Alexander Fargus Arvind Balasubramanian and B. Prabhakaran Chonghui Guo
Pyry Avist Seniha Esen Yuksel and Paul Gader Jundong Li and Osmar R. Zaiane Soon-hwan Kwon, Jong Ho Lee Saeid Rashidi
Peng-Yi Lai Alexandros Iosifidis Bilel Ben ali Chen Yun Yanchang Zhang
Yong Fu Rana Hussein and Neamat Farouk El-Gayar Oya Celiktutan, bulent sankur and Cem Sübakan ndre Rodrigo Sanches and Nina Sumiko Tomita Hirata S R Kannan
Soheil Bahrampour Matteo Danieletto Michael Pettigrew Claudio Piciarelli and Gian Luca Foresti FRANCISCO JAVIER CUBEROS GARCIA-BAQUERO
Long Yao and Meng Bo Ying Zhang Katarzyna Kaczmarek and Olgierd Hryniewicz Wei T. Yue Dr. SOTIRIOS P. CHATZIS
Robert Moskovitch and Yuval Shahar Saeid Rashidi Jun ha Park Michael Botsch and Josef A. Nossek Ming Luo and Igor G. Zurbenko
Abdellali Kelil Xiaobo Wu and S Chen Daniele Codecasa and Fabio Antonio Stella Bingyu Sun Yuya Sakamoto
Puspadevi Kuppusamy Sun Lei and Yu-Jiu Yang Anqi Wu and Raghu Raghavendra Sun, Fu-Shing Yosi Keller
Qun Dai and Songcan Chen Belen Martin Barragan Nur Zuria Haryani and Abdul Razak Babak Amiri David Corney
Lisa Gralewski Mustafa Ahmed and Hua-Liang Wei Hamdan Ana Arribas Gil Xing ChunXiao and Du Xutao, Tsinghua University Alexander Gray and Nishant Mehta
Maria Teresinha Arns Steiner and Rosangela Villwock KHALIL BRAHMIA Zhe Bao and Junpeng Bao Elloumi Samir , Sondess Bentekaya Tomoyuki Hiroyasu and Takuma Nishii
Amir Ahmad and Galvin Brown Maria Luisa Sapino and Rosaria Rossini Alexio Kotsifakos and Vassilis Athitsos Li Shijin Blaz Strle and Martin Mozina
Abhijit Jayant Kulkarni Peiman Barnaghi and Azuraliza Abu Bakar Vincent S. Tseng Erik Learned-Miller, Marwan A. Mattar Yasin Bakis
Xingquan (Hill) Zhu Hammoud Aljoumaa and Dirk Soeffker Ryan Kleck and Gu Quanquan Chiranjib Bhattacharya,Karthik K Mrinal Mandal and Cheng Lu
Amol Deshpande and Qiang Qiu Haris Gavranovic Gavin Smith Nicandro Cruz Ramirez Hendrik Blockeel and Kurt Driessens
Vercellis Carlo and Gianni Alberti Matthias Mouton Usue Mori, Jose A. Lozano, Alexander Mendiburu Jiankui Guo Fudan University Jonas Haustad and Yongluan Zhou
Pamela Nerina Llop Mohammad Reza Daliri Mohammed Hasan Al-Weshah Bin Z Zhang IBM Julien Rabin
Tobias Scheffer FABIO STELLA Sarah Brockhaus and Sonja Greven Yi-Dong Shen and Zhiyong Shen Oscar Gerardo Sanchez Siordia and Isaac Martin de Diego
Jochen Fischer Rohit J. Kate Fenghuan Li Georgios Evangelidis, Leonidas Karamitopoulos Scott Deeann Chen
Mao Ye and Yingying Zhu Ruoqian Liu, Yi Lu Murphey Wang Yu Hendrik Purwins Lixia Wu
Cintia Lera jennifer bavani krishnan kehoe and Rahul Singh Mayank Mohta and Adit Madan Jignesh M. Patel and Michael Morse Tewari, Ashutosh
George Runger and Mustafa Baydogan Gert Van Dijck and Marc Van Hulle
George Runger and Rohit Das Srinivasulu Reddy and Naga sundaram RANGEENA T V radhakrishnan
Mohamed Ghalwash and ZORAN OBRADOVIC
Omar U. Florez and Seungjin Lim Chiatung Mao and Jia-Dong MAO Chao Hui Lee and Vincent Tseng Zhou Zhou
Jieyue Li
Ruy Luiz Milidiu and Pedro Teixeira Josif Grabocka Maciej Luczak Linh Tran (Boeing) Troy Raeder
Mykola Galushka and Dave Patterson Marco Cristani Zhiguang Wang and Tim Oates Hugo Alonso Vilares Monteiro and Joaquim Fernando Pinto da Costa Amit Thombre
Tomas Olsson David Minnen
Rahul Sinha Lukas Pfahler and marco stolpe Nhut Ta Hoang
Tianwei Liu
Minh Hoai Nguyen and Fernando de la Torre Rickey Agarwal Tsuyoshi Mikami Oisin Mac Aodha
Cexus Jean-Christophe
Qiang Yang and Sinno Pan
Eric Eaton Antonello Rizzi and Antonello Rizzi Yada Zhu and Jingrui He Ali Farahpoor
Paolo Tormene
NGUYEN Van Hanh Zhai Ting Ting Thanhvinh Vo and Duong Tuan Anh Xingwang Zhao
Hui Ding and Peter Scheuermann
Ming Zhang
Lucas F. Rosada Chris Carbone Ronaldo Cristiano Prati Wu Gang
Shanghai Jiao Tong and Zhang Zhongneng
Nicky Van Thuyne Xiaohui Huang Christian Gruber and Bernhard Sick Tetsuji Hidaka
Zhang Zhang and Dacheng Tao
Silvia Chiappa
Skopal Tomas and Michal Vajbar Trevor Tian Nicolas Ragot Wang He Nan
Ankur Jain
Carlo Piccardi and Martina Maggio Ta Minh Thuy Tanmoy Mondal David Weston
Maria Cristina Ferreira de Oliveira and Aretha Barbosa Alencar
Yin Zhou and Kenneth Barner
Muhammad Aamir Khan Guillem Rigaill Febri Andriani Dr. Huayang (Jason) Xie
Suzanne Tamang and Simon Parsons
Larry Deschaine Fatemeh Kaveh-Yazdy and mohammad reza zare Myeong-Seon Gi Jia-Lien Hsu
carlotta orsenigo and carlo vercellis
Pengtao jia
Janosa Andras Bahaeddin Eravci and Hakan Ferhatosmanoglu Azer Kerimov Saeed R. Aghabozorgi
Farid Seifi.
Andrew Starkey Haobo Jiang Hanan Shteingart Paolo Giussani
Clodoaldo Aparecido de Moraes Lima
Bingyi Kang
Karthik Marudhachalam and Ansaf Salleb-Aouissi Xin Qi Konstantinos Blekas Zhenhui (Jessie) Li and Jiawei Han
Jing Zhang
Rayner Alfred and Samry Mohd Shamrie Sainin S. Mohanavalli Juan Prada Jeff Patti
Jon Froehlich and Yi-Chun Ko
Ben Fulcher and Nick Jones Muthyala Kartheek,Navneet Goyal
Evins lio and Oumsis Oumsis Matthias Klusch and Josenildo Silva Emanuele Ruffaldi and Leonard Johard
Victor Sheng Mojtaba Najafi, Dr. Kangavari
Hanjing, Su Jing Yang Tomasz Pander Cedric Frambourg and Ahlame Douzal Chouakria Philippe ESLING
Komang Sidhi Artha
Marco Cuturi Pierre Allain and Thomas Corpetti Rakia JAZIRI and Mustapha Lebbah Zoe Falomir Llansola
Philip Knaute and Nick Jones
Wang Zhimin Changcheng Xiang Dave Marshall and Andrew Aubrey Xiaoxu He
Ivan Mitzev and Nick Younan
Wojciech Mioduszewski and Jurek Blaszczynski Omar Torfason Cheng Po Man and Wai-Kuen Cham
Miao Zeng and Yubao Liu Juhua Hu and Jian Pei
Zhang Daokun Le Huu Thanh and Duong Tuan Anh. Doina Precup and Jordan Frank
Rene Vidal and Rizwan Chaudhry Hasari Tosun and John Sheppard
Xiangzeng Kong Laila Fatehy and Mahmoud Gabr Laurens van der Maaten
Md. Abdul Awal
Susan Cheng and Min Ding Amparo Alonso Betanzos Alle veenstra GUSTAVO CACERES CASTELLANOS
Andrey Sukhanov
Amit Ganatra and Dhaval Bhoi Olivia Maoy and Yixin Chen Nima Riahi Deepa Krishnaswamy
Cecil Schmidt
Carlo GAETAN and Paolo Girardi Mohiuddin Ahmad and md. Abdul Awal Wang Jialin
Sebastian Schmitz,Marcin Grzegorzek, Lukas Köping
Farzad Noorian Hyokyeong Lee and Rahul Singh
Tao Quang Bang Tanzy Love and Kyra Singh Dhaval Patel, Wynne Hsu and Lee Mong Lee. Abhishek Sharma Ahmad, Faraz and Smith, David
Jeremie Mary Manuele Bicego and Pietro Lovato Ville Hautamaki shouyi wang and W. Art Chaovalitwongse Velislav Batchvarov
Kai-Wei Chang and Dan Roth Vladimer Kobayashi Peter Sunehag Hyrum Anderson Gautier Marti
Joan Serrà and Josep Lluis Arcos Talayeh Razzaghi and Petros Xanthopoulos Richard Clements Qian Chen Dimitrios Kosmopoulos
Risa Myers and Risa Myers Mohit Sharma Hichem Frigui and Walid MISSAOUI Tomas Bartos and Tomas Skopal Rana Haber and Adrian Peter
Ankit Gupta Juergen Schmidhuber and Klaus Greff KASHIMA Toru Vitali Loseu Shachar Afek Kaufman and Ruth Heller
Weng-Keen Wong and Yonglei Zheng Heriberto Avelino Tang Ke and Guojie Song Alessandro Antonucci Chengzhang Qu and Jianhui Zhao
Cássio M. M. Pereira and Rodrigo Fernandes de Mello Manoj Apte Jinfu Fan Andrii Cherniak and Vladimir Zadorozhny Basabi Chakraborty
Subutai Ahmad Nehal Magdy Saber Ruchira Guha Julia Gabler and Stephan Spiegel Grzegorz Dudek
Visuru Lafi Mohamed Rizny Webber Chen and Dan Stashuk Fan Zhou and Wu Yue Wasyl Malyj Sindhu Ghanta and Jennifer Dy
Eunice Carrasquinha, Ana Pires, Conceição Amado Huaihou Chen and Philip Reiss Chen Duansheng Piotr Sobolewski and Michal Wozniak Yu Fang and Huy Xuan Do
Huaming Huang and Mehrotra Seung-Kyu Lee Cheng wencong Iris Adae, Michael Berthold and Sebastian Peter Ali Raza Syed and Andrew Rosenberg
Abdulallah Balamesh Marcelo de Azevedo Ávila Xiaoli Li Omid Geramifard and Xu Jianxin Agada Joseph
Feng Zhou and Fernando De la Torre Subhajit Dutta and Anil K. Ghosh Fedor Zhdanov and Vladimir Vovk Ng Wee Keong and NGUYEN HAI LONG ricardo garcia rodenas
Johannes Schneider MAILLARD, Bruno WU Quan-Yuan Li Tao Om Prasad Patri
José Manuel Benítez Sánchez (and Students) Yazhou Ren and Carlotta Domeniconi Anuradha Kodali Willian Zalewsk and Fabiano Silva Leonardo Nascimento Ferreira
Xin Zhao and Xue Li Eithon Cadag and Johan Grahnen Keith Noah Snavely Neelanjana Dutta, Sriram Chellappa, Donald Wunsch II Michal Prílepok
Daniel Gordon, Danny Hendler, Lior Rokach Dinkar Sitaram and Varun Shenoy Hilario Navarro Veguillas and Jesus Bouso Dominique GAY Ke Yi and LUO,Ge
Anirvan Chakraborty and Probal Chaudhuri Chan Syin and Vuong Nhu Khue Myong K. Jeong Mai Thai Son and Christian Böhm Bruno Ferraz do Amaral and Elaine Sousa
Qing He and DongZhi Rodrigo Araujo Luca Gastaldi Ioannis Kaloskampis and Dave Marshall Edward GONG and Si Yain-Whar
Ioannis Paparrizos and Luis Gravano Mohamed Kamel Dileep George MYAT SU YIN Chaoyi Pang
Marcos Quiles. Borgwardt Karsten M. and Llinares Lopez Felipe Roni Khardon and Yuyang WANG Aiping Zhou Vida Groznik and Aleksander Sadikov
Chris Grieves and Rida E. Moustafa (Shalash) Jiang Liyang Alexandros Nanopoulos Houtao Deng and George Runger Abdulhakim Qahtan and Xiangliang Zhang
Ben Marlin Timothy Ravasi and Gregorio Alanislobato Beibei Zhang and Rong Chen Tianyu Li and Dr. Dong Cao Duy Truong
maryam tayefi James Zhang Sile O'Modhrain May D Wang and James Cheng Oliver Obst
Huiping Cao and Zhe Xie Heloisa de Arruda Camargo and Fábio José Justo dos Santos Amy McGovern Thomas Rueckstiess and Patrick van der Smagt Yung-Kyun Noh
Christian Hundt and Elmar Schömer D.Pradeep Kumar Ya-Ju Fan and W. Art Chaovalitwongse Rebecca Willett Wei Song
M. en C. Jorge Ochoa Somuano Steffen Dienst and Stefan Kühne Xiaobin Li krystal taylor Yu Su
Goce Ristanoski, James Bailey and Wei Liu Christian Hahn and gudrun stockmanns Smruti Sarangi Pengfei Zhu Uwe Aickelin, Eugene Ch'ng, Xinyu Fu
Paul Viola (Yes, as in Viola- Jones) haoran xu Peter Grabusts Bohdan Kozarzewski Quang Nguyen
JYH-CHARN (STEVE) LIU and Hao Wang Chuanlei Zhang Fernando Cela Diaz Quangang Li Or Zuk, Avishai Wagner, Tom Hope
TSUJIMOTO Takaaki and Prof. Uehara Youngha Hwang and Saul Gelfand Thomas Dyhre Nielsen and Shengtong Zhong Sun Xiangzhi Subhadeep Mukhopadhyay
Tomáš Kepic and Gabriela Kosková Tetsuya Nakamura Zhenzhou Chen Yutao Fan WEN GUO
Eva Ceulemans and Joke Heylen Daniel Alejandro Garcia Lopez Wesley Kerr and Paul Cohen Dimitri Mavris, Megan A Halsey, Curtis Iwata poyraz umut hatipoglu and Cem Iyigun
Alexis BONDU Richard J. Povinelli laurent baumes Wang Wei Yongjie Cai
Che-Jui Hsu Yi-Dong Shen and Zhiyong Shen Luis Antunes MªIsabel Borrajo García Mehdi Faramarzi
Wang Chao Emmanuel Viennet Chih-Chun and Zeeshan Syed Dennis Shasha and Lefteris Soulas Tarhio Jorma and Chhabra Tamanna
Wong, Weng-Keen and Nick Sullivan Remi Gaudin nicolas nicoloyannis Rene Vidal and Merve Kaya Andrew Cohen Xiaoqing Weng
Pavel Senin Phil Gross Zhaozhong Wang Weng-Keen Wong and Xinze Guan Simon Shim
Qian Xu Francois Portet Duangmalai Klongdee and Chuleerat Jar Nagesh, Karthik Yan Gao Gao and Daqing Hou
Fengzhen Tang and Peter Tino Qi He , Dr. Kuiyu Chang, Dr. Ee-Peng Lim Rakesh Babu Thomas Steinke and Patrick Schäfer Maria Rifqi, Marcin Detyniecki and Xavier Renard
Qiang Wang Fabio Antonio Pereira Reis Bao Yubin Antonio Penta Frank Englert and Sebastian Kößler
Hesam Izakian Damien Tessier Rayner Alfred Katherine Anderson and Castle, Joseph P Paul Grant
Witold Pedrycz Cuvelier Etienne Jim Howard Bressan, Stephane juan colonna
Yun Chen and Yang, Hui Moataz El Ayadi & Mohamed S. Kamel Xiaojun Zeng and Geng Li Michael Böhlen and Mourad Khayati Elias Egho
Tanatorn Tanantong and Ekawit Nantajeewarawat Hillol Kargupta Antonio Neme Yi Zheng QUANG LE and Chung Pham
Cauchy Lee Andrey Ustyuzhanin MITTAPALLI SAI SUDHEER and V.Bhavani Lijuan Zhong Jitesh Khandelwal
Chedy Raissi Hussam Alshraideh Julien JACQUES Eduardo Gerlein and Martin McGinnity Karima Dyussekeneva
BARRE Anthony and RIU Delphine John M. Trenkle Abdulla Al Maruf and Kyoji Kawagoe Dr.Ing. MAURICIO OROZCO-ALZATE Wu, Ruhao and Wang, Bo
Yang ZHANG Boris Martinez Jimenez & Francisco Herrera Fernandez Neuza Filipa Martins Nunes and Hugo Gamboa Kevin Shi and Layne Lori Yuki Kashiwaba
Flavio Vinicius Diniz de Figueiredo Juan Jose Rodriguez Liang Ge Sharon Goldwater and HAIDER Adnan Ade Bailly
Jussara Marques de Almeida Zoltan Banko Geovanny Giorgana and Paul Ploeger Youqiang Sun Abhinav Venkat
Dr. J. Figueroa-Nazuno Anjan Goswami & Debashis Mondal Ugo Vespier and Arne Koopman Koki Nakatani and Kumiko Tanaka-Ishii Michael Hauser and Asok Ray
Victor Romero-Cano and Juan Nieto Simon Kagwi Mwangi Boaz Nadler Uttam Kumar Sarkar Tuan-Minh Nguyen and Michael Bromberger
Jianhua Zhao Simone Fontolan and Alessandro Garghetti jankim luo Aalaa Mojahed and Beatriz De La Iglesia Ruako TAKATORI
Leslie S. Smith, Abdulrahman A. Lin Zhang and Joe Song Mike Dessauer and Sumeet Dua Prof. L. Zhang Rahul Singh and Ryan Eshleman
Samuel J. Blasiak and Huzefa Rangwala Zhenzhi Huang and Zhenfeng He Changxin Gao Hrishav Agarwal Laura Beggel
Harm de Vries and Azzopardi, G. (George) Waleed Kadous YIN Hong-sheng Jimin Wang Tom Arodz
Folly Adjogou and Alejandro Murua Hao Hu and Qiang Yang Baiming Ma Piyush Kumar Victoria Sanchez
Dustin Harvey and Todd, Michael D. Kay Robbins and Dragana Veljkovic Ramon Huerta Qing Xie and Xiangliang Zhang Mike Sips and Carl Witt
Kilian Weinberger Daniel Pena, Regina Kaiser, Ana Laura Badagiani Petr Volny Abhay Harpale, Tianbao Yang and Daniel Marthaler Segolene Dessertine-Panhard
Daniel Kohlsdorf and Thad Starner Daniel Graves and Witold Pedrycz Francois Rheaume Andrew Finch Zhao Xu
Zengwen Mo Anne Denton Alexander Kolesnikov Mona Rahimi Lovekesh Vig
Faezeh Eshragh and Xin Wang Julia Hunter and Martin Colley Vishwajeet Singh Thakur Fred Nicolls Angelo Maggioni e Silva
Tuan Dang and Leland Wilkinson Lucia Sacchi Tomasz Gorecki Paulo Borges and George Darmiton da Cunha Cavalcant Abdelwaheb Ferchichi and Mohamed Saleh Gouider
Kang Li and Yun Raymond Fu Bernhard Seeger, Michael Grau. Kerem Muezzinoglu John Lach and Davis Blalock Wenlin Chen and Yixin Chen
Andrés Eduardo Castro Ospina John Maindonald Ritwik Kumar Heggere S. Ranganath and Vineetha Bettaiah Gui Zi-Wen and Yuh Jye
Zheng Zhang Balazs Torma YANG Yuhang Keith Henderson Romain Tavenard
Peter Fox and Yanning(Yu) Chen Nikolaos Chatzis Ali Shokoufandeh , Terence Tuhinanshu, Ernst Pisch Steve Lalley and Herbert Xiang Zhu Chandan Gautam
Jeremiah Deng and Feng Zhou Daniel Smith Maria Titah Jatipaningrum Behzad Mansouri Jinglin Peng
Franz Pernkopf and Nikolaus Mutsam Abdul Razak, Khairuddin Omar Anita Santanna, Nicholas Wickstrom Megha Agrawal Ahlame Douzal and Jidong Yuan
Alexey Chernyshev Elwin (Yong) lee seyma ketenci Zhihua Chen and Wangmeng Zuo Chen Jing
Benjamin Auder Alex Smola and Xinhua Zhang Kristian Hindberg Diep Nguyen Ngoc Zeda Li and Alan J. Izenman
Jiawei Yuan and Shucheng Yu Rudolf Kruse and Christian Moewes Godtliebsen Fred BATCHELOR Andrew Peng Zhang
Heidelinde Roeder and Jan Freund Pekka Siirtola Mary She and Jing Wang Jinwook Kim and Min-Soo Kim. Hanyuan Zhang and Xuemin Tian
Toon Calders and Thanh Lam Hoang Michael Berthold Justin Bayer and Patrick van der Smagt Warat Safin Georgios Giantamidis and Stavros Tripaki
Feixiang Gao David Bong and James Tan Ammal Al-Anazi pradeep polisetty Goutam Chakraborty and Takuya Kamiyama
Nancy Pérez-Castro Zhengzheng (Crystal) Xing and Jian Pei Kevin Shen Houshang Darabi and anooshiravan sharabiani Pipiras, Vlada and Stone, Eric
Tracy Hu H Leticia Arco Garcia and Rafael Bello Stephan Chalup and Arash Jalalian Chen Qiang Ping Li
André Gensler Nuno Constantino Castro and Paulo Azeved Leandro MinkuMichele Dallachiesa and Themis Liying ZHENG Herbert Kruitbosch
zairul hadi Ng Kam Swee Palpanas Sarat Kr. Chettri Adam Oliner, Jacob Leverich, Tian Chen, Tom LaGatta
Beau Piccart Antonio Irpino Andreas Brandmaier and von Oertzen, Timo Graham Taylor and Ethan Buchman Yulin Sergey
Tomas Pfister Jong Myoung Ko Garcia-Trevino, Edgar and Javier A. Barria Cai Qinglin Huan Liu
John Costanzo and Nathan Cahill Jonas Richiardi Hao Zhu and He Zhongsh Anuj Srivastava and J. Derek Tucker Vadim Vagin
Conceição Amado and Diogo Silva Hassani, Marwan and Sebastian Schaub Nipotephat Muangkote Ajithkumar Warrier Nikolaos Passalis
Zhang, Zhifei and Qi, Hairong Chen, Po-Yu,McCann, Julie, Yu, Weiren Aleksandar Pechkov Kyle Johnston, and Adrian Peter Graham Mueller
Diego Rivera García. Laetitia Chapel Myat Su Yin Serim Park and Gustavo Kunde Rohde
Michelle Zhang and Sirajul Salekin Lee Seversky and Jack M Fischer
 Appendix A: function plot_texas_sharpshooter()

% Compute a Texas Sharpshooter plot of DTW over Euclidean Distance. See SDM 2011 paper
% Batista, Wang and Keogh (2011) A Complexity-Invariant Distance Measure for Time Series. SDM 2011 [

Sharpshooter Plots texas_names = data_names;

texas_values = 1-error_rates;
% Note that the order of texas_names and texas_values must be the same.
% Note that here we convert error to accuacy, by subtracting from 1

expected_accuracy_gain = texas_values(:,2)./texas_values(:,1);
actual_accuracy_gain = texas_values(:,3)./texas_values(:,1);

plot(expected_accuracy_gain,actual_accuracy_gain,'r.'); % Produce plot just so we can get Xlim and Ylim

Xaxis = get(gca,'XLim');
Yaxis = get(gca,'YLim');

Here is the code we clf

hold on;

used to produce the axis square;

patch([Xaxis(1) 1 1 Xaxis(1)],[Yaxis(1) Yaxis(1) 1 1 ],[0.9843 0.8471 0.5765]); % Bottom left quadrant

sharpshooter plots. patch([1 Xaxis(2) Xaxis(2) 1],[1 1 Yaxis(2) Yaxis(2) ],[0.9843 0.8471 0.5765]); % Top right quadrant

plot(expected_accuracy_gain,actual_accuracy_gain,'r.');

xlabel('Expected Accuracy Gain');

ylabel('Actual Accuracy Gain');

for i = 1: length(texas_values(:,1))
%text(expected_accuracy_gain(i),actual_accuracy_gain(i),int2str(i))
end

for i = 1: length(texas_values(:,1))
text(expected_accuracy_gain(i),actual_accuracy_gain(i),texas_names(i,:),'rotation',+30)
end

end

function names = data_names()

names =...
['Plane '
'Car '
'Synthetic Control '
'Gun-Point '
'CBF '
'Face (all) '
'OSU Leaf '
'Swedish Leaf '
'50Words '
'Trace '
'Two Patterns '
'Wafer '
'Face (four) '
'Lightning-2 '
'Lightning-7 '
'ECG '
'Adiac '
'Yoga '
'Fish (readme) '
'Beef '
'Coffee '
'OliveOil '
'CinC_ECG_torso '
'ChlorineConcentration '
'DiatomSizeReduction '
'ECGFiveDays '
'FacesUCR '
'Haptics '
'InlineSkate '
'ItalyPowerDemand '
'MALLAT '
'MedicalImages '
'MoteStrain '
'SonyAIBORobot SurfaceII '
'SonyAIBORobot Surface '
'StarLightCurves '
'Symbols '
'TwoLeadECG '
'WordsSynonyms '
'Cricket_X '
'Cricket_Y '
'Cricket_Z '
'uWaveGestureLibrary_X '
 The Password
• As noted above. My one regret about creating the UCR archive is that some
researchers see improving accuracy on it as sufficient task to warrant a
publication. I am not convinced that this should be the case (unless the
improvements are very significant, or the technique is so novel/interesting it
might be of independent interest).
• However, the archive is in a very contrived format. In many cases, taking a
real world dataset, and putting it into this format, is a much harder problem
than classification itself!
• Bing Hu explains this nicely in the introduction to her paper [2], I think it
should be required reading for anyone working in this area.
• So, the password is the three redacted words from this sentence “Every item
that we ******* ## @@@@@@@ belongs to exactly one of our well-
defined classes”, after you remove the two spaces.
• The sentence is on the first page of [2].