Bibliometric study of the scientific research

on Learning to Rank between 2000 and

2013

Oscar J.Alejo-Machado, Juan Manuel

Fernndez-Luna & Juan F.Huete

Scientometrics
An International Journal for all
Quantitative Aspects of the Science of
Science, Communication in Science and
Science Policy

ISSN 0138-9130
Volume 102
Number 2

Scientometrics (2015) 102:1669-1686

DOI 10.1007/s11192-014-1467-4

1 23
Your article is protected by copyright and
all rights are held exclusively by Akadmiai
Kiad, Budapest, Hungary. This e-offprint is
for personal use only and shall not be self-
archived in electronic repositories. If you wish
to self-archive your article, please use the
accepted manuscript version for posting on
your own website. You may further deposit
the accepted manuscript version in any
repository, provided it is only made publicly
available 12 months after official publication
or later and provided acknowledgement is
given to the original source of publication
and a link is inserted to the published article
on Springer's website. The link must be
accompanied by the following text: "The final
publication is available at link.springer.com.

1 23
 Author's personal copy
Scientometrics (2015) 102:16691686
DOI 10.1007/s11192-014-1467-4

Bibliometric study of the scientific research on Learning

to Rank between 2000 and 2013

Oscar J. Alejo-Machado Juan Manuel Fernandez-Luna

Juan F. Huete

Received: 19 May 2014 / Published online: 4 November 2014

 Akademiai Kiado, Budapest, Hungary 2014

Abstract The application of machine learning algorithms in the construction of ranking

models is a relatively new research area which has emerged during the last 10 years within
the field of artificial intelligence and information retrieval. This paper presents a biblio-
metric study of scientific output on learning to rank (L2R) between 2000 and 2013. For this
procedure to be successful, every relevant bibliographic L2R record retrieved from the
Scopus database was considered. The records were processed according to a series of one-
dimensional and multi-dimensional metric indicators which were selected for the study.
The results of this research provide the scientific community with reliable, up-to-date
information about the state of L2R research and trends, and will enable researchers to
develop valuable studies to reinforce research, development and innovation.

Keywords Learning to rank  Bibliometric  Scientific production

Introduction

Since the end of the twentieth century, there has been an exponential growth not only in the
sources of scientific information but also in the ways that these are accessed. The iden-
tification of relevant information so that correct decisions can be made therefore represents
one of the challenges of the information society. As a result, studies need to be conducted

O. J. Alejo-Machado
Informatic Faculty, University of Cienfuegos, Cienfuegos, Cuba
e-mail: alejo@ucf.edu.cu

J. M. Fernandez-Luna  J. F. Huete (&)

Departamento de Ciencias de la Computacion e I.A., E.T.S.I. Informatica y de Telecomunicacion,
CITIC-UGR, Universidad de Granada, Granada, Spain
e-mail: jhg@decsai.ugr.es
J. M. Fernandez-Luna
e-mail: jmfluna@decsai.ugr.es

123
 Author's personal copy
1670 Scientometrics (2015) 102:16691686

in order to analyse the behaviour of the scientific output generated as a consequence of

worldwide research. This thereby ensures that knowledge is updated and subsequently
exploited in development and innovation in the business sector, research centres and
academic institutions. In order to achieve concrete, feasible results, mathematical models
have been linked to the library and information sciences and this has given rise to various
metric disciplines (e.g. bibliometrics, informetrics, scientometrics, etc.) which enable the
state of art of sciences, disciplines and knowledge areas to be measured by evaluating
scientific output on the basis of these indicators (Pritchard 1969; Lariviere et al. 2012; Tan
et al. 2014; Braam and Van-Den-Besselaar 2014; Yan et al. 2010).
The problem of ranking, which is considered to be a leading research topic in the field
of artificial intelligence (AI) and information retrieval (IR), is known as the learning-to-
rank (L2R) problem. The ranking problem lies at the core of IR: given a query representing
the users information needs a set of documents must be ranked according to their rele-
vance to the query. The ability to accurately tune a ranking function is particularly
important for web search engines such as Google, Yahoo!, Bing, Yandex, Baidu since it
directly affects the search experience of millions of users. In order to solve this problem,
L2R attempts to construct such a ranking model by using training data consisting of
queries, their corresponding retrieved documents and relevance levels provided by humans.
For this purpose, automatic machine learning technologies were used to optimize an IR
measure on test data. Once the model has been built, when a new query is received, the
system returns a list of documents which are ranked by the trained ranking model.
Interest in the L2R task began about 15 years ago and since 2005, a great number of
studies have been conducted on L2R and its application to IR (Joachims 2005; Yeh et al.
2007; Xu et al. 2008; Liu 2009, 2011; Valizadegan et al. 2009; Wu et al. 2011; Cao et al.
2006; Xu and Li 2007; Lin et al. 2013; Dong et al. 2014). In order to obtain an overview of
the field, it is appropriate to evaluate scientific output and this will enable researchers not
only to identify new lines of work but also to continue or redirect existing ones. With this
goal in mind, this paper presents a metric study on the L2R topic. This evaluation was
carried out based on a process of search and recovery of relevant bibliographical records
from the Scopus database between 2000 and 2013. All these bibliographical records were
processed according to a series of metric indicators selected for the study.
The main contribution of this paper is to provide reliable information about L2R
behaviour and trends, and more specifically, the most explored descriptors, the countries
with the greatest scientific output, the areas of knowledge where L2R is applied, validated
scientific sources that store the most relevant information, relationships between authors,
etc. Additionally, one of the most significant aspects is provided by the clustering of the
descriptors that enabled us to detail and interpret their behaviour from recent years to the
present.
All these tasks and the obtained results will provide researchers into this topic with
relevant, up-to-date knowledge about how and where L2R research is moving. These
aspects, among others, form the basis for elaborating programs, guiding research trends,
creating bibliographic libraries and teaching resources which are based on extremely
relevant scientific information.
In order to present the metric study, this paper is organized as follows: Major
approaches to learning to rank section outlines the various types of L2R approaches;
Data sources section presents the general procedure for obtaining the necessary L2R
scientific output; One-dimensional indicators and Multi-dimensional indicators sec-
tions analyse the data by considering one-dimensional and multi-dimensional indicators,
respectively; and finally Conclusions section discusses the conclusions of this work.

123
 Author's personal copy
Scientometrics (2015) 102:16691686 1671

Major approaches to learning to rank

Various methods based on machine-learning techniques have been proposed and applied
for L2R in IR. While discussing the frameworks and theoretical properties of such methods
is beyond the scope of this paper [a comprehensive review can be found in Liu (2011)], we
shall outline the three main approaches (although certain approaches cannot be categorized
in this way):
the point-wise approach, which transforms ranking into classification or regression on
single documents. Each training example comprises a set of document features and
their corresponding relevance value relating to a query. The learning process attempts
to map features into relevance degrees.
the pair-wise approach, which formalizes ranking as classification on document pairs,
so a classifier learns to label each object pair as correctly-ranked or incorrectly-ranked.
the list-wise approach, which directly minimizes a loss function defined on document
lists. In this category, the list-wise approach takes the entire list of documents
associated with a query in the training data as input and predicts their ground truth
labels in terms of a ranked list (or permutation) of the documents. Two main
alternatives are presented which depend on whether or not the loss function is explicitly
related to the optimization of an IR evaluation measure.
It is possible to observe how these approaches used for learning have evolved and the list-
wise approach is currently the most promising. It has been proven empirically that this
approach has certain advantages over the others, especially when predicting the top-ranked
documents. Moreover, pair-wise algorithms seem to outperform point-wise ones.
The ability to combine a large number of features is one of the advantages of learning-
to-rank methods. Any new progress on a retrieval model can easily be incorporated by
including the model output as one dimension of the features. Such a capability is highly
desirable for real search engines, but also in various related problems such as question
answering, multimedia retrieval, text summarization, online advertising, collaborative
filtering, machine translation, etc.

Data sources

The reliability of the results obtained from a bibliometric study largely depends on the
information source from where the data were extracted. Among the top databases in terms
of prestige and international impact we should mention Web of Science and SciVerse
Scopus (commonly called Scopus). If the compilation of arbitrary scientific journals listed
on Ulrichs International Periodicals Directory is taken as a reference, only 25 % are
processed by Web of Science against 50 % by Scopus. Moreover, Scopus processes 95 %
of the sources that enter the Web of Science (Canedo et al. 2010; Torres-Salinas and
Jimenez-Contreras 2010; ScienceDirect 2013).
Our study was conducted using bibliographic references recovered from the Scopus
database. The search strategy was formulated in such a way as to enable retrieval of
documents containing the English phrase Learning to Rank, in quotation marks,
in the fields of title, summary, keywords, thereby ensuring the relevance of the recovered
records. The search was limited to documents in the period 20002013. The results of these
14 years are representative for determining L2R behaviour and trends and will constitute
our research sample.

123
 Author's personal copy
1672 Scientometrics (2015) 102:16691686

As a result of the search, a total of 627 bibliographic records were recovered. These
were exported in .ris format to the Endnote X41 program (EndNote X4: Getting started
guide 2010) to build the bibliographic database. A search was conducted to verify whether
any records were duplicated in the database, and two such duplications were detected. The
bibliographic records were also verified in order to ensure that the fields contained the
necessary information for analysis of the selected indicators. Finally, the list of fields
comprising the bibliographic records were generated and these include authors, descriptors,
years, countries, etc. These lists were exported to text files (.txt) for subsequent processing
in Excel. The text documents were opened with Excel and this was initially used to
normalize field data in the case of a synonym or homograph. In order to accomplish this,
the fields were filtered and the data were organized alphabetically to eliminate risks.
The total number of bibliographic records obtained and the number of documents
published each year were then counted and organized. This offered a clearer view of the
years of greatest research activity; all the fields used to analyse the one-dimensional
indicators were analysed in the same way. Excel graphs were used to visualize this type of
indicator.
The analysis of multi-dimensional indicators, however, required all the bibliographic
records to be processed in Excel tables and the use of the Toolinf program (Excel com-
plement) functions. By compiling field lists with their corresponding codes and occurrence
dates, it is possible to interrelate variables so that they can be counted automatically using
the Toolinf program to generate matrixes and to visualize the processed information using
self-organizing maps (SOM) (Kohonen 2001) and social networks.
The following programs were used to visualize the multi-dimensional indicators: Vis-
covery SOMine2 (topologic maps) and Ucinet3 (Version 6.2) and NetDraw4 (social net-
works) (Version 2.06). NetDraw, which generates the social network presented in
Collaboration of the authors producing more than 5 papers section, is part of the Ucinet
package.

One-dimensional indicators

There was a significant increase in L2R scientific output between 2000 and 2013 as Fig. 1
shows. The points on the red line represent the number of records identified each year,
while the black line represents the increasing trend of this scientific output. Although only
four records on this topic were identified in 2005, scientific output increased to 107 records
in 2010. In December 2013, 134 records were recovered. This trend shows that the field of
L2R research is gaining interest among the scientific community and is a worthy research
area.

Areas where L2R is applied

The left-hand side of Fig. 2 represents the fields of knowledge (subject areas) where L2R is
mainly applied. As we can see, the field of knowledge which is most linked to L2R is

1
Available in http://www.endnote.com/pr-enx4win.asp.
2
Available from http://www.viscovery.net/somine/.
3
Available from http://www.arschile.cl/ucinet/.
4
Available from http://www.analytictech.com/netdraw/netdraw-versions.htm.

123
 Author's personal copy
Scientometrics (2015) 102:16691686 1673

Fig. 1 Evolution of L2R: number of publications and approaches used

computer science with 525 publications, representing 55.9 % of the sum of the values of all
the subject areas. This is evidently associated with the fact that the discipline of IR is
closely connected to the field of computer science. However, L2R has also been applied
with very promising results in other important fields where it is necessary to learn a ranking
model, such as for example mathematics with 107 publications, decision sciences with 82,
engineering with 80, business, management and accounting with 66, among others, which
reflects its multi-disciplinarity.
It is also interesting to focus on those topics or descriptors most explored in papers
published on a specific area of knowledge. In order to analyse these indicators, the key-
word field was selected. A total of 10,259 descriptors were obtained from our database.
Before analyzing these descriptors, we removed any duplicates and performed a normal-
ization process to combine the descriptors referring to the same topic, such as for example
information retrieval and information retrieval systems, and this gave rise to 3,104 dif-
ferent descriptors. The main topics, with an occurrence of over 40 records in the studied
period, are shown on the right-hand side of Fig. 2. The interrelationships between these
descriptors are analysed in detail in Multi-dimensional indicators section.
From the most frequently used descriptors we can distinguish two groups: the first
relating to the major L2R application fields, such as information retrieval or search
engines, and the second including those descriptors related to the technologies and
methodologies used, such as data sets, ranking functions, learning algorithms, etc.
A snapshot of the importance of L2R in the search industry is the fact that there are
eight patents (seven from Microsoft Corporation and one from Yahoo! Inc.) about different
strategies for improving result ranking. It is widely acknowledged that L2R is particularly
important for other web search companies such as Google, Bing, Yandex, etc. L2R enables
rankings to be accurately tuned and directly affects the search experience of millions of
users the world over. The strong connections between field and industry can be demon-
strated by the fact that around 20 % of the papers in the field were published by authors in
such companies, reaching a peak during the years between 2008 and 2011. Industry interest
can also be appreciated if we consider that companies such as Yahoo!, Microsoft or

123
 Author's personal copy
1674 Scientometrics (2015) 102:16691686

Fig. 2 Subject areas and descriptors associated with L2R

123
 Author's personal copy
Scientometrics (2015) 102:16691686 1675

Yandex actively promote research in the field by providing internal datasets for the
research community that include data gathered from users interacting with their systems.
These enormous datasets comprise thousands of queries and hundreds of features extracted
from millions of documents and are essential for the research community to develop new
algorithms.

L2R output of authors

Once the authors have been identified and their work quantified, the number of authors who
have published L2R research currently stands at 1,083. An analysis of the authors and their
output reveals that the most productive authors are Hang Li with 28 papers, Tie Yan Liu5
with 21, then Y. Wang with 17, and so on as shown on the left-hand side of Fig. 3. While
many other authors have written on the subject, the vast majority of papers are written by a
small group of authors as shown in Fig. 3.
We shall now focus on those countries with the highest L2R scientific output. By means
of a pie chart, the right-hand side of Fig. 3 depicts the number of publications for each
country, with the most productive being China with 245 publications and the United States
with 172, followed by the United Kingdom with 47, Canada with 23, etc. An in-depth
analysis of author affiliation reveals that the most relevant research in the field has been
conducted by Microsoft Research in Asia, which has strong ties with the Chinese Uni-
versities of Tsinghua and Beijing, and also by Microsoft Research in Redmond, USA.

Most cited authors in the last 10 years

One essential parameter to be considered when analyzing the state of the art of a specific
discipline is to study the most cited authors in scientific literature. In order to determine
this, we initially obtained every bibliographic reference for each recovered record. We then
counted all the bibliographic references where each of the known authors was cited for the
term in question. Figure 4 shows the top 20 most cited authors in the last 10 years, with the
two most notable being Tie Yan Liu and Thorsten Joachims. The horizontal bars represent
the number of times their work was cited.
By jointly examining each authors output and how many times each has been cited, we
can conclude that the field leaders (Bornmann et al. 2012), i.e. those combining excellence
(in terms of the most cited papers) and output, are from Microsoft Research, and in par-
ticular Hang Li, Tie-Yan Liu, Tao Qin and Ju Xu (from Microsoft Research Asia), Thorsten
Joachims (from Cornell University, which also collaborates with Microsoft Research and
Yahoo! Labs) and Christopher Burgess (from Microsoft Research at Redmond).

Most relevant L2R information sources

In order to analyse this indicator, we have identified the most relevant information sources
according to reference type with the following number of publications in the field: 397
conference papers, 143 journal articles, 81 chapters, but only two books. In terms of
international conferences, the top five most relevant information sources are the Annual
International ACM SIGIR Conference on Research and Development in Information
Retrieval (84), the ACM International Conference on Information and Knowledge Man-
agement (63), the Annual Conference on World Wide Web (19), the ACM International

5
Interested readers can consult his comprehensive review of the major approaches to L2R (Liu 2011).

123
 Author's personal copy
1676 Scientometrics (2015) 102:16691686

Fig. 3 Authors/countries with the greatest scientific L2R output

123
 Author's personal copy
Scientometrics (2015) 102:16691686 1677

Fig. 4 Most referenced authors in the last 10 years

Conference on Web Search and Data Mining (17), and the Annual International Confer-
ence on Machine Learning (12). The numbers in brackets represent the number of pub-
lished papers. The number of publications for these top 5 information sources represent
49.12 % of all L2R publications at international conferences. By establishing itself as an
important reference pillar in the area, the ACM SIGIR Conference is currently at the
forefront of state-of-the-art L2R research.
The most relevant information sources in terms of articles were published in the fol-
lowing journals: Information Retrieval (17), Machine Learning (11), Information Pro-
cessing and Management (7) and the Journal of Information and Computational Science
(5). As expected, the Information Retrieval Journal is the most predominant one with the
largest impact in the field. Finally, two of the most relevant information sources in the
sphere of serial publications are the European Conference on Information Retrieval (11)
and the Asia Information Retrieval Societies Conference (8).

Multi-dimensional indicators

If we consider that 3,104 different descriptors were obtained, it becomes necessary to carry
out a detailed study in order to discover how they tend to be used. In order to carry out this
analysis, we have classified them into three different groups:
Group 1: descriptors where research has boomed and then remained stable. In addition
to being the one most covered in published papers, these must have been regularly
covered at least annually in the study period.

123
 Author's personal copy
1678 Scientometrics (2015) 102:16691686

Group 2: incipient descriptors. Two different types will be considered:

Type 2.1: promising descriptors that have been regularly explored by area
researchers annually over the last 3 years. No frequency restriction is imposed on
these descriptors. It is possible that they might belong to Group 1 in the future.
Type 2.2: sporadic descriptors that have been dealt with in one of the recent years
but not developed regularly during the corresponding period.
Group 3: obsolete and abandoned descriptors. These descriptors have only one
associated paper and have not been dealt with in the last five or more years. They
correspond to partly resolved problems or topics of relatively low interest which are no
longer worthy of research.

Descriptors which have boomed and stabilised

Figure 5 shows the obtained descriptors as topological maps and their relationships
(frontiers in the map) during the 4-year periods 20062009 and 20102013 on the left- and
right-hand sides of the figure, respectively. We have selected these years as they will help
us identify not only recent trends but also the evolution of the most frequent L2R topics. In
each map, the intensity of the scientific output of each topic is represented on a colour scale
(from blue to red).
Although many descriptors have the same publication intensity, they are not necessarily
concentrated in one specific area. The position of the clusters indicates the relationship and
proximity to other descriptors. In order to aid interpretation of these figures, we have used
bold, black fonts to illustrate descriptors with a greater than average frequency.
Various conclusions can be drawn: the most frequent descriptors pinpointing the main
application domain of the discipline (information retrieval, ranking model and search
engine) remained stable during both periods. We can also see how in the 20062009 period
researchers were concerned about the ranking problem and the importance of using a good
dataset for their research. The lower part of the figure for the 20062009 period shows a
cluster of frequent descriptors about benchmark databases and training data. In July 2009,
the LETOR datasets were released 6 and so in the years that followed, researchers changed
the test databases and exploited the data by using different artificial intelligence tech-
niques, as shown by the importance of algorithm-related descriptors (e.g. data mining,
supervised learning, genetic programming, active learning, etc.). The success of the dif-
ferent approaches in the field of IR resulted in an increase in efforts devoted to the
application of L2R in related disciplines such as recommender systems (where the goal is
to learn the best order for recommending appropriate products to a user), question
answering (the aim of which is to rank candidate answers), image retrieval or natural
language processing.

Incipient descriptors in general

Incipient descriptors can be catalogued as those that in one way or another appear as a
result of new work and research. Generally speaking, the majority of these descriptors will
have a limited impact in the scientific community. The descriptors might occasionally be
dealt with irregularly and are considered by the authors of this work as incipient or

6
Available from http://research.microsoft.com/en-us/um/beijing/projects/letor/.

123
 Scientometrics (2015) 102:16691686
Author's personal copy

Fig. 5 Descriptors where research has boomed and then remained stable in the 4-year period 20062009 and 20102013, respectively
1679

123
 Author's personal copy
1680 Scientometrics (2015) 102:16691686

Table 1 Schema of incipient descriptors in L2R

# of papers Incipient and punctual 20092013 Incipient and promising 20112013

[ 10 Clickthrough data Computational linguistics

Evaluation measures Data processing
Information Services Mathematical models
Labeled data Objective functions
Neural Networks Query processing
Robot learning Scoring functions
Training sets Social media
User interfaces Websites
 6 and  10 Adaptative boosting Crowdsourcing
Clustering algorithms Evaluation metrics
Computer vision Database systems
Direct optimization High quality
Features selection Information Systems
Gradient descent Learning approach
Implicit feedback Learning frameworks
Kernel methods Learning process
Matrix factorization Micro-blog
Preference learning Semantic web
Probability density function Signal processing
Semi-supervised learning Similarity measure
Transfer learning Software engineering
Transductive learning
\6 Effective learning Algorithm
Evolutionary algorithms Complex structure
Intelligent systems Content-based image retrieval
Model adaptation Document ranking
Query expansion Information quality
Rank aggregation Parameter estimation
Ranking strategy Personalizations
Set theory Regression Trees
SVM Trees (Mathematics)
Unified framework
Video retrieval

sporadic descriptors (Group 2.2). We also considered the promising descriptors (Group
2.1) which have been regularly dealt with annually over the last 3 years. It is important to
acknowledge the importance of these descriptors as they may well become Group 1
descriptors in the near future.
Table 1 shows an overview of the descriptors classified as incipient (64) within the field
of L2R. For a better interpretation of the incipient descriptors, the schema was structured
into six representative quadrants, the descriptors in each quadrant are displayed alpha-
betically sorted. These quadrants comprise a vertical column separating the incipient and
promising descriptors (period 20112013) from the incipient or sporadic descriptors
(period 20092013). Looking at the schema horizontally, three rows were established: the

123
 Author's personal copy
Scientometrics (2015) 102:16691686 1681

first row is located at the bottom and contains the L2R descriptors that appear in up to five
scientific papers; the second row is located in the middle and contains the descriptors
where work has been slightly more intense (i.e. between six and ten papers); and the third
and final row is located at the top and contains the descriptors that have resulted in a
scientific output of more than ten papers. Following this division, we can easily identify
computational linguistics as the subject category most covered in 2010, 2011, 2012 and
2013.
Other descriptors are considered promising and the most suggestive of these include
transductive learning, semantic web, document ranking, personalization, crowdsourcing. It
would be interesting to focus on the following sporadic descriptors: evaluation measures,
robot learning, adaptive boosting, semi-supervised learning, feature selection, evolutionary
algorithms and rank aggregation. It is important to mention that in Version 4.0 of LETOR,
one of the proposed tasks directly relates to the semi-supervised ranking and rank
aggregation descriptors.

Obsolete and abandoned descriptors

A large number of descriptors are dropped with time, possibly because they represent
solved problems or because a new label has been identified for the particular topic.
These descriptors belong to Group 3 and are associated with the smallest number of
publications (only one paper), which was published more than 5 years ago.
At the final count, 53 descriptors were considered to be obsolete or neglected. Figure 6
illustrates a symbolic schema that represents the tendency for obsolescence when dealing
with certain L2R descriptors. In order to illustrate the behaviour of such descriptors, the
schema uses different colours to label the descriptors: red is used to represent those that
have not been covered for at least 6 years or at all; orange represents a period of 7 years
prior; blue represent a period of 8 years prior; and black a period of 9 years prior. The
symbols (?) () in conjunction with the direction of the arrows depict the tendency for
each subject category (according to its location) to become more or less obsolete. Thus, for
instance, probabilistic logics, reinforcement learning and time varying systems are obsolete
whereas the topic of error analysis has shifted with time towards the use of evaluation
measures.

The most frequently used descriptors by the highest impact authors

Analysing the descriptors from another point of view, we considered it interesting to

determine which descriptors are most frequently used by the highest impact authors.
With this in mind, we chose the top 50 authors and analysed the 10 most frequently
used descriptors (see Table 2). The column labelled as Total represents the total
number of times that these high impact authors studied each subject category in their
work. Finally, the results obtained corroborated the fact that most descriptors where
research has boomed and then remained stable (see Section Descriptors which have
boomed and stabilised) are covered by the greatest impact authors. By way of
additional information, we should mention that Hang Li (78), T.Y. Liu (64), T. Qin
(47), H. Zha (41), Y. Huang (35), Z. Zheng (31), G. Sun (26), J. Xu (26), Y. Chang
(20) and D. Metzler (19) are the top ten authors working on these descriptors, with the
number in brackets representing the number of times each author has covered them.

123
 Author's personal copy
1682 Scientometrics (2015) 102:16691686

Fig. 6 Tendency for

obsolescence in the treatment of
certain L2R descriptors

Most frequently used descriptors within the most relevant information sources

In order to analyse this indicator it was necessary to identify the most frequently used
descriptors in the most relevant information sources, in other words those sources that
publish the most about L2R-related descriptors. According to the description in Section
Most relevant L2R information sources, these are the Proceedings of the Annual
International ACM SIGIR Conference on Research and Development in Information
Retrieval, with 84 papers. Focusing on journal articles, the most relevant source is
Information Retrieval with a total of 17 papers. We shall now use topological maps to
summarize the relationship between the most frequently used descriptors in each relevant
information source by considering their co-occurrence.
The left-hand side of Fig. 7 represents the most frequently used descriptors at the SIGIR
conference and their relationships, and in particular information retrieval, search engines,
information services, ranking functions, etc. It is worth mentioning that 68 papers have
been published with the keyword information retrieval. On the other hand, the most
frequently used descriptors in the journal Information Retrieval (see the right-hand side
of Fig. 7) are information retrieval, learning algorithms, data sets, ranking functions, etc.

123
 Author's personal copy
Scientometrics (2015) 102:16691686 1683

Table 2 Most frequently used

Order Descriptors Total
descriptors by the top 50 highest
impact authors
1 Information retrieval 130
2 Learning algorithms 81
3 Ranking models 77
4 Machine learning 43
5 Support vector machines 34
6 Data sets 33
7 Information services 31
8 Optimization 30
9 Education 29
10 Search engines 25

Fig. 7 Most frequently used descriptors at the ACM SIGIR conference and in the Information Retrieval
journal, respectively

According to the distribution and intensity of the descriptors in both topological maps,
we can conclude that the most discussed descriptors in these information sources are also
consistent with the descriptors where research has boomed and then remained stable.
Nevertheless, a different pattern can be appreciated between the two groups. Journal
articles, therefore, seem to be more related to theoretical models (descriptors such as
learning algorithms, machine learning, algorithms have a greater impact) whereas SIGIR
papers seems to be more related to how to implement the results (descriptors such as World
Wide Web, search engines, user interface, web searches are more frequent). This can be
explained by considering the different publication objectives for both academia and
industry: academic researchers tend to publish in journals (since this is generally one of
their professional requirements) whereas industrial researches usually publish their results
at top conferences, such as for example SIGIR, which enables faster dissemination of the
results. The differences between both maps also reflect the fact that industry is closer to the
users, with direct data access for evaluating user preferences.

123
 Author's personal copy
1684 Scientometrics (2015) 102:16691686

Fig. 8 Collaboration among authors in L2R

Collaboration of the authors producing more than 5 papers

In order to analyse this indicator, we elaborated a graph to represent the level of co-
authoring through a social network of collaboration among the most prolific authors, i.e.
those authors who had presented more than five papers. By eliminating any independent
authors, we obtained a network of 51 authors, see Fig. 8 where the weights associated to
the arrows represents the number of collaborations, i.e., the strength of collaboration. In
this network, a dense subgraph of 26 authors can be identified (on the left-hand side). In
this graph, we can highlight J. Liu that works with a strong network of co-authors and also
those authors from Microsoft Research in Asia, standing out T.J. Liu, Hang Li and T. Qin
in particular. It should be mentioned that the presence of small groups (two groups of two,
one of three, one of four, one of six and one of eight) proves that certain authors work
almost independently.

Conclusions

Ranking is a central problem in many IR applications and given the availability of training
data it has been possible to use automatic technologies to learn a ranking model. Learning
to rank has consequently emerged as a hot research topic in the field of IR. This has
encouraged us to develop a metric study about the scientific output in Learning to Rank.
While this study provides a snapshot of L2R, anyone interested in the L2R problem should
read further on the subject, with the book by Liu (2011) serving as a good starting point
since it covers the major approaches, their foundations and theoretical properties, which
are beyond the scope of this paper. One limitation of the analysis presented in this paper is

123
 Author's personal copy
Scientometrics (2015) 102:16691686 1685

that no semantic information has been considered. Such kind of information could be very
useful to tackle problems as novelty, diversity or originality in the context of the authors
research. By means of this semantic-based approach new dimensions for the research
activities could be discovered but, in order to tackle this objective, it could be necessary the
development of new methods able to extract (automatically) the necessary semantic
information by mining the content of the different papers.
In order to accomplish our study, we processed all the relevant bibliographic records on
the topic under study from the Scopus database between 2000 and 2013. By analyzing the
conceptual and theoretical framework on bibliometric studies of information relating to
information sciences, we were able to discern the fundamental concepts and techniques on
which these were based. These concepts enabled us not only to conduct the bibliometric
analysis but also define the metric indicators so as to evaluate L2R scientific output.
Among the main results of the metric study, the most relevant was the fact that there had
been a significant increase in research into L2R topics over the past 5 years with computer
science being the most explored area of knowledge. The reason for this is the high level of
development and efficiency required by the technologies and trends in this field, both for
scientific progress and the development of computer tools and applications.
The most relevant information sources for each type of reference are the Annual
International ACM SIGIR Conference on Research and Development in Information
Retrieval (conference), Information Retrieval (journal) and European Conference on
Information Retrieval (editorial series). With this analysis, we can corroborate that the
ACM SIGIR Conference establishes a strong frontier of relevance, which is why it con-
stitutes one of the main reference pillars for news of the most recent L2R studies currently
being developed.
Another of the contributions of this work is given by the clustering and characterization
of every L2R-related descriptor. According to this clustering, the main areas of research
which have boomed and stabilised are IR, ranking functions, ranking model, knowledge
management, web searches, learning algorithms, search engines, algorithms, optimization,
machine learning and learning systems. A more detailed analysis enables us to corroborate
that most descriptors which have boomed and stabilised are covered by the authors with the
greatest impact and are identified in the main paper in the most relevant information
sources. Promising incipient descriptors include computational linguistics, transductive
learning, semantic web, document ranking, personalization, crowdsourcing, etc.

Acknowledgments This paper has been supported by the Spanish Ministerio de Ciencia e Innovacion
under projects TIN2011-28538-C02-02 and TIN2013-42741-P.

References

Bornmann, L., de Moya-Anegon, F., & Leydesdorff, L. (2012). The new excellence indicator in the world
report of the SCImago institutions rankings. Journal of Informetrics, 6, 333335.
Braam, R., & Van-Den-Besselaar, P. (2014). Indicators for the dynamics of research organizations: A
biomedical case study. Scientometrics, 99(3), 949971.
Canedo, R., Rodrguez, R., & Montejo, M. (2010). Scopus: The largest database of peer-reviewed scientific
literature available to underdeveloped countries. ACIMED, 21(3). http://www.acimed.sld.cu/index.php/
acimed/article/view/14/45.
Cao, Y., Xu, J., Liu, T.-Y., Li, H., Huang, Y., & Hon, H.-W. (2006). Adapting ranking SVM to document
retrieval. In SIGIR 2006: Proceedings of the 29th annual international ACM SIGIR conference on
research and development in information retrieval, pp. 186193.

123
 Author's personal copy
1686 Scientometrics (2015) 102:16691686

Dong, Y., Huang, C., & Liu, W. (2014). RankCNN: When learning to rank encounters the pseudo preference
feedback. Computer Standards & Interfaces, 36(3), 554562.
EndNote X4: Getting started guide (2010). Chapter 5 Entering a References. United States, Thomson
Reuters.
Joachims, T. (2005). A support vector method for multivariate performance measures. In ICML 2005:
Proceedings of the 22nd annual international conference on machine learning, pp. 377384.
Kohonen, T. (2001). Self-organizing maps (3rd ed.). Berlin: Springer.
Lariviere, V., Sugimoto, C.-R., & Cronin, B. (2012). A bibliometric chronicling of library and information
sciences first hundred years. Journal of the American Society for Information Science and Technology,
63(5), 9971016.
Lin, Y., Lin, H., Xu, K., & Sun, X. (2013). Learning to rank using smoothing methods for language
modeling. Journal of the American Society for Information Science and Technology, 64(4), 818828.
Liu, T.-Y. (2009). Learning to rank for information retrieval. Foundations and Trends in Information
Retrieval, 3(3), 225331.
Liu, T.-Y. (2011). Learning to rank for information retrieval. Berlin, Heidelberg: Springer.
Pritchard, A. (1969). Statistical bibliography or bibliometrics. Journal of Documentation, 25(4), 348359.
ScienceDirect. What does Scopus cover? j SciVerse. (Online). http://www.info.sciverse.com/scopus/scopus-
in-detail/facts. Accessed: 17-dic-2013.
Tan, J., Fu, H.-Z., & Ho, Y.-S. (2014). A bibliometric analysis of research on proteomics in Science Citation
Index Expanded. Scientometrics, 98(2), 14731490.
Torres-Salinas, D., & Jimenez-Contreras, E. (2010). Introduccion y estudio comparativo de los nuevos
indicadores de citacion sobre revistas cientficas en Journal Citation Reports y Scopus. El Profesional
de la Informacion, 19(2), 201207.
Valizadegan, H., Jin, R., Zhang, R., & Mao, J. (2009). Learning to rank by optimizing NDCG measure. In
NIPS 2009: Proceedings of 23rd annual conference on neural information processing systems,
pp. 18831891.
Wu, J., Yang, Z., Lin, Y., Lin, H., Ye, Z., & Xu, K. (2011). Learning to rank using query-level regression. In
SIGIR 2011: Proceedings of the 34th annual international ACM SIGIR conference on research and
development in information retrieval (pp. 10911092). ACM Press.
Xu, J., & Li, H. (2007). Adarank: A boosting algorithm for information retrieval. In SIGIR 2007: Pro-
ceedings of the 30th annual international ACM SIGIR conference on research and development in
information retrieval, pp. 391398.
Xu, J., Liu, T.-Y., Lu, M., Li, H., & Ma, W.-Y. (2008). Directly optimizing evaluation measures in learning
to rank. In SIGIR 2008: Proceedings of the 31st annual international ACM SIGIR conference on
research and development in information retrieval, pp. 107114.
Yan, E., Ding, Y., & Zhu, Q. (2010). Mapping library and information science in China: A coauthorship
network. Scientometrics, 83(1), 115131.
Yeh, J.-Y., Lin, Y.-Y., Ke, H.-R., & Yang, W.-P. (2007). Learning to rank for information retrieval using
genetic programming. In SIGIR 2007: Proceedings of SIGIR 2007 workshop on learning to rank for
information retrieval.

123