Deep Layered Learning in MIR
Anders Elowsson
KTH Royal Institute of Technology
elov@kth.se
ABSTRACT
Deep learning has boosted the performance of many music
information retrieval (MIR) systems in recent years. Yet,
the complex hierarchical arrangement of music makes
end-to-end learning hard for some MIR tasks – a very deep
and structurally flexible processing chain is necessary to
extract high-level features from a spectrogram representa-
tion. Mid-level representations such as tones, pitched on-
sets, chords, and beats are fundamental building blocks of
music. This paper discusses how these can be used as in-
termediate representations in MIR to facilitate deep pro-
cessing that generalizes well: each music concept is pre-
dicted individually in learning modules that are connected
through latent representations in a directed acyclic graph.
It is suggested that this strategy for inference, defined as
deep layered learning (DLL), can help generalization by
(1) – enforcing the validity of intermediate representations
during processing, and by (2) – letting the inferred repre-
sentations establish disentangled structures that support
high-level invariant processing. A background to DLL and
modular music processing is provided, and relevant con-
cepts such as pruning, skip connections, and layered per-
formance supervision are reviewed.
1. INTRODUCTION
Many of the skills necessary for successfully navigating
modern human life are interrelated, but still so divergent
that it is practically unfeasible to learn them all from one
overall objective function. Although it may be possible to
formulate an overarching goal for learning, progress gen-
erally stems from achieving smaller intermediate goals,
gradually expanding a toolbox of concepts and procedures
in a supervised and structured manner.
Music is a fitting environment for studying learning, an
intricate arrangement of overlapping sounds across many
dimensions. Complex sinusoids in the time domain with
frequencies at integer multiples give rise to the pitch per-
cept. Pitches are layered across frequency to form har-
mony, and the pitched sounds are combined with percus-
sion to form complicated rhythmical patterns. How can we
teach machines to excel at music? Consider for example
the complex task of writing and performing a song with
certain perceptual qualities with guidance from only a few
music audio examples. Such a task requires the machine to
be able to represent and understand music at a high level –
something that necessitates many intermediate layers of
non-linear transformations and representations. Many such
relevant intermediate representations are not perceptually
well-defined, and to design these, human intuition only
goes so far – decades of traditional research in MIR gave
steady but rather slow incremental improvements. Deep
learning has become an essential tool for bridging the un-
known territory between input data and targets [26, 48],
well suited for MIR [38]. At the same time, some (often
intermediate) representations of music involving pitch,
harmony, and rhythm, are well-established and fundamen-
tal to musical comprehension. Music intelligence systems
may rely on such representations to partition complex MIR
tasks into several supervised learning problems, thus
breaking down complexity. The learning steps necessary
for untangling the musical structure can be specified in a
directed acyclic graph (DAG), using, for example, deep
learning with musically appropriate shared weights at each
individual step. This strategy for inference, defined here as
deep layered learning (DLL), is starting to become preva-
lent in MIR systems with state-of-the-art performance.
Figure 1 provides a very high-level illustration of the
rich machine learning repertoire that can be used as build-
ing blocks in DLL for MIR. An overview of these learning
strategies, withDeep Learning
examples, is provided in Sections 2.1-3,
while 2.4-5 reviews modular processing and layered learn-
ing in MIR. Section 3 explores relevant concepts of DLL
in MIR, and Section 4 offers closing remarks.
2.3 Layered Classifiers 2.1 Representation Learning
Deep Learning
Cascading y
Classifiers 3. Deep Layered
y
Learning
y Decision y
y1
y y Trees
y
2.2 Transfer
Learning
y1 y2 y1 z1
Classifier
Chains y3 y2 z2 y z
Figure 1. Simplified flowgraphs for learning methodolo-
gies which are building blocks of DLL (Sections 2.1-3).
The input is marked with a black line, processes with ar-
rows (red = processes learned and later applied), represen-
tations with circles (red = learned for other processes), and
annotated targets marked with y or z (red = used as output;
subscripts = different aspects of the same problem). The
depth of each learning step may vary (not specified).
Disentangled Invariance
Pruning
 2. BACKGROUND
2.1 Representation Learning
A common challenge in machine learning is that available
data is not organized in a way that facilitates inference.
Classification will become much easier after the data is
transformed into a new representation. For MIR, some
suitable representations can be defined beforehand from
knowledge about music and auditory perception. Resear-
chers may compute a time-frequency representation (TFR)
of the audio signal, and, as will be expanded upon in this
paper, extract tonal or temporal information as intermedi-
ate representations. Commonly however, it is not known
how to optimally represent the data at intermediate levels,
so suitable representations (and transformations) must be
discovered automatically with representation learning [3].
Oftentimes, the best such latent representations (i.e., in-
ferred and not observed) are derived after several layers of
transformations. Depth generally leads to richer represen-
tations, yielding better results for MIR [38]. As suggested
by Bengio et al. [3] (p. 1800), the “hierarchical organiza-
tion of explanatory factors” exploited in deep learning pro-
motes “progressively more abstract features.”
Learning can be performed in an unsupervised way, by
identifying compact representations from which the train-
ing set can be recreated [37, 81]. In MIR, this has been
done for genre recognition [33] and piano transcription
[54] with a deep belief network (DBN). Supervised deep
learning systems can also be conceived of as performing
representation learning during processing in hidden layers
[26]. The ability to automatically form such intermediate
representations is one of the powers of deep learning [48].
In MIR, some of the earliest deep learning systems recog-
nized genres [49] instruments [40], and chords [39].
2.2 Transfer Learning
To successfully tackle new problems, previous knowledge
from similar problems can be used [62]. In machine learn-
ing, this induction from knowledge is called transfer learn-
ing [57], which can be used to manage a task when anno-
tated data is scarce. A typical example is to recognize face
emotions [55] from latent representations extracted with a
deep learning system trained on a big set of images [46].
In MIR, several successful systems have first been
trained on a large dataset with annotated tags, such as the
million songs dataset, and then applied to smaller datasets
for, e.g., genre classification [11, 32, 33, 79]. Transfer
learning has also been used for predicting musical prefer-
ences [50] or playlist generation [10].
2.3 Architectures with Multiple Layered Classifiers
Many machine learning architectures have been proposed
over the years involving multiple, layered classifiers, i.e.,
ensemble learning with a dynamic structure [26]. Early ex-
amples focused on dividing the input into nested sub-re-
gions. Such algorithms, based on decision trees, include
CART [9], ID3 [65], and MARS [24]. The strategy was
extended with neural networks (NNs) to perform the clas-
sification split [30], and separate gating networks for hier-
archical mixtures of experts [41] to smoothly divide the
input space. The structure of such architectures can be de-
termined with genetic algorithms [2, 34]. Although these
systems have been referred to as modular NNs, they do not
use modularity in the same way as DLL for MIR, described
in Section 3.1. With DLL, a task is solved by subjecting all
examples to the same (deep) learning modules, but the
modules can also be used for other tasks.
A way to increase computation speed for classification
is to use cascading classifiers [1]. With this methodology,
weak classifiers are applied in a linear sequence to succes-
sively reject a larger and larger portion of the input space
until (ideally) only true examples of the input space re-
main. A cascade of classifiers was used for face recogni-
tion back in 2001 [82], training small decision tree classi-
fiers with AdaBoost. In visual media, they can be used for
pedestrian detection [13, 83]. Another example is Google's
system for address number transcription [27], where one
classifier locates addresses and another transcribes them
(as clarified in [26]). Cascades have also been applied in
MIR, to utilize the sparse distribution of pitched onsets for
polyphonic transcription [80].
A related methodology, classifier chains [68], represent
an automated method for transforming a multi-label clas-
sification task into a chain of binary ones. Labels are clas-
sified in consecutive stages, with the classification result
at one stage supplied together with the original features for
the next stage (with a new set of labels). The benefits can
be understood from the effect of depth on the structure of
the learning. The labels predicted at intermediate steps can
be conceived of as hidden layers in a multilayer perceptron,
thereby functioning as a universal approximator [66]. Alt-
hough they emulate these end-to-end networks, intermedi-
ate layers are trained with supervised learning. As will be
discussed, DLL can be understood as a way to reframe
complex tasks as multi-label classification problems, where
labels in early layers preimpose the structural representa-
tion of the data. Classifier chains have been used in several
systems for multi-label classification in MIR [31, 67, 68].
Image segmentation and object classification has re-
cently seen many models with intricate learning architec-
tures [90], including: the part-based R-CNN [89], convo-
lutional neural networks (CNNs) combining bottom-up
and top-down [87] or locally shifting (through a recursive
neural network, RNN) [72] attention mechanisms, and the
CNN tree [84] using several CNNs in a classifier chain.
Notably, end-to-end DAG networks can use intermediate
targets and average their gradients with those of the overall
target during backpropagation, e.g., to guide the structural
segmentation of tissue images during cancer classification
[21]. In robotics, neuro-controllers have been trained for
several subtasks to perform a target task better; described,
among other things, as modular decomposition [12, 14, 77].
2.4 Modular Music Processing
Mid-level representations are very useful in MIR [5].
There is an abundance of such relevant representations
covering different dimensions of music. Across the tem-
poral dimension, researchers may predict or utilize, e.g.,
meter, tempo, beats, and downbeats; across frequency,
e.g., fundamental frequencies (f0s), etc.
 Many MIR systems have been designed in a modular
fashion, utilizing musical representations to express struc-
tural properties of the data at intermediate levels. Frame-
wise f0 activations are commonly used for pitched onset
detection, but they can also be used for beat tracking [17]
and for computing a refined chromagram for chord detec-
tion [52]. Various perceptual features of music such as the
“speed,” rhythmic clarity, and harmonic complexity, can
be predictive of music mood [23]. The computed speed can
be used together with periodicity estimates and tempo es-
timates for beat tracking [17]. Beat estimates can be useful
when trying to detect chords and chord changes [52] (to-
gether with a prior key estimation [88]), or downbeat po-
sitions [15, 45]. The opposite is also true; chord change
information can be used when computing beat and down-
beat positions [28, 58-60, 86]. Many systems have used
harmonic-percussive source separation (HPSS) as an early
processing step [20, 25, 56, 69, 70]. This interdependence
of musical concepts is probably why several MIR
toolboxes [16, 47, 53] have a modular design. At a higher
level of processing, Schenkerian analysis [22], extended
with probabilistic computational models [43], can be used
to understand the hierarchical structure of compositions.
Modular music processing can be found when studying
music impairments in brain-damaged patients [61], lead-
ing researchers to suggest various modules for the tonal and
temporal organization. A biologically-inspired hierarchical
unsupervised learning model has also been proposed [63].
2.5 MIR implementations using layered learning
Modular music processing facilitated by suitable mid-level
representations, as described in Section 2.4, can easily be
extended to modular learning systems by introducing sev-
eral supervised learning steps. Often in MIR, one type of
annotations can be restructured to facilitate learning of
several perceptually relevant aspects for a task. Two such
tasks are shown in Figure 2 and reviewed in this Section.
1. (4) Beat activation
2. (5) Tempo
Beats
3. (6) Beat sequence modeling
Pitch
2. Onset 3. Offset
1. Frames
Time
Figure 2. Tasks that can be learned in sequence from one
set of annotations. For polyphonic transcription (blue),
systems can first learn to perform framewise predictions,
then onset detection and offset detection. For beat track-
ing (green), systems can compute activations at beats, use
these activations to infer tempo, and finally model the se-
quence of beat from the previous two representations.
For polyphonic transcription, onset and offset annotations
can be used to generate annotations for framewise predic-
tions. For beat tracking, the annotated beat positions can
be used for predicting a beat activation, the tempo, and to
find the most likely sequence of beats. As indicated in the
Figure, polyphonic transcription (1-3) can be used as a pre-
processing to beat tracking (4-6). For example, detected
notes can be supplied to a beat activation network.
2.5.1 Polyphonic Transcription
In polyphonic transcription, framewise f0 estimation (or
partial estimation) and note transcription have been sepa-
rated into more than one learning algorithm in many sys-
tems. Marolt [51] used networks of oscillators for partial
tracking, and separate networks for detecting note pitches
and repeated notes. Another early implementation [64]
used a support vector machine (SVM) for the frame level
classification, and a hidden Markov model with state pri-
ors determined from the training set, to identify onsets and
offsets across time. A similar methodology was applied by
Nam et al. [54], using a DBN to extract features for classi-
fication. A musical model was then applied on top of the
framewise predictions in a separate study [7], using an
RNN combined with a restricted Boltzmann machine.
A recent system tracks pitch onsets and uses these for
framewise estimates with CNNs [35]. Another recent sys-
tem uses six interconnected networks, each trained sepa-
rately with supervised learning [18]. Cascading classifiers
are used for framewise f0 estimation. Regions of connected
f0s (corresponding to tones) are identified with tone-shift-
invariant onset and offset detection. Finally, tentative
notes are extracted and classified as true or false with the
last network.
In some implementations, the first step has been per-
formed with unsupervised learning of latent variable mod-
els (e.g., non-negative matrix factorization). Supervised
classifiers have then been applied to refine the frame-level
classification [71] or for the note detection step [78, 85].
2.5.2 Beat Tracking
Another MIR task where separate supervised learning
steps can be useful is rhythm tracking (e.g., tempo estima-
tion, beat tracking, and downbeat tracking). Some systems
use one network for computing a time-varying activation
curve, and then, to find the best sequence of beats, tune a
few parameters for Viterbi decoding [44] or for a dynamic
Bayesian network [6]. For downbeat tracking, the learning
step for detecting downbeats has used beat synchronous
input features, with the beats derived from a previous RNN
learning layer [45]. In [20], linear regression is used to es-
timate the speed of the music and logistic regression is
used to pick the tempo between several tentative tempi.
Some of these methods are combined in [17], a system
described as using “layered learning.” The most salient pe-
riodicity is estimated by the first network and then used to
subsample periodicity invariant input features for a second
network computing a beat activation. The speed is esti-
mated with a third network (regression) from global fea-
tures. Finally, the tempo is computed in a fourth network,
using previously computed representations as input.
 3. DEEP LAYERED LEARNING IN MIR
As presented in Section 2.5, many recent systems for MIR
use multiple learning steps, a few with a considerable
depth in at least one learning step. The architectures and
flow of representations can be described by a DAG, with
learning steps trained in a sequence, starting with the most
low-level. These designs are not generally motivated by
the same circumstances as when multiple steps are used
for transfer learning. The purpose is not to transfer proce-
dures acquired from one dataset to another, but to use sev-
eral subtasks to distill knowledge of structural properties
within intermediate representations. This is a method for
approaching an overarching task with multiple annota-
tions in a way that promotes generalization (see Sections
3.2 and 4) and, also, modularity in terms of the overall ar-
chitecture (Section 3.1). As stated in the introduction, the
approach will be defined as deep layered learning (DLL),
referencing a desirable depth of each learning step (i.e.,
deep learning), but also a depth of the layering of the
learning steps.1 The term was originally used in a headline
for a shorter abstract on polyphonic transcription [19],
and, as mentioned in Section 2.5, a system for beat track-
ing has been described as using “layered learning” [17].
There are several concepts, evident in [17, 18] and ap-
plicable to MIR, that are important to understand to build
successful DLL systems (here highlighted in bold with the
corresponding subsection in parenthesis). The modularity
of music (see Section 2.4) enables the architecture to be
divided into learning modules (3.1), which are the build-
ing blocks of DLL. Each module performs a mapping be-
tween latent representations (3.4) (together with the out-
put) of previous learning modules and an annotated target.
The validity (3.2) of the target (with respect to the over-
arching task) for these intermediate learning module is im-
portant; learning modules should only be employed when
the module will transform the input in a direction that fa-
cilitates prediction of the overall target. If this is not the
case, a (deeper) learning architecture must instead be de-
vised within-module, reaching an even higher-level target.
It may often be unfeasible to devise intermediate targets
comprising all relevant information for the final predic-
tion. Skip connections (3.5) from earlier points can
therefore be employed. Such an overall strategy is espe-
cially useful if the purpose of the earlier learning module
was to infer “disentangled structures (3.3)” – structural
components of the music facilitating invariant processing.
Such invariances are hard to encode for with end-to-end
learning. Computation time can be reduced by pruning
(3.6) the search space before the next module. The system
can also benefit from applying layered performance su-
pervision (3.7), identifying weak links.
3.1 Learning Modules
The cornerstone of DLL is the learning module, in which
the relationship between input data and a target is inferred.
The modules are arranged in a DAG, producing higher-
level representations at each hierarchical layer. The input
data at lower modules for MIR can be a TFR, whereas the
subsequent modules use representations from lower-level
learning modules. The predictive power of deep learning
is best utilized if modules perform well-defined tasks that
require several layers of transformation within each mod-
ule. Learning should only be broken down into modules if
the module meaningfully untangles the musical structure.
Figure 3 outlines important representations for DLL in
MIR, drawing on the examples in Sections 2.4-5. The ar-
rows of the model indicate how learning modules can be
connected, starting from a TFR of the music audio. All rep-
resentations have a rather clear perceptual and music-the-
oretical interpretation (the importance of which is dis-
cussed in Section 3.2) and have been computed with ma-
chine learning in the past. Skip connections can (and
should) be used, but these are not drawn. The model fo-
cuses on source perception at lower layers, extracting har-
monic and percussive audio, as well as framewise f0s (the
tonal “source” of harmonics across the spectrum). Onsets
and pitched notes can be estimated from these sources, or
directly from the TFR. Estimated pitches facilitate key and
chord detection, rhythmical processing (e.g. beats and
tempo), and can be used as a precursor to pitched instru-
ment recognition. Percussive audio and onsets can instead
be used to guide the recognition of percussive instruments.
As indicated, there is a strong interaction between repre-
sentations covering the rhythmical organization of music.
Higher-level music analysis can use a transcription of the
music, which could also be used together with perceptual
features for estimating mood. Genre is depicted as a rather
high-level concept, contrary to the shallow architectures of
many successful genre recognition systems [4, 76]. This is
inspired by research suggesting that such systems rely on
“irrelevant confounding factors” for classification [73-75].
Music Percussive
Framewise f0s Harmonic TFR audio
Pitched notes audio
Onsets
Key Beats
det. Percept.
Tempo Downbeats features
Chord
Meter Periodicities
det. Mood
Music Analysis
Transcription Instr. Rec.
(perc.)
Instr. rec. (pitched) Genre
Figure 3. Important representations of music audio, and
their hierarchical organization. There is a strong interac-
tion between all rhythm representations (green rounded
rectangle, RR), pitch-related representations (blue RR),
and the vertical arrangement of pitch (purple RR). Esti-
mates that generally are global are marked with yellow,
and the output representations of HPSS are marked with
red. Structural analysis performed on musical scores is
marked in black (a deep multi-layered task).

1
I.e. both “deep-layered learning” and “deep, layered learning”.
3.1.1 Parallels to Modular Programming
In computer programming, code modularity is an im-
portant design principle, allowing well-defined tasks to be
abstracted into higher-level modules. This gives develop-
ers benefits with clear parallels to those offered by DLL
and learning modules.
1) Modular code is reusable since many modules can be
used for other projects. In DLL, each learning module can
be used by all succeeding tasks in the chain.
2) When well-defined subtasks are divided into mod-
ules, it becomes easier to perform bug fixing, performance
improvements, and other under-the-hood changes for each
module separately. With DLL, it becomes possible to iden-
tify modules that are underperforming, through layered
performance supervision (Section 3.7). Performance can
be maximized for each module, for instance, by experi-
menting with different weight-sharing mechanisms.
3) Modular code can be developed by large teams, as
team members can focus on separate code parts that inter-
act only through pre-defined interfaces. Modular DLL sys-
tems can be developed by several research groups, as mod-
ules can interact through pre-defined representations.
3.2 Enforced Validity
Deep layered learning can be used to force the system into
using certain intermediate representations of the data that
the researcher knows are important – to enforce validity.
This can be useful for tasks composed of rather independ-
ent subtasks [29], and also prevent overfitting, as the sys-
tem cannot to the same extent make predictions by infer-
ring complex and irrelevant relationships between the in-
put data and the target. Irrelevant in this context refers to
those relationships that will not generalize outside of the
training set, as discussed in [73, 75]. For example, down-
beats may be inferred from pitched onsets, beats [45], and
chord changes [58-60, 86]. Such representations were pre-
sented in Figure 3. If these representations are computed
with supervised learning, and then supplied to the down-
beat tracker in a relevant form, it may generalize better
than if it computes downbeats directly from a TFR with
end-to-end deep learning. The system will not to the same
extent rely on assumptions about how, e.g., the rhythmical
structure covaries with vocal characteristics, instrumenta-
tion, or timbre. Such relationships may exist only in the
training set and are therefore less “valid” for the task. For
tasks that to a larger extent are defined from perception,
DLL could be contrived as enforcing perceptual validity.
There is a close connection between using layered
learning modules with high validity and using weight shar-
ing with convolutional neural networks (CNNs). In MIR,
weight sharing can generally be used to encode for invari-
ant properties of music. They thereby enforce restrictions
on the computations to improve generalization and allow
for greater depth. Deep layered learning systems can use
weight sharing within modules, but also use intermediate
targets to enforce restrictions on the representations and
structures that are used.
3.3 Disentangled Structures
It is important to account for invariances in music when
developing a MIR system. Sometimes it is easier to encode
for invariant properties within musical structures when not
relying on an end-to-end learning architecture. The previ-
ous learning module can infer structural components of the
music as shown in Figure 4, which then are incorporated
into the processing chain of subsequent modules.
For example, listeners can track tones across both pitch
and time during vibrato, so that fluctuating f0s are associ-
ated with the same tone stream. A tone and its features can
therefore be traced across both pitch and time [18]. Two
tones with the same envelope (blue in Figure 4) can
thereby be represented by the same envelope values in the
next step of processing, regardless of their variation in
pitch across time (red in Figure 4). By collecting infor-
mation from the note attack (onsets), the sustained part
(ridge across the length of the note), and decay (offsets), it
also becomes possible to represent the note as a vector of
fixed size, independent of note length. Another example is
downbeat tracking (bottom pane in Figure 4), where initial
periodicity trackers on a finer (faster) level can make the
downbeat tracker's input features invariant with regards to
phase, tempo, and tempo variations in [15, 45].
f0-ridge to note envelope
Time
Beats to downbeats
Figure 4. Disentangled structures for pitched note tracking
and downbeat estimation. The extracted representation for
future processing is invariant with regards to variations
that are of less perceptual relevance.
3.4 Latent Representations
The latent representations in the last hidden layer of each
learning module consist of various features useful for pre-
dicting the target of the module. When computing a beat
activation (see Figure 2) with a feedforward network for a
single time frame using input features from surrounding
time frames, one neuron in the hidden layer may activate
if a kick drum is present close to the processed frame, an-
other neuron for the snare drum, and some neurons may
activate if periodical musical accents intersect the present
frame. Many neurons will also activate to attributes for
which no clear musical terminology exist. It is reasonable
to assume that these hidden layer activations can be use-
ful, in addition to the actual output prediction, for predict-
ing the target of the next learning module – after all, a ma-
jor point of DLL is to create high-level representations for
prediction. Therefore, for most steps of a DLL system, it
should be beneficial to use representations from the last
hidden layer of the earlier learning module when predict-
ing a new higher-level target. In this respect, DLL is re-
lated to the layerwise training of DBNs.
3.5 Skip Connections
Skip connections were introduced more than 20 years ago
for feedforward networks [42]. They allow lower-level
representation to skip layers of processing in the network.
Recently, skip layers or “residual layers” have also been
proposed for deep learning (in DAGs) with CNNs [36],
with a gated variant used in deep “highway networks.”
In DLL systems, learning modules can receive input
from previous modules or from the TFR via “skip connec-
tions” to cover for aspects of the data not predicted for by
later modules. As each module is trained separately, there
will be no gradient flowing through such connections. The
reason for using skip connections from earlier learning
modules or raw input is rather obvious. As discussed in
Section 3.2, the layered learning can be used to enforce the
validity of mid-level representations. Depending on the
mid-level target, these representations may not have fully
captured all aspects relevant to the overall target. There-
fore, it can be useful to provide representations from ear-
lier layers or the original input when making predictions at
the next layer. This structure can be especially motivated
if the mid-level processing is not merely aimed at extract-
ing relevant representations, but also to extract disentan-
gled structures as outli[35]ned in Section 3.3.
3.6 Pruning
Deep layered learning can be used to prune the search
space during run-time with a functionality inspired by cas-
cading classifiers (Section 2.3). Many tasks consist of de-
tecting local elements from a large input space. For MIR,
researchers may be interested in finding music concepts
such as tones, beats, instrumental parts, or onsets, that are
sparsely distributed across the whole input space. In the
case of polyphonic transcription, pruning can occur in sev-
eral stages [18, 80]. For example, if the spectrogram mag-
nitude is low at harmonically related frequency bins and
the spectrum lacks the ridge typically associated with par-
tials, it is unlikely that the bin corresponds to an f0. With a
very basic processing scheme, it is therefore possible to
prune the search space significantly. More advanced (and
computationally expensive) models can then be applied to
the bins where there is a higher probability that the bin fre-
quency corresponds to an f0. Given high-resolution
processing of 5 cents/pitch-bin across a range of 80
pitches, and with 10 evaluated f0s per frame, the search
space can be reduced to around 6 ‰ of its original size,
10 × 0.05⁄80 = 0.00625. After pruning across pitch, an
onset activation can be computed at activated f0s across
time. Finally, only the most likely onset positions may be
processed further to evaluate if they are a correct onset of
a note or not. Given a hop size of 0.01 seconds (s) and 20
evaluated notes/s, the final search space for classification
can be reduced to almost 1/10000th of its initial size,
0.05×20×0.01/80 = 0.000125.
The described method for pruning differs from that of
cascading classifiers in that the input is processed to prune
across several dimensions (and modulations of annotations
as shown in Figure 2) iteratively. The resolution can be up-
sampled at each pruning stage with parabolic interpol-
ation; e.g., for f0 estimation [18] or for subsampling of in-
put vectors in beat tracking [17].
3.7 Layered Performance Supervision
It becomes easier to identify strengths and weaknesses in
a system when various subtasks can be evaluated sepa-
rately. A system with three supervised learning steps may,
for example, first compute framewise f0s, use these to com-
pute pitched notes, and finally perform chord tracking
from note features. As annotations have already been cre-
ated for each supervised learning step, the performance of
each subtask can be evaluated individually, assessing
whether the performance is satisfactory, given the com-
plexity of the subtask. Such layered performance supervi-
sion makes it possible to identify parts of the overall sys-
tem design that need to be improved. As outlined in Sec-
tion 2.6, when applying cascaded classifiers, performance
can be tuned to keep recall high during initial steps. The
same is generally true for detection tasks in DLL. For the
above example of chord recognition, recall of the first step
could be defined as the proportion of annotated onsets that
are close to any activations of framewise f0s, and in the
second step depend on whether the played chord tones are
detected during an annotated chord. Relevant evaluation
measures may take a bit of engineering to get right.
4. CLOSING REMARKS
In MIR, there are many tasks for which it is very hard to
produce a detailed formula for the solution, while it still is
possible to surmise predictive mid-level features. Deep
layered learning has been discussed as a design principle
in MIR for balancing the power of deep learning to infer
complex relationships with the foresight of a researcher
acting like a teacher. In this regard, there is a parallel to the
role of a researcher for using prior knowledge when de-
signing the weight sharing used in CNNs.
When the system is trained with input from the earlier
learning module and skip connections from the input spec-
trogram, it may become over-reliant on deeper representa-
tions, as these have already been fitted to the training set.
The system can then be regulated by distorting (e.g., equal-
ization, compression, added noise) the musical excerpts
randomly before training each learning module. However,
it should be noted that some over-reliance on deeper rep-
resentations may be desirable, under the assumption that
the enforced validity of these representations promotes
generalization to test sets with different characteristics.
As described in Section 2.3, end-to-end DAG networks
can use intermediate targets, guiding the learning (see, e.g.,
[21, 35]). If it is not necessary to process disentangled mu-
sical structures such as notes and beats (see Section 3.3),
intermediate targets in an end-to-end DAG network can in-
stead be used to enforce validity. In this context, it could
be interesting to explore weighting schemes when combin-
ing gradients at hidden layers during backpropagation
(other than taking the mean). With high weights for the
gradients of intermediate targets, the DAG network will
function more like the DLL systems described in this paper
(or as the DBN used for chord recognition in [8]), with low
weights it will perform more conventional end-to-end
learning. For example, the desired contribution from chord
changes to downbeat tracking could be tuned by the re-
searcher, discovered by parameter sweeps, or even learned
under gradient descent in more advanced training setups.
 5. ACKNOWLEDGMENT
Thanks to Pawel Herman for many relevant comments.
6. REFERENCES
[1] E. Alpaydin and C. Kaynak, "Cascading classifiers,"
Kybernetika, vol. 34, no. 4, pp. 369-374, 1998.
[2] F. Azam, "Biologically inspired modular neural
networks," Ph.D dissertation, Virginia Tech, 2000.
[3] Y. Bengio, A. Courville, and P. Vincent,
"Representation learning: A review and new
perspectives," IEEE transactions on pattern
analysis and machine intelligence, vol. 35, no. 8, pp.
1798-1828, 2013.
[4] J. Bergstra, N. Casagrande, D. Erhan, D. Eck, and
B. Kégl, "Aggregate features and adaboost for
music classification," Machine learning, vol. 65, no.
2-3, pp. 473-484, 2006.
[5] S. Böck, "Event Detection in Musical Audio," Ph.D
dissertation, Johannes Kepler University Linz, Linz
Austria, 2016.
[6] S. Böck, F. Krebs, and G. Widmer, "Joint Beat and
Downbeat Tracking with Recurrent Neural
Networks," in ISMIR, 2016, pp. 255-261.
[7] N. Boulanger-Lewandowski, Y. Bengio, and P.
Vincent, "Modeling temporal dependencies in high-
dimensional sequences: application to polyphonic
music generation and transcription," in Proc. of the
29th Int. Conf. on Machine Learning, 2012, pp.
1881-1888: Omnipress.
[8] N. Boulanger-Lewandowski, Y. Bengio, and P.
Vincent, "Audio Chord Recognition with Recurrent
Neural Networks," in ISMIR, 2013, pp. 335-340.
[9] L. Breiman, "Classification and regression trees,"
1984.
[10] K. Choi, G. Fazekas, and M. Sandler, "Towards
playlist generation algorithms using rnns trained on
within-track transitions," arXiv preprint
arXiv:1606.02096, 2016.
[11] K. Choi, G. Fazekas, M. Sandler, and K. Cho,
"Transfer learning for music classification and
regression tasks," presented at the ISMIR, 2017.
[12] S. Doncieux and J.-B. Mouret, "Beyond black-box
optimization: a review of selective pressures for
evolutionary robotics," Evolutionary Intelligence,
vol. 7, no. 2, pp. 71-93, 2014.
[13] J. Dong, J. Ge, and Y. Luo, "Nighttime pedestrian
detection with near infrared using cascaded
classifiers," in IEEE Int. Conf. on Image Processing,
2007, vol. 6, pp. VI-185-VI-188: IEEE.
[14] M. Duarte, S. Oliveira, and A. L. Christensen,
"Hierarchical evolution of robotic controllers for
complex tasks," in IEEE Int. Conf. on Development
and Learning and Epigenetic Robotics (ICDL)
2012, pp. 1-6: IEEE.
[15] S. Durand, J. P. Bello, B. David, and G. Richard,
"Downbeat tracking with multiple features and deep
neural networks," in IEEE Int. Conf. on Acoustics,
Speech and Signal Processing (ICASSP), 2015, pp.
409-413: IEEE.
[16] T. Eerola and P. Toiviainen, "MIR In Matlab: The
MIDI Toolbox," in ISMIR, 2004.
[17] A. Elowsson, "Beat tracking with a cepstroid
invariant neural network," in ISMIR, 2016, pp. 351-
357.
[18] A. Elowsson, "Polyphonic Pitch Tracking with
Deep Layered Learning," arXiv preprint
arXiv:1804.02918, 2018.
[19] A. Elowsson and A. Friberg, "Polyphonic
transcription with deep layered learning," presented
at the MIREX Multiple Fundamental Frequency
Estimation & Tracking, 2014.
[20] A. Elowsson and A. Friberg, "Modeling the
perception of tempo," The Journal of the Acoustical
Society of America, vol. 137, no. 6, pp. 3163-3177,
2015.
[21] G. Flood, "Deep learning with a dag structure for
segmentation and classification of prostate cancer,"
M. theses, Lund University, 2016.
[22] A. Forte, "Schenker's conception of musical
structure," Journal of Music Theory, vol. 3, no. 1,
pp. 1-30, 1959.
[23] A. Friberg, E. Schoonderwaldt, A. Hedblad, M.
Fabiani, and A. Elowsson, "Using listener-based
perceptual features as intermediate representations
in music information retrieval," Journal of the Ac.
Soc. of Am. (JASA), vol. 136, no. 4, pp. 1951-1963,
2014.
[24] J. H. Friedman, "Multivariate adaptive regression
splines," The annals of statistics, pp. 1-67, 1991.
[25] A. Gkiokas, V. Katsouros, G. Carayannis, and T.
Stajylakis, "Music tempo estimation and beat
tracking by applying source separation and metrical
relations," in IEEE Int. Conf. on Ac., Speech and
Signal Proc. (ICASSP), 2012, pp. 421-424: IEEE.
[26] I. Goodfellow, Y. Bengio, A. Courville, and Y.
Bengio, Deep learning. MIT press Cambridge,
2016.
[27] I. J. Goodfellow, Y. Bulatov, J. Ibarz, S. Arnoud,
and V. Shet, "Multi-digit number recognition from
street view imagery using deep convolutional neural
networks," presented at the Int. Conf. on Learning
Representations, 2014.
[28] M. Goto and Y. Muraoka, "Real-time rhythm
tracking for drumless audio signals–chord change
detection for musical decisions," in Working Notes
of the IJCAI-97 Workshop on Computational
Auditory Scene Analysis, 1997, pp. 135-144.
[29] Ç. Gülçehre and Y. Bengio, "Knowledge matters:
Importance of prior information for optimization,"
 The Journal of Machine Learning Research, vol. 17,
no. 1, pp. 226-257, 2016.
[30] H. Guo and S. B. Gelfand, "Classification trees with
neural network feature extraction," IEEE
Transactions on Neural Networks, vol. 3, no. 6, pp.
923-933, 1992.
[31] M. Haggblade, Y. Hong, and K. Kao, "Music genre
classification."
[32] P. Hamel, M. Davies, K. Yoshii, and M. Goto,
"Transfer learning in MIR: Sharing learned latent
representations for music audio classification and
similarity," presented at the ISMIR, 2013.
[33] P. Hamel and D. Eck, "Learning Features from
Music Audio with Deep Belief Networks," in
ISMIR, 2010, vol. 10, pp. 339-344: Utrecht, The
Netherlands.
[34] B. L. Happel and J. M. Murre, "Design and
evolution of modular neural network architectures,"
Neural networks, vol. 7, no. 6-7, pp. 985-1004,
1994.
[35] C. Hawthorne et al., "Onsets and Frames: Dual-
Objective Piano Transcription," arXiv preprint
arXiv:1710.11153, 2017.
[36] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual
learning for image recognition," in Proceedings of
the IEEE conference on computer vision and pattern
recognition, 2016, pp. 770-778.
[37] G. E. Hinton, S. Osindero, and Y.-W. Teh, "A fast
learning algorithm for deep belief nets," Neural
computation, vol. 18, no. 7, pp. 1527-1554, 2006.
[38] E. J. Humphrey, J. P. Bello, and Y. LeCun, "Moving
Beyond Feature Design: Deep Architectures and
Automatic Feature Learning in Music Informatics,"
in ISMIR, 2012, pp. 403-408.
[39] E. J. Humphrey, T. Cho, and J. P. Bello, "Learning
a robust tonnetz-space transform for automatic
chord recognition," in Acoustics, Speech and Signal
Processing (ICASSP), 2012 IEEE International
Conference on, 2012, pp. 453-456: IEEE.
[40] E. J. Humphrey, A. P. Glennon, and J. P. Bello,
"Non-linear semantic embedding for organizing
large instrument sample libraries," in Int. Conf. on
Machine Learning and Applications (ICMLA),
2011, vol. 2, pp. 142-147: IEEE.
[41] M. I. Jordan and R. A. Jacobs, "Hierarchical
mixtures of experts and the EM algorithm," Neural
computation, vol. 6, no. 2, pp. 181-214, 1994.
[42] B. L. Kalman and S. C. Kwasny, "High performance
training of feedforward and simple recurrent
networks," Neurocomputing, vol. 14, no. 1, pp. 63-
83, 1997.
[43] P. B. Kirlin, "A probabilistic model of hierarchical
music analysis," Ph.D dissertation, University of
Massachusetts Amherst, 2014.
[44] F. Korzeniowski, S. Böck, and G. Widmer,
"Probabilistic Extraction of Beat Positions from a
Beat Activation Function," in ISMIR, 2014, pp. 513-
518.
[45] F. Krebs, S. Böck, M. Dorfer, and G. Widmer,
"Downbeat Tracking Using Beat Synchronous
Features with Recurrent Neural Networks," in
ISMIR, 2016, pp. 129-135.
[46] A. Krizhevsky, I. Sutskever, and G. E. Hinton,
"Imagenet classification with deep convolutional
neural networks," in Advances in neural information
processing systems, 2012, pp. 1097-1105.
[47] O. Lartillot, P. Toiviainen, and T. Eerola, "A matlab
toolbox for music information retrieval," in Data
analysis, machine learning and applications:
Springer, 2008, pp. 261-268.
[48] Y. LeCun, Y. Bengio, and G. Hinton, "Deep
learning," nature, vol. 521, no. 7553, p. 436, 2015.
[49] T. L. Li, A. B. Chan, and A. Chun, "Automatic
musical pattern feature extraction using
convolutional neural network," in Proc. Int. Conf.
Data Mining and Applications, 2010.
[50] D. Liang, M. Zhan, and D. P. Ellis, "Content-Aware
Collaborative Music Recommendation Using Pre-
trained Neural Networks," in ISMIR, 2015, pp. 295-
301.
[51] M. Marolt, "A connectionist approach to automatic
transcription of polyphonic piano music," IEEE
Transactions on Multimedia, vol. 6, no. 3, pp. 439-
449, 2004.
[52] M. Mauch, "Automatic Chord Transcription from
Audio Using Computational Models of Musical
Context," Ph.D dissertation, 2010.
[53] B. McFee et al., "librosa: Audio and music signal
analysis in python," in Proceedings of the 14th
python in science conference, 2015, pp. 18-25.
[54] J. Nam, J. Ngiam, H. Lee, and M. Slaney, "A
Classification-Based Polyphonic Piano
Transcription Approach Using Learned Feature
Representations," in ISMIR, 2011, pp. 175-180.
[55] H.-W. Ng, V. D. Nguyen, V. Vonikakis, and S.
Winkler, "Deep learning for emotion recognition on
small datasets using transfer learning," in Proc. of
the 2015 ACM int. conf. on multimodal interaction,
2015, pp. 443-449.
[56] N. Ono et al., "Harmonic and percussive sound
separation and its application to MIR-related tasks,"
in Advances in music information retrieval:
Springer, 2010, pp. 213-236.
[57] S. J. Pan and Q. Yang, "A survey on transfer
learning," IEEE Transactions on knowledge and
data engineering, vol. 22, no. 10, pp. 1345-1359,
2010.
[58] H. Papadopoulos and G. Peeters, "Simultaneous
estimation of chord progression and downbeats
 from an audio file," in IEEE Int. Conf. on Acoustics,
Speech and Signal Processing, (ICASSP), 2008, pp.
121-124: IEEE.
[59] H. Papadopoulos and G. Peeters, "Joint estimation
of chords and downbeats from an audio signal,"
IEEE Transactions on Audio, Speech, and
Language Processing, vol. 19, no. 1, pp. 138-152,
2011.
[60] G. Peeters and H. Papadopoulos, "Simultaneous
beat and downbeat-tracking using a probabilistic
framework: Theory and large-scale evaluation,"
IEEE Transactions on Audio, Speech, and
Language Processing, vol. 19, no. 6, pp. 1754-1769,
2011.
[61] I. Peretz and M. Coltheart, "Modularity of music
processing," Nature neuroscience, vol. 6, no. 7, p.
688, 2003.
[62] D. N. Perkins and G. Salomon, "Transfer of
learning," International encyclopedia of education,
vol. 2, pp. 6452-6457, 1992.
[63] M. Pesek, A. Leonardis, and M. Marolt, "A
Compositional Hierarchical Model for Music
Information Retrieval," in ISMIR, 2014, pp. 131-
136.
[64] G. E. Poliner and D. P. Ellis, "A discriminative
model for polyphonic piano transcription,"
EURASIP Journal on Advances in Signal
Processing, vol. 2007, no. 1, 2006.
[65] J. R. Quinlan, "Induction of decision trees,"
Machine learning, vol. 1, no. 1, pp. 81-106, 1986.
[66] J. Read and J. Hollmén, "A deep interpretation of
classifier chains," in Int. Symp. on Intelligent Data
Analysis, 2014, pp. 251-262: Springer.
[67] J. Read, L. Martino, and D. Luengo, "Efficient
Monte Carlo optimization for multi-label classifier
chains," in IEEE Int. Conf. on Acoustics, Speech and
Signal Processing (ICASSP), 2013, pp. 3457-3461:
IEEE.
[68] J. Read, B. Pfahringer, G. Holmes, and E. Frank,
"Classifier chains for multi-label classification," in
Joint European Conference on Machine Learning
and Knowledge Discovery in Databases, 2009, pp.
254-269: Springer.
[69] H. Rump, S. Miyabe, E. Tsunoo, N. Ono, and S.
Sagayama, "Autoregressive MFCC Models for
Genre Classification Improved by Harmonic-
percussion Separation," in ISMIR, 2010, pp. 87-92.
[70] E. M. Schmidt and Y. Kim, "Learning Rhythm And
Melody Features With Deep Belief Networks," in
ISMIR, 2013, pp. 21-26.
[71] R. Schramm and E. Benetos, "Automatic
transcription of a cappella recordings from multiple
singers," in AES International Conference on
Semantic Audio, 2017.
[72] P. Sermanet, A. Frome, and E. Real, "Attention for
fine-grained categorization," arXiv preprint
arXiv:1412.7054, 2014.
[73] B. L. Sturm, "Classification accuracy is not
enough," Journal of Intelligent Information
Systems, vol. 41, no. 3, pp. 371-406, 2013.
[74] B. L. Sturm, "The GTZAN dataset: Its contents, its
faults, their effects on evaluation, and its future use,"
arXiv preprint arXiv:1306.1461, 2013.
[75] B. L. Sturm, "A simple method to determine if a
music information retrieval system is a “horse”,"
IEEE Transactions on Multimedia, vol. 16, no. 6,
pp. 1636-1644, 2014.
[76] G. Tzanetakis and P. Cook, "Musical genre
classification of audio signals," IEEE Transactions
on speech and audio processing, vol. 10, no. 5, pp.
293-302, 2002.
[77] J. Urzelai, D. Floreano, M. Dorigo, and M.
Colombetti, "Incremental robot shaping,"
Connection Science, vol. 10, no. 3-4, pp. 341-360,
1998.
[78] J. J. Valero-Mas, E. Benetos, and J. M. Inesta,
"Classification-based Note Tracking for Automatic
Music Transcription," 2016.
[79] A. Van Den Oord, S. Dieleman, and B. Schrauwen,
"Transfer learning by supervised pre-training for
audio-based music classification," in ISMIR, 2014.
[80] C. G. vd Boogaart and R. Lienhart, "Note onset
detection for the transcription of polyphonic piano
music," in Int. Conf. on Multimedia and Expo
(ICME), 2009, pp. 446-449: IEEE.
[81] P. Vincent, H. Larochelle, Y. Bengio, and P.-A.
Manzagol, "Extracting and composing robust
features with denoising autoencoders," in
Proceedings of the 25th international conference on
Machine learning, 2008, pp. 1096-1103.
[82] P. Viola and M. Jones, "Rapid object detection using
a boosted cascade of simple features," in Computer
Vision and Pattern Recognition (CVPR), 2001, vol.
1, pp. I-I: IEEE.
[83] P. Viola, M. J. Jones, and D. Snow, "Detecting
pedestrians using patterns of motion and
appearance," in Computer vision, 2003, p. 734:
IEEE.
[84] Z. Wang, X. Wang, and G. Wang, "Learning fine-
grained features via a CNN tree for large-scale
classification," Neurocomputing, vol. 275, pp.
1231-1240, 2018.
[85] F. Weninger, C. Kirst, B. Schuller, and H.-J.
Bungartz, "A discriminative approach to polyphonic
piano note transcription using supervised non-
negative matrix factorization," in IEEE Int. Conf. on
Acoustics, Speech and Signal Processing (ICASSP),
2013, pp. 6-10.
[86] C. White and C. Pandiscio, "Do Chord Changes
Make Us Hear Downbeats?," 2015.
[87] T. Xiao, Y. Xu, K. Yang, J. Zhang, Y. Peng, and Z.
Zhang, "The application of two-level attention
models in deep convolutional neural network for
fine-grained image classification," in IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR),
2015, pp. 842-850.
[88] V. Zenz and A. Rauber, "Automatic chord detection
incorporating beat and key detection," in IEEE Int.
Conf. on Signal Processing and Communications
(ICSPC), 2007, pp. 1175-1178.
[89] N. Zhang, J. Donahue, R. Girshick, and T. Darrell,
"Part-based R-CNNs for fine-grained category
detection," in European Conf. on computer vision,
2014, pp. 834-849: Springer.
[90] B. Zhao, J. Feng, X. Wu, and S. Yan, "A survey on
deep learning-based fine-grained object
classification and semantic segmentation," Int.
Journal of Automation and Computing, vol. 14, no.
2, pp. 119-135, 2017.