IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 14, NO.
12, DECEMBER 2017
Zero-Shot Learning of SAR Target Feature Space
With Deep Generative Neural Networks
Qian Song, Student Member, IEEE, and Feng Xu , Senior Member, IEEE
Abstract— Zero-shot learning (ZSL) is of critical importance
for practical synthetic aperture radar (SAR) automatic target
recognition (ATR) as training samples are not always available for
all targets and all observation configurations. We propose a novel
generative-based deep neural network framework for ZSL of
SAR ATR. The key component of the framework is a generative
deconvolutional neural network referred to as generator. It learns
a faithful hierarchical representation of known targets while
automatically constructing a continuous SAR target feature space
spanned by orientation-invariant features and orientation angle.
It is then used as a reference to design and initialize an interpreter
convolutional neural network, which is inversely symmetric to the
generator network. The interpreter network is then trained to
map any input SAR image, including those of unseen targets,
into the target feature space. In a preliminary experiment with
the Moving and Stationary Target Acquisition and Recognition
data set, seven targets are used in the training of generator and
interpreter networks. Then, the eighth target is used to test the
interpreter, where it is correctly mapped to the reasonable spot
spanned by the previous seven targets and its orientation can
also be estimated.
Index Terms— Deep generative neural network, orientation-
invariant feature space, synthetic aperture radar (SAR).
I. I NTRODUCTION
S YNTHETIC aperture radar (SAR) images are very dif-
ferent from optical images and are difficult to interpret
because of the microwave wavelength employed and the
phase-coherent nature of SAR imaging. Nevertheless, a SAR
image contains rich information about the target and scene
under observation, e.g., geometry, material, and structure.
Human interpretation of SAR imagery, which needs experi-
enced experts to find small targets in massive SAR images, is
challenging, time consuming, and impractical in the big data
era of remote sensing [1]. After being trained on enough sam-
ples, deep learning can imitate the mechanism of human brain,
learn the latent features of targets, and help machine to inter-
pret SAR images automatically. It has revolutionized first the
computer vision area and then many other machine learning
areas including the SAR image interpretation, e.g., automatic
target recognition (ATR) [2], terrain surface classification [3],
and parameter inversion [4]. Chen et al. [2] first applied
Manuscript received July 21, 2017; revised September 5, 2017 and
September 27, 2017; accepted September 28, 2017. Date of publication
November 10, 2017; date of current version December 4, 2017. This work was
supported in part by the National Key R&D Program of China under
Grant 2017YFB0502700 and in part by NSFC under Grant 61571134.
(Corresponding author: Feng Xu.)
The authors are with the Key Laboratory for Information Science of
Electromagnetic Waves (MoE), Fudan University, Shanghai 200433, China
(e-mail: fengxu@fudan.edu.cn).
Color versions of one or more of the figures in this letter are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/LGRS.2017.2758900
1545-598X © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
2245
a convolutional neural network (CNN) to SAR ATR and
achieved a state-of-the-art accuracy of 99% on the ten-types
of supervised classification tasks of the Moving and Station-
ary Target Acquisition and Recognition (MSTAR) benchmark
data sets.
Although being powerful in classification, CNN as a super-
vised discriminative network is highly sensitive to selection
and size of training samples and thus susceptible to overfitting
problems. It is particularly well known for its weakness
in generalization. Its performance would rapidly degrade if
applied to different observation configurations or tasked to
discriminate variant target types. It would become completely
useless for new target types which have never been seen in the
training samples, which is a practical need in many applica-
tions. This becomes a major hindrance in practical application
of CNN to SAR ATR.
In deep learning regime, this is an important research
topic which has motivated the so-called zero-shot learn-
ing (ZSL) [5], [6] and one-shot learning (OSL) [7].
ZSL requires no training samples while OSL requires only few
training samples or just one training sample. ZSL/OSL is often
achieved by semantic representation, which maps the low-
level image features into mid-to-high-level space, where the
distance calculated between two samples reveals their relation-
ships. Usually, semantic representation can be categorized into
two types: attribute-based binary vectors [8] and word-based
continuous vectors [9]. The former requires manually defining
the attributes of targets and then assigning the corresponding
labels for each training sample, which is apparently too costly
for annotation. The later works only for common label words
which have simple meanings, e.g., car, airplane, and dog.
This is not the case for a specific class of SAR targets
such as the eight vehicles in the MSTAR data set, e.g.,
T72 and T62.
Generative neural network [10], [11] as opposed to discrim-
inative neural network is often used for unsupervised learning
such as representation learning. This letter addresses the
ZSL problem of SAR ATR using deep generative neural
networks. It learns to mimic SAR images by training a
generative deep neural network (DNN). It automatically learns
a hierarchical representation of SAR target features from given
SAR images. In the meanwhile, it constructs a continuous SAR
target feature space where target orientation and orientation-
invariant intrinsic features are disentangled. Such feature space
is spanned by known target samples and will be used as the
reference to interpret targets which have never been seen in
the training samples. Thereafter, a deep CNN is trained to
continuously map any input SAR image to the learned target
2246
Fig. 1. Proposed framework of the generative DNN-based SAR target feature
space construction and interpretation.
feature space. Thus, it can faithfully interpret unseen target
images.
The remainder of this letter is organized as follows. The
overall architecture is explained in Section II, while Section III
presents the results of generative neural networks of the
MSTAR data set. Section IV gives the results of interpreter
CNN. Section V concludes this letter.
II. OVERALL A RCHITECTURE
Suppose we have cs known targets with n s samples
S = {cs , vs , Fs , X s }, and ct unseen targets with n t samples
T = {vt , Ft , X t }. X denotes the SAR images, and c and v
are the label and orientation information of targets. The key
idea of this letter is to construct a feature space and learn the
mapping from the SAR images to orientations and orientation-
invariant features Fs using cs , vs , and X s of S. The learned
mapping can be used to infer vt and Ft of unseen targets T
given X t .
Fig. 1 illustrates the overall framework of DNNs for the
proposed ZSL SAR ATR. In the forward generative direction,
the input is discrete labels and orientation information of
targets. Labels are first fed into a fully connected construc-
tor NN, which constructs a continuous target space spanned
by orientation-invariant features, which are then concatenated
with orientation information to form the complete feature
vector. Subsequently, this complete feature vector is fed
into a deep DNN, i.e., the generator DNN, which generates
pseudo-SAR images of the corresponding target at a particular
orientation. The generative networks, both the constructor and
the generator DNNs, are trained using SAR images of known
targets. They learn from the data to construct a continuous fea-
ture space where all known targets can find their corresponding
position.
Once the orientation-invariant feature vector is constructed,
it is then used as the goal to train the inverse direction, i.e.,
from an input SAR image to its feature space. The inverse
direction is a simple CNN which maps an SAR image to
the orientation-invariant feature space and at the same time
extracts its orientation information. Such a CNN is referred to
IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 14, NO. 12, DECEMBER 2017
as an interpreter deep neural network in this letter, and trained
on real SAR images of known targets.
The goal of the whole network is to construct the inter-
pretable feature space F automatically. In previous works,
F was often defined manually or semimanually or learned
automatically from available textual documents, while in our
framework, it is learned automatically from SAR images using
the constructor–generator framework. It derives the distribu-
tion of F given target labels c and the interpreter DNN to
estimate the distribution of F given real SAR with the goal of
min L( p(F|c)|| p(F|X x∼D )). (1)
To obtain p(F|c), the generator DNN tries to minimize the loss
function defined as L 2 norm between the generated images
and actual images with respect to parameters θc and θg of
constructor and generator DNNs
min L(G(C(c;θc ), v; θg )||X). (2)
θc ,θg
Then learned θg acts as prior knowledge for the inter-
preter DNN. Note that here in order to ensure that the feature
space is physically sound, we pretrained the constructor NN on
training samples by manually defining the prototype of each
class F0c . Thus, θc is initialized by
 
min L C(c;θc )||F0c . (3)
θc0
After obtaining p(F|c), the interpreter aims at minimizing
the loss defined between p(F|c) and p(F|X x∼D ). The whole
network is trained only by limited types of targets, i.e.,
cs classes of S; however, the trained interpreter DNN can also
be utilized to interpret other types of targets such as ct types
of T , which is the principle of ZSL.
Compared to conventional generative neural networks,
the proposed framework includes a supervised constructor NN
which tends to map known labels to a continuous intrinsic
feature space. The constructor NN is simultaneously trained
with the deconvolutional generator network. The key is to take
advantage of hierarchical representation learning to construct
the desired feature space.
III. C ONSTRUCTOR –G ENERATOR DNN OF SAR I MAGES
The proposed framework is then implemented on the famous
SAR ATR benchmark data set MSTAR data set [12], which is
widely used for SAR target recognition and classification task.
In this letter, eight types of vehicle targets as observed by the
X-band one-foot-resolution SAR from 360° orientations have
been used. For each type, there are ∼300 images. The orien-
tation (i.e., aspect angle) of each SAR image can be obtained
in the header of its binary file. To test the ZSL capability, only
seven types (i.e., cs = 7) of targets are used in training. The
eighth type is used for testing. This is to mimic the fact that
not all targets are available as training samples, which is the
key challenge for ZSL.
The forward-pass constructor–generator network archi-
tecture is illustrated in Fig. 2. The inputs of the network are
one-hot target label c and orientation vector v of the target.
In this case, the constructor is a two-layer fully connected
SONG AND XU: ZSL OF SAR TARGET FEATURE SPACE 2247

Fig. 2. Constructor–generator network for MSTAR data.

Fig. 4. Convergence of MSE during training.

Fig. 3. Visualization of generated images during training.

network with the first layer consisting of 20 neurons and the

output layer consisting of two neurons corresponding to a
2-D feature space. Since the number of target types available is
limited, the feature space is designed to be a 2-D space, which Fig. 5. Comparison between real SAR images and generated images.
is also easier for visualization. Note that the feature space can
be of higher dimension, if more sophisticated feature partition
is needed. Label c is mapped to a continuous orientation-
invariant feature space F via this constructor NN C(c;θc ).
On the other side, orientation information is represented
as a 2-D vector v =[cos φ, sin φ]T , where φ ∈ [0, 360°]
denotes the orientation angle. Note that sin/cos representation
is employed to reflect the 360° periodicity [10].
The orientation-invariant feature F and orientation-only
feature v are concatenated and fed into the generator DNN,
which worked as an image generator. The generator consists
of a three-layer fully connected network and a deep DNN.
For each deconvolutional layer, the up-pooling operation takes
place by setting the stride to be 2. The final output image is
matched with the real SAR image with an L2 loss function.
The constructor–generator network is then trained via reg-
ular stochastic gradient descent using all training samples of Fig. 6. Autoconstructed target feature space as spanned by the seven targets
the known seven types of target. All the weights are randomly included in training samples.
initialized except θc , which is initialized by (3).
Fig. 3 shows the generated SAR images during the training and brightness. Apparently, the generator had learned to cap-
process. The network converges after 200 epochs. It is interest- ture the intrinsic characteristics of the data set. This capability
ing to find that the network first learns the location of targets could be potentially explored for speckle reduction application.
(e.g., at the second epoch), then learns the general characteris- Now let us look at the constructed orientation-invariant
tics of targets (e.g., at the 72nd epoch, target shape and shadow feature space spanned by the 2-D F. The seven types of
area are observable), and finally learns the textures of the targets as mapped to the feature space are shown in Fig. 6,
target image (e.g., at the 177th epoch). Fig. 4 shows the mean along with the optical images of each target. The constructed
squared error (MSE) between real and generated SAR images feature space is not directly linked to any physical parameters.
at different training epochs. The network is trained with one However, it faithfully reflects the distances or similarities
TITAN X GPU. Training takes about 2 s per epoch. among different targets. For example, BRDM2, BTR60, and
Fig. 5 compares nine samples of the real SAR images with 2S1 are located nearby as all three are armored vehicles.
the corresponding generated SAR images. It is interesting to D7, the backhoe, and ZIL131, the truck, are located farthest
note that the generated SAR image appears to be smoother away from the rest of the targets as they differ greatly in either
and speckle free. However, it preserves the distinct features shape or scale. Note that the learned distribution might be
of the target itself, e.g., target geometric profile, shadow, sensitive to the network initialization, but the basic topology
2248
Fig. 7. Generated arbitrary target full-aspect SAR images.
Fig. 8. Inverse interpreter DNN for MSTAR data.
reflecting the mutual distances among different targets should
be stable.
To demonstrate the efficacy of the generator DNN, we sam-
ple arbitrary from the feature space and then examine the
generated fake SAR images. For example, we can sample from
the orientation-invariant feature space a point between any two
targets. Then, full-aspect SAR images of such an imaginary
target can be generated. Fig. 7 shows examples of generating
0°–360° SAR images of an imaginary target between D7 and
T62, T62 and 2S1, and between 2S1 and ZIL123, respectively.
We believe that the generative DNN is crucial in construct-
ing the continuous feature space which faithfully reflects the
feature of each target and the distances among them.
IV. I NTERPRETER DNN FOR U NSEEN SAR TARGET
Once we constructed the target feature space via training
the constructor–generator DNN, the interpreter DNN can be
easily designed and trained. The interpreter DNN is symmetric
to the generator network, but in backward direction (Fig. 8).
It is a CNN as opposed to the generative DNN. Its architecture
is the same as the generator DNN except that the input and
output are swapped at each layer. In addition, the weights of
interpreter DNN are initialized as the same weights as the
generator DNN. It serves as a weak regularization to bind the
inverse interpreter to the generator whose SAR representative
power had been tested.
IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 14, NO. 12, DECEMBER 2017
Fig. 9. Distribution of test samples of known targets as interpreted in the
feature space.
Fig. 10. Orientation angle estimation of test samples of known targets.
(Left) Scatterplot. (Right) Error histogram.
The key difference between the interpreter DNN and con-
ventional supervised classifier DNN is that the goal output is
a continuous target feature space instead of discrete labels.
Another key factor of interpreter DNN is that it has to
be the inverse network of a successfully trained generator
network. It guarantees that the interpreter will correctly map
the input SAR image to the continuous feature space. Again,
the generative DNN plays a critical role in constructing and
training the interpreter DNN.
The interpreter DNN is trained using 80% of the known
seven types of real SAR images, and the rest (20%) is used
to validate the fitness of the network. The GPU takes 36 s
to train the network and 0.2 s to test one SAR image.
Fig. 9 shows the distribution of the 20% validation samples
as interpreted in the orientation-invariant feature space. It can
be seen that the known seven types are well separated and
clustered around respective centers. An overall classification
accuracy of 96.8% can be achieved if we simply apply nearest
neighbor clustering. Fig. 10 shows the estimated orientation
angle versus true orientation angle. The standard deviation
error for orientation estimation is 16°.
Finally, let us examine the ZSL capability of the interpreter
network. We pick the T72 target from the rest target types
which are not included in the training samples. It means that
the interpreter has never seen any SAR images of a T72.
In an ideal case, we want the interpreter network to be able to
SONG AND XU: ZSL OF SAR TARGET FEATURE SPACE
Fig. 11. T72 SAR images as interpreted by the zero-shot learned interpreter
DNN. (a) Distribution of “new target” T72 on the feature space. (b) Similarity
of T72 subtypes A04 and A05 to T62 and 2S1. (c) Orientation estimation error
of T72-A04 and T72-A05.
interpret T72 SAR images in the context of the seven known
targets.
Feeding 0°–360° SAR images of T72 (subtypes A04
and A05) into the interpreter, it generates the corresponding
scatter points in the feature space as well as the corresponding
orientation angle. Fig. 11(a) shows the clusters of
T72-A04 and T72-A05 in the target feature space. It shows
that both T72 targets are distributed among T62, 2S1, BTR60,
and BRDM2. It reflects the fact that T72 looks similar to these
armored vehicles. The T72-A05 is shifted slightly toward T62,
while T72-A04 is closer to 2S1. In Fig. 11(b), we compare
the optical and SAR images of these four targets, from which
we believe that the interpreter DNN is able to accurately
reflect the similarity/dissimilarity of a new unseen target to
known targets. Thus, one could roughly say that the unseen
target T72 is an armored vehicle similar to T62 and 2S1 but
not a truck like ZIL131 or a backhoe like D7. Note that the
2249
clusters of unseen targets are more dispersive which means
that the interpreter is more uncertain with unseen targets.
On the other hand, the interpreter can also extract the
orientation angle of an unseen target. Fig. 11(c) shows the
error histograms of T72-A04 and T72-A05. The interpreter
performs reasonably well in this task with a minor degradation
in terms of standard deviation error (∼35°).
V. C ONCLUSION
ZSL or OSL is of particular importance for SAR ATR
applications due to lack of training samples of many targets
and many observation configurations. This letter proposed
a generative DNN-based framework for ZSL of SAR ATR,
which was demonstrated on the MSTAR data set, where sam-
ples of only seven targets were used in training and the eighth
target type was used for testing. The generator DNN employs
an L2-norm loss function and successfully learns hierarchical
representation of the selected seven types of SAR target
images. From the generated images of training targets and
sampled images of imaginary targets, we see that the generator
DNN helped to construct a continuous feature space spanned
by the selected seven types. The generator DNN plays a critical
role not only in constructing a physically sound feature space
but also in initializing and regularizing the interpreter DNN,
which is later trained to interpret unseen targets in the context
of known targets. The interpreter DNN is inversely symmetric
to the generator DNN, and it is trained to map an input
SAR image to the feature space. The interpretation results of
unseen T72 targets demonstrated the efficacy of the proposed
ZSL framework. It correctly maps T72 to the feature space
and estimates its orientation angle.
R EFERENCES
[1] F. Xu, Y.-Q. Jin, and A. Moreira, “A preliminary study on SAR advanced
information retrieval and scene reconstruction,” IEEE Geosci. Remote
Sens. Lett., vol. 13, no. 10, pp. 1443–1447, Oct. 2016.
[2] S. Chen, H. Wang, F. Xu, and Y.-Q. Jin, “Target classification using
the deep convolutional networks for SAR images,” IEEE Trans. Geosci.
Remote Sens., vol. 54, no. 8, pp. 4806–4817, Aug. 2016.
[3] Y. Zhou, H. Wang, F. Xu, and Y.-Q. Jin, “Polarimetric SAR image
classification using deep convolutional neural networks,” IEEE Geosci.
Remote Sens. Lett., vol. 13, no. 12, pp. 1935–1939, Dec. 2016.
[4] L. Wang, K. A. Scott, L. Xu, and D. A. Clausi, “Sea ice concentration
estimation during melt from dual-pol SAR scenes using deep convo-
lutional neural networks: A case study,” IEEE Trans. Geosci. Remote
Sens., vol. 54, no. 8, pp. 4524–4533, Aug. 2016.
[5] C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and
K. Q. Weinberger, “Zero-shot learning through cross-modal transfer,”
in Proc. Adv. Neural Inf. Process. Syst., 2013, pp. 935–943.
[6] Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong, “Transductive multi-
view zero-shot learning,” IEEE Trans. Pattern Anal. Mach. Intell.,
vol. 37, no. 11, pp. 2332–2345, Nov. 2015.
[7] L. Fei-Fei, R. Fergus, and P. Perona, “One-shot learning of object
categories,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 4,
pp. 594–611, Apr. 2006.
[8] Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong, “Learning multimodal
latent attributes,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 2,
pp. 303–316, Feb. 2014.
[9] A. Frome et al., “DeViSE: A deep visual-semantic embedding model,”
in Proc. Adv. Neural Inf. Process. Syst., 2013, pp. 2121–2129.
[10] A. Dosovitskiy, J. T. Springenberg, M. Tatarchenko, and T. Brox.
(2015). “Learning to generate chairs, tables and cars with convolutional
networks.” [Online]. Available: https://arxiv.org/abs/1411.5928
[11] I. Goodfellow et al., “Generative adversarial nets,” in Proc. Adv. Neural
Inf. Process. Syst., 2014, pp. 2672–2680.
[12] The Air Force Moving and Stationary Target Recognition Database.
Accessed: Sep. 2013. [Online]. Available: https://www.sdms.afrl.af.mil/-
datasets/mstar/