A COMPLETE PROCESSING CHAIN FOR

SHADOW DETECTION AND

RECONSTRUCTION

SEMINAR DRAFT REPORT

Submitted by:
ASHWIN VASUNDHARAN
FIRST SEMESTER M.TECH SIGNAL PROCESSING
ROLL NO. 04

Contents
1 INTRODUCTION

2 PROBLEM FORMULATION

3 PROPOSED METHOD
3.1 MASK CONSTRUCTION . . .
3.1.1 Binary Classification . .
3.1.2 Postprocessing . . . . .
3.2 BORDER CREATION . . . . .
3.3 CLASSIFICATION MAPS . . .
3.3.1 Multiclass Classification
3.3.2 Postclassification . . . .
3.3.3 Quality control . . . . .
3.4 Shadow Reconstruction . . . .
3.5 Border Reconstruction . . . . .
3.6 CONCLUSION . . . . . . . . .
3.7 REFERENCES . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

5
5
6
6
7
8
8
8
9
9
10
12
13

Chapter 1
INTRODUCTION
Very high resolution (VHR) images opened a new a new field of remote sensing
and it has become an increasingly important source of information. VHR images give
detailed information about the little objects like trees, buildings, roofs, and vehicles. But high resolution includes some draw back too, unwanted presence of shadow.
Even though we can exploit shadow characteristics to recognize building position and
estimate their height and other useful parameters, shadows are considered as undesired parameter, because it affects images badly. The presence of shadow will cover
information and characteristics of image highly. Shadows may cause severe problems like presence of false colour tones, distorted image shape, merge, or lose objects
from the VHR image, which is not desired. The presence of shadows may negatively
affect VHR image exploitation, while mapping, leading to erroneous classification
or interpretation, due to the partial or total loss of information. These problems
can be removed by detecting and removing the shadows present in the VHR images.
There are mainly two approaches to detect shadows, 1) model based 2) shadow
property based approaches. The first approach needs prior information about the
scenario and sensor. However, since such information is not available, nearly all
detection algorithms use shadow property for detection of shadow. Shadow properties like lower brightness, higher saturation, greater hue values etc. are used.
In this paper shadow detection is done using a hierarchical supervised classification scheme, and proposed reconstruction relies on linear correlation, which utilises
the information from the classification. The whole information also includes two
capabilities, 1) a rejection mechanism to limit as much as possible reconstruction
errors and 2) explicit handling of shadow borders.
2

Chapter 2
PROBLEM FORMULATION
In VHR image especially in urban areas the presence of shadows will erase the
details of the image. And this missing information directly affects the processing.
Shadows are normally produced when an object obstructs the light. Shadows can
be divided into two, self shadows and cast shadows. Cast shadows are the shadows
which produced by the projection of light source in the direction of the object. Self
shadow represents the unilluminated part of the object. Self shadows can have nonlinear behaviour.
This work for simplicity does not distinguish between self shadow and cast shad-

Figure 2.1: Illustration of cast and self shadows

ows assuming that producing homogeneous dark areas with a loss of information.
In this work the detection of shadow is done by means of hierarchical supervised
classification process because,
1). To separate between shadow and non-shadow region in the given image.
2). To identify the different non-shadow classes as well as their corresponding shadow
counterpart. Ground truth is needed for both categories of classes and this is the
only part human help is needed. Reconstruction of image is done with the assumption that all classes follow a Gaussian distribution, although it wont hold always
helpful for simplicity.
The shadow class is represented as
X N (s , s2 )
The non-shadow class is represented as Y N (s, s2 )
And the reconstruction of shadow class can be done simply by random variable
transformation
X N (s , s2 ) X 0 N (s, s2 ).
(1)

Chapter 3
PROPOSED METHOD
The proposed method consists of several steps. The process is done by considering a VHR image I of dimension mxn composed of N bands and characterized
by the presence of shadow areas. In the first step the ground truth information is
collected by selecting different regions of interest (ROI) and used to discriminate the
image into different classes, shadow and counter parts. This first step is the only
one step where human interaction is needed, to build the ROI for the classes. The
image is discriminated into shadow and non-shadow classes using binary classification with the help of ground truth. Two mathematical operations, namely opening
and closing by reconstruction is used to avoid noise b making a mask. These two
operations are also used to create border between classes as the binary classification
does not give a precise edge between the classes. In the successive step shadow
and non-shadow classes are classified separately with the same initial ROI. This
classification allows location of the available couples of shadow and non-shadow is
based on a linear regression method, where intensities of shaded areas are adjusted
accordingly according to the statistical of non-shaded areas. In the final step the
border of reconstructed shadow and non shadow areas are linearly interpolated to
give a smooth transition. In the following section detailed description of all these
steps is given.

3.1

MASK CONSTRUCTION

The shadow versus non-shadow mask is created in two steps namely, binary
classification followed by a post processing.
5

Figure 3.1: Flowchart of the proposed method.

3.1.1

Binary Classification

Binary classification procedure is done using support vector machine. The

choice SVM is motivated by its effectiveness in classification of remote sensing images. The feature space where classification is done is defined b the original image
bands, and the features are extracted b means of wavelet transform. A one-level
stationary wavelet is applied on each spectral band, thus obtained four space- frequency features corresponding to each spectral band. The symlet wavelet is adopted
in order to maximize the sparseness of the transformation while enforcing the texture areas. If an original image I consists of B spectral bands, the resulting feature
space will have B (1 + 4) dimensions.

3.1.2

Postprocessing

The binary image may be characterized by a salt and pepper effect due to the
presence of noise in the image. This problem is solved by morphological operations.
An opening by reconstruction, followed by a closing by reconstruction is done. These
morphological filter are used because they have the characteristics of shape preser6

vation and possibility to adapt them according to the image filtering requirements.
Both the morphological operations are needed to remove isolated non-shadow pixels
in shadow area and also isolated shadow pixels in non-shadow area. For illustration,
an example of enhanced mask M2 is shown in fig. 3.2(b), where a 3 3 pixel sized
morphological structuring element (SE) is used , as shown in fig. 3.3(a).

Figure 3.2: Example of (a) initial mask image M1, (b) mask postprocessed by
opening and closing M2, and (c) final mask with borders MF

3.2

BORDER CREATION

The transition between non-shadow and shadow areas raises problems such as
boundary ambiguity, colour inconstancy, and illumination variations. And the presence of penumbra induces mixed pixels in the transition regions which are difficult
to classify. Penumbra is a region where light is only partially obstructed. Due to
this reason, a border is defined to properly handle the border pixels. These areas
are not processed during shadow reconstruction and done separately. Border region
is constructed by using morphological operators. The mask c imgB2 is dilated ()
and eroded(). Then the difference between these two is taken to form border image
B.
B[x, y] = (c imgB2[x, y]) (c imgB2[x, y]).
(2)
Keep in mind that the border is not needed in all directions. Indeed to only
focus on penumbra, the sun direction is taken into account for adopting proper structure element for the morphological operations in order to track the actual shadow
direction, like shown in fig.3.3(b). The final mask image becomes

B[x, y], if B[x, y] = 1
ci mgBN EW [x, y] =
(3)
ci mgB2[x, y], if B[x, y] = 0
7

Figure 3.3: Examples of SEs used for (a) the standard morphological operator and
(b) the morphological operator adapted to the shadow geometry related to the image
in Fig.3.2.

3.3
3.3.1

CLASSIFICATION MAPS
Multiclass Classification

The previously obtained mask is used to guide further level of classification

applied separately to shadow and non-shadow areas. The aim is to distinguish
predefined shadow classes on one side and corresponding non-shadow classes on
other side. And the result is final classification map C. This is an important part
to define spectral relationship between the shadow and non-shadow version of same
object and thus to perform customized reconstruction of shadow areas. For these
purposes two multiclass SVMs are trained in predefined feature space for shadow
and non-shadow classes. After training phase, to generate C , both are applied to
predict labels of each pixels on shadow and non-shadow areas defined by MF .

3.3.2

Postclassification

In order to improve the classification map C before exploiting it for the reconstruction of shadow areas, post classification is done b using a 3 3 majority filter
for removing isolated labels and thus smoothing the map.
8

3.3.3

Quality control

Since the spectral signature of shadow classes have usually low radiometric
behaviours, the classification of shadow areas are rather tricky. And it becomes
necessary to control its quality in order to decide the feasibility of reconstruction,
because reconstruction directly based on the classification map. For such purposes,
a confusion matrix is created for shadow classes based on the available ground truth.
For each shadow class, both user (corresponding to commission errors or inclusion)
and producer (corresponding to omission error or exclusion) accuracies area derived.
If one of these two accuracies are below a predefined value, the classification corresponding to that shadow class is considered as of low quality and dont take into
account when reconstruction is done. In other words, reconstruction is carried out
only if the classes give a minimum guaranty of correct shadow recognition is available
in order to avoid error propagation in the processing chain.

3.4

Shadow Reconstruction

Image reconstruction is the most important step in the methodology. For the
sake of getting simple but satisfactory reconstruction model, we assume that the underlying relationship between the non-shadow class(Y) and corresponding shadow
class(X) is of linear type. We empirically observed that non-shadow classes and
corresponding shadow classes exhibit a linear relationship. Regarding the statistical
model of the classes, three estimation way may be envisioned. 1) histogram estimation by box counting 2) kernel density estimation 3) parametric estimation. In our
case we will adopt parametric estimation by assuming that classes follow a Gaussian
distribution. This is used because of need to derive an analytically tractable and
easy to implement reconstruction method. Under this assumption, X N (s , s )
and Y N (s, s) where and stands for the mean and covariance matrix, respectively. Since the two relations are assumed linearly correlated, x and y may linked
by
y = Kx + c
(4)

s = Ks + c
(5)
s = Ks K T
Where K is a transformation matrix,K T is its transpose , and c a bias vector.
To estimate K and c, the Cholesky factorization is applied.

c = s Ks
(6)
K = UsVs1
9

Figure 3.4: Illustration of the reconstruction of a border pixel with a 7 7 size

window.
WhereUsandVs are the lower and upper triangular Cholesky matrices related to
the non-shadow and shadow classes , respectively. Once K and c is estimated (4)
is applied to compensate the pixels of the shadow class. When we apply shadow
compensation the restored area may appear as noisy because the initial shadow
distribution is much more concentrated than the non-shadow areas. To minimise
this effect, the coefficient of variation(CV) defined as CV [i] = ii (i and i are
the standard deviation and mean along the i th image band,respectively is used to
weigh the target variance of the re constructed area. In particular, we find the ratio
CVs[i]
between CV of non-shadow and shadow classes,i.e, NCV = CV
, for the ith original
s [i]
image band(i = 1, ...., N ). Then if NCV [i] 1, the covariance matrix is corrected to
reduce the variability of the non-shadow class
covN EW [i, k] =

3.5

cov[i, k]
NCV [i].Ncv [k]

with(i, k) [1, N ]2

(7)

Border Reconstruction

After the reconstruction of shadow areas the borders between shadow and nonshadow areas still remain with their original aspects, which may be in contrast with
the adjacent areas. In order to smooth these, pixels of the borders undergo an easy10

to-implement and fast contextual linear interpolation. In greater detail, a sliding

window of predefined SxS size is adopted and within which four directional linear
interpolations are considered. Among the directions, namely, North-South, WestEast, NE-SW and NW-SE, just those crossing the given reconstructed shadow and
non-shadow areas are retained. Along a given direction the linear interpolation is
simple affine transformation defined as follows
Z = m.i + q

(8)

Where i is the coordinate along that direction, m and q are interpolation parameters. In order to estimate these parameters we use the least square estimator, which
consists in our case in collecting the set of points which are as follows 1) within the
widow; 2) outside the border; and 3) along the considered direction crossing the
central pixel of the window:


Z1
i1 1 
m
... = ... 1
Z = .
(9)
q
ZN
iN 1
And the getting parameter estimate as follows
= (T )1 T Z

(10)

An interpolator is derived for each available direction. Different strategies could

be used for merging estimates. We found that by visual inspection that best fusion
rule among four common rules, which are MIN, MAX, average, and median rules,
is archived by retaining as estimate for the central pixel the largest value yield, ie,
the MAX rule. This can be easily explained by the fact that the MAX rule provides
values which are brighter and less contaminated by shadow or penumbra.

11

3.6

CONCLUSION

This paper dealt with the important and challenging problems of reconstruction
of VHR images consisting shadows. The shadows are detected and classified to allow customized compensation. The classification is done by means of state-of-the-art
SVM approach. A quality check mechanism is also done to avoid misreconstruction
problems. Borders are handled by using morphological filters and linear interpolation. In general, from the obtained results, different considerations ma be deduced.
1) The proposed methodology can yield visually realistic shadow free images with
promising preservation of spectral and textural properties of the obscured objects.
Moreover, it improves the results in terms of classification accuracy.
2) Because of the quality check mechanism, not all shadows are systematically reconstructed. Part of them may remain unchanged due to the difficulty that they
are recognized correctly.
3) The recognition of the couples of non-shadow and shadow classes can help in more
accurate compensation while classification creating false shadow class will lead to
misreconstruction.
4) In the case of multi modal non-shadow classes the implemented linear compensation ma find inappropriate.
The proposed processing chain relies on different advanced technologies. Although
interesting result have been obtained more improvements are required.They may be
the followings,
1)reconstructability evaluation in order to a prior asses if a particular shadow area
can be potentially constructed or not
2)active learning process to adapt as most as possible the ground truth to the image
reconstruction requirements.
3)reinforce the quality check mechanism with multimodality assessment
4)accompanying the reconstruction map with a kind of confidence map as done for
the problem of cloud-contaminated image.

12

3.7

REFERENCES

1). E. Salvador, A. Cavallaro, and T. Ebrahimi, Shadow identification and classification using invariant color models, in Proc. IEEE Int. Conf. Acoust., Speech,
Signal Process., 2001, vol. 3, pp. 15451548.
2).S. Wang and Y. Wang, Shadow detection and compensation in high resolution
satellite images based on retinex, in Proc. 5th Int. Conf. Image Graph., 2009, pp.
209212.
3).P. Sarabandi, F. Yamazaki, M. Matsuoka, and A. Kiremidjian, Shadow detection and radiometric restoration in satellite high resolution images, in Proc. IEEE
IGARSS, Sep. 2004, vol. 6, pp. 37443747.
4).J. A. Benediktsson, M. Pesaresi, and K. Amason, Classification and feature extraction for remote sensing images from urban areas based on morphological transformations, IEEE Trans. Geosci. Remote Sens., vol. 41, no. 9, pp. 19401949, Sep.
2003.
5).N. Ghoggali and F. Melgani, Genetic SVM approach to semisupervised multitemporal classification, IEEE Geosci. Remote Sens. Lett., vol. 5, no. 2, pp. 212216,
Apr. 2008.

13