Proceedings

22nd International Conference

on Pattern Recognition

2428 August 2014

Stockholm, Sweden

Los Alamitos, California

Washington

Tokyo

2014 22nd International Conference on Pattern Recognition

A defect recognition system for automated

inspection of non-rigid surfaces
Sebastian von Enzberg and Ayoub Al-Hamadi
Otto-von-Guericke-University Magdeburg
Institute for Information Technology and Communications (IIKT)
39016 Magdeburg, PO Box 4210, Germany
Email: {sebastian.vonenzberg, ayoub.al-hamadi}@ovgu.de
AbstractThe goal of this work is the automated recognition of 3D surface defects for quality inspection in industrial
production. For complexly shaped workpieces that are non-rigid
and have non-uniform tolerance ranges, it is hard to distinguish
acceptable surface deviations from defects.
We propose a 3-stage defect recognition system based on 3D
measurement of the defective part. First, a variable B-Spline
surface model is used to adapt to acceptable tolerance ranges.
The remaining model deviations are then used for segmentation
of possible defects. Finally, a SVM-based classifier separates true
defects from pseudo defects.
On a real world data set of a series of measurements for a
car front hood, the effectiveness of the approach is proven.

I.

I NTRODUCTION

A faultless surface quality is crucial for customer acceptance of high-cost products, e.g. sheet metal car body parts
in the automobile industry [1]. Typical surface defects like
dents or bumps with heights down to 10 m can be visually
perceived on specular reflecting surfaces [2] due to the distortion of reflected patterns. While this effect is successfully used
by deflectometry systems for automated inspection [3], [?], it
allows a detection only on assembled and painted surfaces.
However to reduce costs for reworking, a defect detection very
early in the production process is preferred - possibly as early
as after sheet metal forming in the stamping plant.
Raw sheet metal parts are still mainly inspected manually
by professional auditors. The inspection process is thus highly
subjective and can only be performed on random samples
rather than on full production volume. The automated recognition of small defects on non-rigid surfaces is still an unsolved
problem due to several challenges. Since workpieces are nonrigid, the actual surface shape depends on the placement onto
the measurement support. Furthermore the allowed geometric
tolerances caused by variations in the stamping process are
several magnitudes higher (typically in the millimeter-range)
than allowed local surface deviations. While defects can be of
very small height, their lateral spread can cover a radius of
several millimeters on the surface. Also in modern automobile
design, surfaces are complexly shaped with a wide range of
designed surface details which possibly share characteristics
of defects. Consequently the allowed variabilities are highly
non-uniform, which renders an explicit modeling of tolerance
ranges impossible.
Stereo vision systems with white light pattern projection
(e.g. the phase shift method [4]) allow acquisition of high
1051-4651/14 $31.00 2014 IEEE
DOI 10.1109/ICPR.2014.318

Fig. 1. Cross-section of a surface with its allowed tolerance range and a

possible bump to be detected.

precision 3D point clouds on a wide range of non-specular

surfaces. The detection of surface defects can thus be posed
as a pattern recognition problem on a 3D point cloud or range
image.
A. Related Work
Li [5] gives a good introduction to automated surface
inspection with various different sensor concepts. Early approaches of vision based defect detection used a single camera
setup, observing texture and shadowing changes with rather
complex lighting setups [6], [7]. First 3D based methods
applied high-pass filtering to the acquired point cloud [8], [9],
assuming defects of smaller scale, and thus failing on complex
surface shapes.
Most model-based approaches are based on the CAD model
[10] and, for non-rigid surfaces, FEM-based deformation models [11]. Since this is a computationally expensive task, approximations to the FEM-deformation have been proposed [12],
yielding a rather imprecise approximation around 0.25 mm.
Recently, non-physically based stochastic deformation has
been proposed [13], [14] which is prone to over- or underfitting
of the surface model with a rather complex parametrization of
the algorithm. The first machine learning based approach was
presented in [15] and has been recently advanced [16].
Apart from surface models for inspection purposes, research effort has been put into defect classification with current
focus on decision tree and rule based classification algorithms
[17], [18].
II.

OVERVIEW OF THE PROPOSED SYSTEM

To overcome the problem of explicitly modeling the allowed surface shape variations, a variable surface model can
be trained by a given set of tolerance samples which is
commonly collected in quality control during the setup of the
1812

78/9:1454
!"#$%&'($)*&+,#$*
!"#$%&'()*&%&+,(-

in terms of B-spline base functions, an orthogonalization is

necessary. Using the matrix notation [20] of equation (1),
the application of the Gram-process can be executed with
linear matrix operations [21], resulting in a component-wise
description of the surface s(u, v):

!"#$%&'()*
+*($%&*+*,-

$./0,*#(..&120+(-01,

$./0,*#(..&120+(-01,

./012123

/45066

l
x(u, v) = bk (u) u T P
x v b (v)

3(&0(4/*#56$./0,*
$%&'()*#+17*/

'*(-%&*#*2-&()-01,

The x-, y- and z-components px,ij , py,ij , pz,ij of the

original surface control points pij are ordered into matrices
Px , Py , Pz and are transformed by
%1
$
1
T
P
Px (v ) .
x = u

)/($$0'0)(-01,

#4;45.$/458321.182

Overview of the defect recognition system

production process. The proposed surface model is based on

the Karhunen-Love-Transform of the B-spline surface representation of the tolerance samples [16], which gives a statistical
model of allowed surface variations.
For the evaluation of a workpiece, its surface is measured
by an optical 3D sensor using structured light and then compared to the output of the variable surface model. The resulting
distance map is then segmented for possible surface defects.
Several morphological and 3D-based features are extracted and
finally a SVM based classifier decides whether a defect is
recognized. The overall system structure is depicted in figure
2.
III.

VARIABLE B-S PLINE M ODEL

The proposed adaptive surface model is based on a tensor

product B-spline surface

s(u, v) =

x(u, v)
y(u, v)
z(u, v)

"

n #
m
#

pij bki (u)blj (v)

(2)

and accordingly for y(u, v), z(u, v), with vectorized base
functions bk (u) = [bk1 (u) . . . bkn (u)] and bl (v) =
[bl1 (v) . . . blm (v)] and transformation matrices u and v
derived from the singular value decomposition of the Gramian
matrices u , u .

.10,-#70$-(,)*
)1+.%-(-01,

$*8+*,-(-01,#1'
.1-*,-0(/#7*'*)-$

Fig. 2.

(3)

Similarily to the computation of Principal Components,

the computation of optimal base functions can now be posed
as the eigenvalue problem for the covariance matrix of the

orthogonalized control points p

x k , py k , pz k :
Cx ex,i

with Cx

x,i ex,i
#
& 'T
p
x k px k

(4)
(5)

The eigenvectors ex,i , ey,i , ez,i can be put into transform

matrices Ex = [ex,1 . . . ex,r ] , Ey , Ez . They are distinctive of the optimal base functions fix (u, v), fiy (u, v), fiz (u, v),
which describe the statistical properties of the tolerance samples.
The inspection of a workpiece consists of the KarhunenLoeve-Transform and the subsequent inverse transform of its
spline representation. Importantly, the dimensionality of the
representation is reduced in the transformed space to the r
optimal base functions with the most significant respective
eigenvalues. In matrix form, this is described by
ax =
p
x =

(1)

Ex p
x
Ex T ax

(KLT)
(KLT)1

(6)
(7)

i=1 j=1

which maps parametric surface coordinates (u, v) R2 to

3D space s = (x, y, z) R3 . It is characterized by the control
points pij R3 and the B-spline base functions bki (u), blj (v)
of order k, l. For 3D point measurements on a regular grid,
the data can be efficiently approximated by a B-spline surface
using the lofting approach [19].
The Karhunen-Loeve-Transform (KLT) is the generalization of the Hotelling Transform used for Principal Component
Analysis (PCA). Analogous to the projection onto the principal
components of a discrete, multi-dimensional random variable,
the KLT is the projection onto the optimal base functions of a
stochastic process represented by a set of continuous functions.
In order to efficiently compute these optimal base functions

The resulting vectors p

x , py , pz represent an approximation of the measured surface in the orthogonalized B-spline
space. After inverse orthogonal transform, they represent the
B-spline control points Px of the adapted surface s (u, v):

&
'
Px = u T P
v
x

(8)

The transforms between the different function spaces are

summarized in figure 3.
IV.

D EFECT D ETECTION

After recall of the variable surface model with a 3D measurement, the result is a B-spline surface description adapted
1813

+1"2'30-'+1"-0-&#
1$*4+0#0
1&#&("!$.#*
+,()&-.',,"!/)0'#)!&
23*"'$.0()&*$*

Fig. 4. Segmentation algorithm with multiple distance thresholds di and a

maximum relative area change criterion amax . Bold green lines denote resulting
segmented areas.

!"#$!%!&'()*'#)!&
!+#,!-!.&'$/01
23*"'$.0()&*$*

Input: image D(x, y)

Output: list of segmented regions RN
Parameters: number of binarization thresholds N
binarization thresholds d1 , d2 , ..., dN
relative area grow threshold max

4'"$1&-&56!78-5
9"'&+2!"0
!"#$%&'()&*$*
:)0-&+)!&'()#;
"-:13#)!&

Fig. 3. Overview of the function spaces (symbolized visualization on the

right) and necessary transforms used for the proposed variable B-spline model.

1
2

to the measurement with regards to the typical tolerances

implicitly given by the tolerance samples. Now the distance
between each of the measured 3D points and the adapted Bspline surface is computed. For a quadrilateral measurement
grid, this results in a regular distance image.

3
4

Ideally, only surface defects significantly deviate from

the surface model. However for real world measurements
slight systematic differences in surface shape appear especially
around edges or design ridges. In addition to measurement
noise, this obstructs the recognition of small defects in the
micrometer range, making additional image processing necessary.

5
6
7
8

A. Segmentation

9
10

Given a single surface measurement, a simple distance

threshold is sufficient for defect segmentation. However, the
choice of the binarization threshold is crucial for correct
segmentation. A low value might be necessary to detect small
defects, but also leads to unnecessary over-segmentation. For
defects close to design ridges or edges, a higher value is needed
to distinguish the defect from systematic shape deviations due
to an insufficient surface approximation.

11
12
13
14
15

R00 {(x, y)|x, y} ; /* initialize to complete image */

for i 1 to N do
/* find connected regions in binarized image,
*/
threshold(di :
0
D(x, y) < di
;
1
D(x, y) di
& '
{r1 , r2 , ..., rMi } connectedRegions D ;
/* test for overlap between regions on binarization
threshold di and di1 :
*/
for n 1 to Mi and m 1 to Mi1 do
i1
if rn Rm
then
i1
if area(rn )/area(Rm
) < amax then
add rn to list of regions Ri ;
else
i1
add Rm
to list of regions Ri ;
end
end
end
end
return RN
D(x, y)

Fig. 5.

For robust defect segmentation, consideration of multiple

binarization levels d1 < d2 < ... < dN is necessary. We
propose a multi-threshold segmentation algorithm outlined in
figure 4. It is motivated by hierarchical watershed segmentation
[22] with an additional relative region area criterion (figure 4).
A new region is segmented if its area on binarization level di
changes significantly compared to its overlapping region on
the previous level di1 .

Pseudocode for the multi-threshold segmentation algorithm.

B. Features and Classification

For the regions returned by the segmentation algorithm
the feature vector is constructed consisting of several 2D
morphological features (e.g. area, major and minor axis length,
circularity) as well as 3D based features. A simple defect
model is given by a 2D gaussian distribution

The algorithm is parametrized by the binarization thresholds d1 , ..., dN and the allowed relative area grow max . For
max = 0 the algorithm output is equivalent to binarization
at the lowest level d1 , max = 1 favors binarization at dN .
A value of max = 0.5 has been proven reasonable. The
choice of di values depends on typical defect characteristics.
N = 2...3 different thresholds are mostly sufficient to segment
a range of small, medium and severe defects.

*
)
(y y0 )2
(x x0 )2

e(x, y) = h exp
2x2
2y2

(9)

and can be fitted to the region distance data using a leastsquares approach. It is reasonable to normalize the center
1814

(mm)
Fig. 7. Color coded distance map of the measured 3D point data for one
sample part including 4 dents. Systematic model deviations are visible along
the design ridge and corners, making defect recognition difficult.
Fig. 6. Example visualization of one measured surface patch of a car front
hood containing a design ridge.

position x0 , y0 as well as the width properties x , y relative

to the region width. The residual of the 2D gaussian fit is
also used as a feature, amounting to an 18 dimensional feature
vector.
A linear Support Vector Machine as well as a SVM with a
radial basis kernel function (Gaussian RBF kernel) have been
implemented to further improve defect recognition. To allow
for good generalization, the classifier can be trained with defect
data from different parts.
V.

Fig. 8. Result of the proposed segmentation algorithm at binarization levels

d = [8m, 15m] applied to the distance map given in figure 7

E XPERIMENTAL R ESULTS

For the system evaluation, measurements of a car front

hood were available with 106 defect-free tolerance samples
for training of the surface model and 21 defective parts with
a total of 127 defects (dents, bumps and pits). The defects
have been manually assessed by professional auditors. Each
part measurement was split into 9 surface patches covering a
surface area of approximately 350 350 mm each with 3D
points given on a 1 1 mm measurement grid. Measurements
have been conducted with a photogrammetric 4-camera setup
using pattern projection (combined gray code and phase shifting) and a given typical depth resolution of 10 m.

(mm)
Fig. 9.

Distance map with four recognized defects after classification.

For one example patch of the front hood, the output

at different stages of the proposed system is visualized in
figures 6 - 10: The measured 3D data of the patch, the
distance map after comparison to the surface model, the output
of the segmentation algorithm, the recognized defects after
classification and the 3D data of one defect.
For comparison with the proposed multi-threshold segmentation, the results of single threshold binarization is given in
figure 11. At d1 = 8m, three of four defects are correctly
segmented, the fourth dent is occluded by systematic deviations caused by the design ridge. At d2 = 15m this defect is
correctly segmented, but two smaller defects are no longer visible. By combining these results with the proposed algorithm
(figure 8), segmentation of all four defects is possible.

Fig. 10.
300.

3D visualization of a single surface dent. Data is scaled by factor

good negative prediction rate is even more important, since this

indicates whether any false negative classification (undetected
defects) has occurred.

The classifiers have been tested using 10-fold crossvalidation (see table I). The kernel SVM significantly outperforms linear SVM classification, especially for positive
prediction rate. This hints to a much lower rate of false positive
classification for the kernel SVM. For practical applications, a

The difference of training samples for true positive defects

is a magnitude lower than for true negative since pseudo
defects can be easily produced and result from the imperfect
surface model. However gathering true defects is an exhausting
1815

R EFERENCES
[1]
[2]

threshold d1 = 8m

[3]

threshold d2 = 15m

[4]

Fig. 11. Result of different single binarization thresholds applied to the

distance map given in figure 7
TABLE I.

[5]

R ECOGNITION RESULTS AFTER 10- FOLD

CROSS - VALIDATION .

recognition rate
positive prediction rate
negative prediction rate

linear SVM
97.60%
55.45%
99.93%

[6]

kernel SVM
99.00%
98.17%
99.02%

[7]

[8]

task due to the required high manual labour. The imbalance

of the two classes samples may lead to potentially poor
generalization.
VI.

[9]
[10]

C ONCLUSION

A system for automated inspection of non-rigid surfaces

has been proposed, that is able to reliably recognize surface
defects of very small heights in 3D point clouds measured
by a multi-camera system with pattern projection. The main
problem of modeling acceptable tolerance ranges is solved by
a variable B-spline model that is trained using a number of
tolerance samples of the surface. We further proposed a multithreshold segmentation algorithm that improves detection of
defects close to surface edges and ridges. For the subsequent
defect classification, a Support Vector Machine with a Radial
Basis Kernel Function is proposed.
The reliability of the system could be proven, especially
for rejecting pseudo defects resulting from imperfections of
the surface model, leading to a correct recognition rate of
99.00%. However, the number of false negative decisions is
comparably high, which is crucial for online application in
a quality control system. Thus for further evaluation of the
system, a larger number of defect samples is desired. Besides
labor-intensive manual auditioning, this can be achieved by
synthetic generation of samples. Future work also includes
investigation of ensemble classifiers, which promise better
handling of data set imbalance. Finally, a detailed analysis
of the used feature space will be performed for improved
understanding of the system.
ACKNOWLEDGMENT
This work is part of the project Oberflaecheninspektion
auf Basis angepasster Oberflaechenmodelle (no. 6060190106)
funded by the federal state of Saxony-Anhalt and the European
Regional Development Fund (EFRE). 3D data sets have been
kindly provided by INB Vision AG.

[11]
[12]
[13]

[14]

[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]

1816

M. Kuhn, Qualitatsanspruche bezuglich des a ueren Erscheinungsbildes von Automobilen der Premiumklasse, Ph.D. dissertation, TU
Munchen, 2009.
Z. Sarosi, W. Knapp, A. Kunz, and K. Wegener, Detection of
surface defects on sheet metal parts by using one-shot deflectometry
in the infrared range, in Infrared Camera Applications Conference
2010, Las Vegas, USA, 2010, pp. 243254. [Online]. Available:
http://www.inframation.org/
H. Loferer, Automatic painted surface inspection and defect detection,
Proceedings SENSOR 2011, pp. 871873, 2011.
E. Lilienblum and B. Michaelis, Optical 3D Surface Reconstruction
by a MultiPeriod Phase Shift Method, Journal of Computers, vol. 2,
Jan. 2007.
Y. Li and P. Gu, Free-form surface inspection techniques state of the
art review, Computer-Aided Design, vol. 36, no. 13, Nov. 2004.
S. Karbacher, J. Babst, G. Hausler, and X. Laboureux, Visualization
and Detection of Small Defects on Car-Bodies, in . . . Modeling and
Visualization, 1999.
A.
Ohl,
Modellierung
von
Gestaltmerkmalen
fur
die
industrielle
Oberflacheninspektion,
ser.
ZESSForschungsberichte.
Shaker,
2005.
[Online].
Available:
http://books.google.de/books?id=GJdZAwAACAAJ
B. Denkena, H. Ahlers, F. Berg, T. Wolf, and H. Tonshoff, Fast
inspection of larger sized curved surfaces by stripe projection, CIRP
Annals-Manufacturing Technology, vol. 51, no. 1, pp. 499502, 2002.
R.-J. Recknagel, Defekterkennung an Oberflachen mittels Waveletmethoden, Ph.D. dissertation, Universitat Jena, 2000.
M. Germani, F. Mandorli, M. Mengoni, and R. Raffaeli, Cad-based
environment to bridge the gap between product design and tolerance
control, Precision engineering, vol. 34, no. 1, pp. 715, 2010.
A. Weckenmann and J. Weickmann, Optical inspection of formed sheet
metal parts applying fringe projection systems and virtual fixation,
Metrology and Measurement Systems, vol. 13, no. 4, pp. 321334, 2006.
A. E. Jaramillo, P. Boulanger, and F. Prieto, On-line 3-D system for the
inspection of deformable parts, The International Journal of Advanced
Manufacturing Technology, vol. 57, no. 9-12, Dec. 2011.
A. Aidibe, A. Tahan, and G. Abenhaim, Dimensioning control of nonrigid parts using the iterative displacement inspection with the maximum normed residual test, International Conference on Theoretical
and Applied Mechanics, Jan. 2011.
G. N. Abenhaim, A. Desrochers, and A. Tahan, Nonrigid parts
specification and inspection methods: notions, challenges, and recent
advancements, The International Journal of Advanced Manufacturing
Technology, vol. 63, no. 5-8, pp. 741752, Nov. 2012.
T. Lilienblum, P. Albrecht, and R. Calow, Dent detection in car bodies,
Pattern Recognition, Jan. 2000.
S. von Enzberg and B. Michaelis, Surface Quality Inspection of
Deformable Parts with Variable B-Spline Surfaces. DAGM/OAGM
Symposium, 2012.
A. Eichhorn, D. Girimonte, A. Klose, and R. Kruse, Soft computing for
automated surface quality analysis of exterior car body panels, Applied
Soft Computing, vol. 5, no. 3, pp. 301313, Mar. 2005.
C. Doring, A. Eichhorn, and X. Wang, Improved classification of
surface defects for quality control of car body panels, Fuzzy Systems,
2006.
K.-R. Koch, Three-dimensional NURBS surface estimated by lofting
method, Int. Journal of Advanced Manufacturing Technology, vol. 49,
pp. 10591068, 2010.
K. Qin, General matrix representations for B-splines, The Visual
Computer, vol. 16, pp. 177186, 2000.
A. Redd, A comment on the orthogonalization of B-spline basis
functions and their derivatives, Statistics and Computing, pp. 17,
2011.
S. Beucher, Watershed, hierarchical segmentation and waterfall algorithm, in Mathematical morphology and its applications to image
processing. Springer, 1994, pp. 6976.