Face Detection

Technical overview

Overview
The Microsoft Face Detection system provides a solution that can automatically detect faces in still images and real time video feeds. The system can detect an arbitrary number of faces at any scale and location. The system takes photographic images or a video stream as input. The output consists of an array of rectangles which corresponds to the location and scale of faces detected. If it detects no faces, it will return an empty array. The following graphic illustrates the Face Detection system:

In the first step the system uses an adaptive pre filter to eliminate candidate rectangles in the input that it can confidently determine do not contain faces.

2005 Microsoft Corporation. All rights reserved. The information contained in this document relates to Microsoft Research pre-release/protot pe products and technologies! "hich ma #e su#stantiall modified #efore their final commercial release # third part $% venture companies. Accordingl ! the information ma not accuratel descri#e or reflect the products and technologies "hen first commerciall released. This document is provided for informational purposes onl and Microsoft ma&es no "arranties! e'press or implied! "ith respect to this document or the information contained in it. Reproduction and redistri#ution of this document re(uires the e'press permission of Microsoft Corporation.

In the second step the system feeds the remaining rectangles to an improved implementation of an algorithm called !"oosting #hain.$ "oosting #hain is a framework for rapid ob%ect detection developed by Microsoft &esearch 'please see the (Documentation) section below for a paper detailing "oosting #hain*. Finally, in the third step the system attempts to eliminate false positives. It applies a color filter and an edge filter to improve the precision of the detection. It uses a support vector machine '+,M* filter, plus lighting correction and histogram e-uali.ation, to further eliminate false positives. The system then outputs each rectangle, which contains a detected face, along with its confidence in each detection.

Components
The Face Detection algorithm can be divided into three parts: /. 0re filtering stage: This stage includes pre color filters and non face skipping algorithm. 1. "oosting filtering stage 2. 0ost filtering stage: This part includes +,M classifier, 3dge boosting filter, post color filter and eye component filter. Face Detection algorithm structural overview:
nput image
( r e p r o c e s s i n g f in-plane detection feature enabled

Compute integral image for color filter and boosting filter

Compute two rotated integral image for color filter and boosting filter

( r e f i l t e r i n g

CrCb pre -color filter for frontal faces )on -face s*ip filter

RGB pre-color filter for profiled / tilted /ext faces )on -face s*ip filter

RGB pre-color filter for profiled / tilted /ext faces )on -face s*ip filter

RGB pre-color filter for profiled / tilted /ext faces )on -face s*ip filter

f i l t e r i n g

b o o s t i n g -

Boosting Chain classifier for frontal faces

"xt Boosting Chain classifiers for frontal faces

! Boosting Chain classifiers for profile faces

! Boosting Chain classifiers for tilted faces

CrCb post -color filter for frontal faces

( o s t f i l t e r i n g

RGB post -color filter for profiled / tilted /ext faces

RGB post -color filter for profiled / tilted /ext faces

RGB post-color filter for profiled / tilted /ext faces

"#e filter

$%& filter

"#e filter

"#e filter

"#e filter

$%& filter

"dge filter

"dge filter

&erge

Final 'utput

2005 Microsoft Corporation. All rights reserved. The information contained in this document relates to Microsoft Research pre-release/protot pe products and technologies! "hich ma #e su#stantiall modified #efore their final commercial release # third part $% venture companies. Accordingl ! the information ma not accuratel descri#e or reflect the products and technologies "hen first commerciall released. This document is provided for informational purposes onl and Microsoft ma&es no "arranties! e'press or implied! "ith respect to this document or the information contained in it. Reproduction and redistri#ution of this document re(uires the e'press permission of Microsoft Corporation.

In order to simplify the face detection problem, the system divides the target faces to be detected into four categories: upright frontal face, e4tend frontal face set with difficulty illumination5lighting and e4pression, profiled face 'left profile and right profile*, and in plane rotated face 'clockwise and anti clockwise*. Three sets of filters are applied to each kind of face. 6fter the filtering stage, regions detected are merged to generate final result.

Core Scenarios
In a video conference scenario, a passive camera with a wide angle lens provides a global view of the persons seated around a conference table. 6n active camera provides a close up of whichever person is speaking. Face detection, operating on the wide angle images, can be used to estimate the location of conference participants around the table. 7hen a participant speaks, the high resolution camera can .oom onto the face of the speaker and hold the face in the center of the image. In the real world usage of video surveillance, multiple cameras are utili.ed in a wide spectrum of applications for the purpose of monitoring. 6s a result, multiple monitors are used to display the various output video streams. Manual monitoring of multiple screens is tedious and operators are prone to fatigue. Face detection can help to reduce the data from multiple screens into one main screen for monitoring. "etter surveillance systems: Face Detection is the first step for face recognition.

Documentation
Microsoft &esearch has published several papers on the face detection algorithms: /. &ong 8iao, Ming%ing 9i, :ong%iang ;hang. &obust Multipose Face Detection in Images . IEEE Transactions on Circuits and Systems for Video Technology '#+,T*, ,olume /<, 06&T/, 1==<. 1. 9ongbin #hen, 9ei ;hang, :ong%iang ;hang, and Mohamed 6bdel Mottaleb. 2D +hape #onstraint Facial Feature 9ocali.ation >sing 0robabilistic like ?utput. 6ccepted by @th IEEE International Conference on Automatic Face and Gesture Recognition 'FA*, +eoul, Borea, 1==<. 2. 9ongbin #hen, 9ei ;hang, 9ong ;hu, Ming%ing 9i and :ong%iang ;hang, 6 Covel Facial Feature 0oint 9ocali.ation algorithm using 0robabilistic like output , in 0roc. of Asian Conference on Computer Vision '6##,*, De%u Island, Borea, 1==<. <. &ong 8iao, 9ong ;hu, and :ong%iang ;hang. "oosting #hain 9earning for ?b%ect Detection , in 0roc. of International Conference on Computer Vision 'I##,*, Cice, France, 1==2. E. +tan.;. 9i, 9ong ;hu, ;heng-iu ;hang, 6ndrew "lake, :ong%iang ;hang, :arry +hum, +tatistical 9earning of Multi ,iew Face Detection , In Proceedings of The th European Conference on Computer Vision '3##,*, #openhagen, Denmark, May, 1==1. In addition, Microsoft &esearch has created an internal document describing the details of the Face Detection prototype implementation, as well as a 0ower0oint presentation summari.ing the highlights of the Face Detection solution. "asic, 60I level documentation for the +DB e4ists as well.

Ca!eats
This algorithm will achieve its best recall rate when the face is upright frontal with non e4aggerated e4pression, simple lighting and illumination condition. The more a face varies from the above conditions, the worse the recall rate will be.

Related Work
0lease refer to the (Documentation) section above for published papers detailing related work.

2005 Microsoft Corporation. All rights reserved. The information contained in this document relates to Microsoft Research pre-release/protot pe products and technologies! "hich ma #e su#stantiall modified #efore their final commercial release # third part $% venture companies. Accordingl ! the information ma not accuratel descri#e or reflect the products and technologies "hen first commerciall released. This document is provided for informational purposes onl and Microsoft ma&es no "arranties! e'press or implied! "ith respect to this document or the information contained in it. Reproduction and redistri#ution of this document re(uires the e'press permission of Microsoft Corporation.

Differentiating Features
The three step detection algorithm achieves both high detection rate and high precision with unprecedented speed performance. It can successfully detect faces with significant lateral and polar rotation.

Demo/Prototype De!elopment Status

The Face Detection techni-ue is mature and well tested on a large scale dataset. +till image detection prototype screenshot:

The real time face detection prototype indicates the detection of a face by overlaying diving goggles.

2005 Microsoft Corporation. All rights reserved. The information contained in this document relates to Microsoft Research pre-release/protot pe products and technologies! "hich ma #e su#stantiall modified #efore their final commercial release # third part $% venture companies. Accordingl ! the information ma not accuratel descri#e or reflect the products and technologies "hen first commerciall released. This document is provided for informational purposes onl and Microsoft ma&es no "arranties! e'press or implied! "ith respect to this document or the information contained in it. Reproduction and redistri#ution of this document re(uires the e'press permission of Microsoft Corporation.

:ere is a screenshot of the prototype in action:

Internal Projects
The system has been used in a research pro%ect regarding video conferencing.

Technical Specifications
Implemented in #FF #urrently optimi.ed for /=1<4G@H video streams.

2005 Microsoft Corporation. All rights reserved. The information contained in this document relates to Microsoft Research pre-release/protot pe products and technologies! "hich ma #e su#stantiall modified #efore their final commercial release # third part $% venture companies. Accordingl ! the information ma not accuratel descri#e or reflect the products and technologies "hen first commerciall released. This document is provided for informational purposes onl and Microsoft ma&es no "arranties! e'press or implied! "ith respect to this document or the information contained in it. Reproduction and redistri#ution of this document re(uires the e'press permission of Microsoft Corporation.