A system for traceable photography.

Provisional patent application filed Jan 200

1, abandoned Jan 2002, placing my rights to the invention in the public domain.
I think a professor at Princeton filed a similar patent in Sep 2000.
Video Authentication Steganography
Inventor: Shanti Rao

Background
Photographic steganography (the hiding of information within a picture) is typic
ally used for storing copyright information within a still picture (U.S. Patents
5,687,236 and 5,613,004) or for embedding control signals within a video stream
(U.S. Patent 6,026,193).
As it becomes possible to alter images in real-time, the authenticity of video i
mages may be called into question. Thus, there is clearly a need for methods for
the authentication of video, allowing for authentication and detection of alter
ations to a video.
Summary
The invention provides a system and methods for placing a digital signature on v
ideo, to allow for authentication and detection of alterations within the digiti
zed video.
The system includes calculating a message digest for a digital video image,
encrypting the digest with the encoder's (e.g. a camera) secret key with hardwar
e or software, and transmitting the the digest and image as part of a video. The
video can be received. The system also can authenticate the video and can notif
y a user or viewer of alterations made to the video after encoding.
The digital signature is created within digital video data by an encoder, for ex
ample, a video camera. The digital signal also can be created after capture of a
n image, by a video encoding device. As used herein, the term camera is to be in
terpreted to include a video encoding device which processes an acquired image.
An integrated circuit controls the image sensor of a digital video camera. A sui
table image sensor can be, for example, a CCD sensor, a CMOS sensor or a hybrid
CMOS-CCD sensor. Each encoder has a unique identification number and public key
pair. As it reads pixel data from the image sensor, this circuit uses digital st
eganography techniques to encode information into a watermark for each frame. Th
e message stored in each frame can include:
The sequence number of the frame, encrypted with the camera's secret key. The ca
mera has a persistent memory device to count each frame. Only the low-order bits
could be stored, if necessary, to reduce image degradation.
A message digest describing the picture, encrypted with the camera's secret key.
The message digest algorithm is described below.
Information about the digest algorithm. We expect that digest algorithms will im
prove over time, and so a few bits of information in the message will identify t
he digest algorithm.
Additionally, information which does not vary from frame to frame may be stored
in partially in each frame, including:
The ID number of the camera.
The public key of the camera.
A message digest of the accompanying audio (which may not be synchronized perfec
tly with the video).
The high-order bits of the frame sequence number.
Information about the watermark algorithm.
Frame encoding
An image watermark consists of a low-level distortion pattern spread over the im
age, encoding a digital message. Watermarks, which are not detected by people, c
an be read by a computer. They have been used primarily for placing copyright in
formation within an image.
The encoder calculates the digest of the frame, then bundles this with other inf
ormation to create the frame message. The watermark is applied so that the messa
ge is stored over the entire image, ensuring that the decoder can detect a cropp
ed frame.
A decoder reads the watermark message from several frames and extracts the camer
a's public key. It then calculates a digest of the message, and compares this to
the digest and sequence number in the watermark, decrypted with the public key.
The decoder can detect:
Re-sequenced frames by comparing the frame sequence numbers
Cropped frames from problems in reading the watermark
Altered segments, resolution, or retouching of the image by comparing the digest
s.
Digest algorithms
Video authentication steganography is relevant for broadcast over lossy channels
, such as compressed digital video or analog broadcast. Thus, the digest algorit
hm is designed to detect major changes in the picture, and is not the same as a
digital signature.
The digest is sensitive to:
Replacement of a portion of the image
Degraded resolution
Rearrangement of the image

The invention also provides a method of steganographically marking a video.

The method includes acquiring a digital video image from an image sensor. The ac
quisition of a digital video image is well known to one of skill in the art and
can be accomplished with a number of techniques and camera systems. The camera c
an be a digital camera having a digital image sensor. The image sensor can be, f
or example, a CCD sensor, using a variety of techniques known to one of skill in
the art. Such acquisition techniques also can be applied to acquire images usin
g other image sensors. Suitable digital image sensors include CMOS detectors and
CMOS-CCD hybrid detectors. Video images also can be acquired using an analog ca
mera with an analog to digital conversion. A variety of preferred embodiments ar
e shown in the table below.

Detailed Description
The invention provides a method for placing a digital signature within a digitiz
ed video. According to the invention, a digital video image is acquired, for exa
mple, using a camera equipped with an image sensor, for example, a CCD. A video
image also can be acquired from an analog source through analog to digital conve
rsion. The video images can represent color images or black and white images.
The image is stored, by pixels in a suitable memory device including, for exampl
e, static, dynamic, flash, magnetic core, persistent, volatile memory devices. T
he pixels can be stored using any suitable strategy including, for example, inte
rlaced or noninterlaced formats.
The image is digitally processed using software running on a processor or coded
into the integrated circuit to calculate a message digest. The message digest ca
n represent: the amount of information that can be stored in the image with the
steganography algorithm without reasonably altering the appearance of the image;
the date and/or time that the image was acquired; the camera ID; or information
derived from the image itself.
The message digest can represent the date and/or time the image was acquired, an
d how the clock was set with information obtained from a clock in the camera, fr
om an external clock, synchronized with a GPS or other time reference set by the
operator. The message digest also can represent the unique camera id which can,
for example, be hard-coded into the camera during manufacture, laser marked on
a chip, uniquely etched on the chip, stored in ROM, stored in a Programmable Gat
e Array, or etched into the PCB.
The message digest also can include the frame number. Frame numbers can be assig
ned directly by the camera circuit numbers every image it takes. The numbering o
f frames can be sequential or not sequential. It can, for example, be a pseudora
ndom sequence based on the camera id. The numbering system also can start indexi
ng at any number, which could be a function of the camera id, date, and/or time.
Using such a system, when the camera stops recording, it causes a break in this
sequence, allowing detection of discontinuous camera shots. When the camera res
umes recording, it resumes the sequence. The frame index, or the number used to
generate the frame index, is stored. Image storage can be accomplished using any
technique available to one of skill in the art. Images can be stored, for examp
le, in flash or other nonvolatile memory; on a disk; in volatile memory. When an
image is stored in volatile memory, it is generally preferred that the image da
ta be moved to nonvolatile memory when the device loses power.
The message digest also can include information derived directly from the image
itself. Such image derived data include, for example, using a digital checksum o
f the brightness and color of the image. Such a digest can be calculated at seve
ral image resolutions, so that a later detect loss of detail can be detected, fo
r example, using an algorithm that can allow detection of changes to the image,
regardless of overall change to brightness, overall change to contrast or minor
degradation from uniform noise. Sources of noise include, for example, minor deg
radation from noise typically found on TV signals (diagonal black lines, etc....
), station watermarks on the corners or borders of the image, horizontal or vert
ical squeezing, to make room for ads on the TV screen.
The message digest also can be calculated using a periodic checksum function, in
cluding , for example, a discrete or continuous sine or cosine transform, or a w
avelet transform. The periodic checksum function also can be generated using a p
seudorandom or nonrepeating pattern that can be generated, for example, by a cyc
lic polynomial function (like CRC) or can be retrieved from a stored table. The
message digest also can be calculated by measuring the noise and information con
tent of the image from the spectral density of the image.
After calculation of the message digest, the digest is then encrypted with the c
amera's secret key using, for example a DSP or a specialized circuit. The can be
used with the RSA public key, or any other public key algorithm.
The system can then bundle the digest and the digital image together. The bundli
ng process can be implemented by a specialized chip or by a DSP handling the enc
ryption. When the video data is bundled for transmission as a digital video, the
message digest can be transmitted along a different channel from the video stre
am. If the desired data format uses lossy digital compression, like JPEG, the di
gest may be added between lossy and non-lossy stages, so that the message digest
can be reconstructed without data loss. The digest also can be overlayed onto t
he image with a steganography algorithm. When the video file is to be transmitte
d as an analog video, the digest is preferably overlayed oonto the image with a
steganography algorithm.
A digital video image thus encoded can be transmitted as part of a video stream
for digital video in a compressed or uncompressed form. When the video stream is
to be sent in digital form, the data can be compressed by sending only the diff
erences between successive frames, or where each frame is compressed independent
ly. The video stream can be transmitted Along parallel wires or in serial on a w
ire, through fiber optic cable, wirelessly. The digital vidoe signal also can be
modulated and demodulated, using digital compression, error correction, and det
ection.
A digital video image also can be transmitted as analog video with pixel values
converted to voltage & current or using a digital-to-analog converter (R/2R ladd
er, digital pot, multiplexer, etc.). An analog video signal can be transmitted o
n wires (e.g., differential or single-ended, coaxial cable. analog fiber optic,
paired cables , R/G/B/Sync, Y/L/C/Sync). The signal also can be broadcast in a v
ariety of formats. The signal can be encoded, modulated, transmitted, demodulate
d, decoded, can be demodulated and re-modulated at a relay or can be digitized,
compressed, and transmitted in packets. Such a video signal can be converted bac
k into an analog or digital TV signal for display.
According to the invention, the encoded video data can be transmitted to a devic
e that records and/or replays the video or transmits it to viewers. The encoded
video image can be squeezed or stretched, so that it appears in a little box on
a screen, or with a visible mark to indicate ownership (e.g., a station or netwo
rk logo) or an invisible mark to indicate ownership.
The encoded video can be displayed to a device including a television, computer
projector, through an analog or digital video recorder. Authentication of an enc
oded video of the invention can be accomplished with such a device by reading th
e message digest. When the video is transmitted as an analog signal, the video i
mage is first digitized with an A/D converter, a DSP or specialized circuit or w
ith an algorithm in software.
reading the steganographic messages on the video images
optionally removing these messages from the video images, for example, through d
igital subtraction or through image degradation, to discourage copying

The video data can be validated, for each frame, by: determining if the message
digest was validly constructed with a known digest algorithm, for example, by re
-calculating the message digest, using the same algorithm as in the encoding ste
p. The video data also can be validated by inspecting the digital signature on t
he digest, and determining if it was signed by the circuit that generated the di
gest. Alterations to the video data can thus be detected by comparing the messag
e digest to the video image to detect: image frames presented out of sequence; c
hanges to the images that can be detected with the digest algorithm; which porti
ons of the frame, if any, were altered. Such a comparison also can detect the am
ount of image quality loss, which could be used to disguise an alteration from t
he "information content" field of the message digest, from calculating the amoun
t of information that can be stored in the image, using steganographic technique
s.
The system can then notify the user or viewer whether or not the video was alter
ed after it was captured with the camera or after encoding by the encoding syste
m. Such notification can include notification if only portions of the image have
been altered, which portions, of how much information has been lost in transmis
sion (wich may indicate alteration) and can include to whom the camera belonged
by looking up the camera id in a database on a computer. The system also can ext
ract the date and time at which the pictures were taken, and the reliability of
the date/time. The system also can detect if any frames were rearranged. The sys
tem also can detect if any frames from different sources were put together.

The system can, for example, notify the user by overlaying text on the video ima
ge being displayed, which can turned on or off by control panel or computer inte
rface, or by transmitting information to a secondary image display device (compu
ter, TV, projector) or by showing a copy of the image, with authentic or non-aut
hentic areas highlighted, displayed, or removed or by re-ordering frames that we
re transmitted out of order. The system also can store authentication data in a
computer file with the digitized image or with tracking numbers for synchronizin
g with the video stream. The system also can store authentication data on an ana
log storage device, for example, magnetic tape, with the authentication data sto
red digitally alongside the video stream.

Particular embodiments of the invention are summarized in the following table:

Sequence of events:
Acquiring a digital video image
From a CCD
pixel-at-a-time
binned pixels
line-at-a-time
FET on chip
FET off chip
A/D on chip
A/D off chip
From a photodiode camera, digitized
SAR A/D
Flash A/D
Hybrid A/D
One pixel at a time
Multiplexed image sensors
Color
Multiple chips with filters
Multiple chips with dichroic beamsplitters
Single chip with filter overlay
Single chip with separate filter
Linear combination of pixel values
RGB
YLC
Other color mode
B&W
Linear
Analog gamma correction
Digitized from an analog video signal
NTSC
PAL
Other encoding method
Various A/Ds
storing, by pixels, in a memory device,
static, dynamic, flash, magnetic core, persistent, volatile
interlaced, noninterlaced
by horizontal lines
by vertical lines
RGB pixel values (usually 8bits/channel)
YLC pixel values
digitally processing
general purpose microcontroller, DSP or ASIC
from a program stored in memory
dynamic memory (static, dynamic, flash, ...)
EEPROM
EPROM
flash card
hard disk
coded into the IC
calculating a message digest, which may represent:
the amount of information that can be stored in the image with the steganography
algorithm without reasonably altering the appearance of the image
the date and time that the image was taken, and how the clock was set
obtained from a clock in the camera
obtained from an external clock
synchronized with a GPS or other time reference
set by the operator
the unique camera id
hard-coded into the camera during manufacture
laser marked on a chip
unique etching
stored in ROM
stored in a Programmable Gate Array
etched into the PCB
the frame number
where the camera circuit numbers every image it takes
may not be sequential. It could be a pseudorandom sequence based on the camera i
d.
can start indexing at any number, which could be a function of the camera id, da
te, and/or time.
when the camera stops recording, it causes a break in this sequence. This will a
llow detection of discontinuous camera shots.
when the camera resumes recording, it resumes the sequence. The frame index, or
the number used to generate the frame index, is stored
in flash or other nonvolatile memory
on a disk
in volatile memory, and moved to nonvolatile memory when the device loses power
the image
using a digital checksum of the brightness and color of the image
at several resolutions, so that we can later detect loss of detail
using an algorithm that can allow detection of changes to the image, regardless
of
overall change to brightness
overall change to contrast
minor degradation from uniform noise
minor degradation from noise typically found on TV signals (diagonal black lines
, etc....)
station watermarks on the corners or borders of the image
horizontal or vertical squeezing, to make room for ads on the TV screen
using a periodic checksum
discrete or continuous sine or cosine transform
wavelet transform
pseudorandom or nonrepeating pattern
generated by a cyclic polynomial function (like CRC)
retrieved from a stored table
measuring the noise and information content of the image
from the spectral density of the image
encrypting the digest with the camera's secret key using, for example a DSP or a
specialized circuit
RSA
another public key algorithm
so that the the DSP or another chip can bundle the digest and the digital image
together.
for digital video
the message digest may be transmitted along a different channel fom the video st
ream
if the desired data format uses lossy digital compression, like JPEG, the digest
may be added between lossy and non-lossy stages, so that the message digest can
be reconstructed without data loss.
overlay the digest onto the image with a steganography algorithm
for analog video
overlay the digest onto the image with a steganography algorithm
transmitting the digest and image as part of a video stream
for digital video
compressed by sending only the differences between successive frames
where each frame is compressed independently
uncompressed
transmitted along parallel wires
transmitted in serial
on a wire
fiber-optic
wireless
modulated & demodulated, using digital compression, error correction, and detect
ion
for analog video
pixel values converted to voltage & current
digital-to-analog converter (R/2R ladder, digital pot, multiplexer, ...)
transmitted on wires
differential or single-ended
coaxial cable
analog fiber optic
paired cables
R/G/B/Sync
Y/L/C/Sync
broadcast
encoded, modulated, transmitted, demodulated, decoded
demodulated and re-modulated at a relay
digitized, compressed, and transmitted in packets
eventually turned back into an analog or digital TV signal
receiving the video, which
records and replays the video
transmits it to viewers
squeezed
stretched
so that it appears in a little box on a screen
with an visible mark to indicate ownership (station or network logo)
with another invisible mark to indicate ownership
and ultimately to a device
television
computer
projector
possibly through an analog or digital video recorder
connected to one of these recording and display devices
which can read the message digest
by digitizing the image with an A/D converter
with a DSP or specialized circuit
with an algorithm in software
by...
reading the steganographic messages on the video images
optionally removing these messages from the video images
through digital subtraction
through image degradation, to discourage copying
and then, for each frame:
determine if the message digest was validly constructed with a known digest algo
rithm
re-calculate the message digest, using the same algorithm as in the encoding ste
p
inspect the digital signature on the digest, and determine if it was signed by t
he circuit that generated the digest
compare the message digest to the video image to detect
image frames presented out of sequence
changes to the images that can be detected with the digest algorithm
which portions of the frame, if any, were altered
amount of image quality loss, which could be used to disguise an alteration
from the "information content" field of the message digest
from calculating the amount of information that can be stored in the image, usin
g steganographic techniques
and notify the viewer
whether or not the video was altered after is was captured with the camera
if only portions of the image have been altered, which portions
how much information has been lost in transmission (wich may indicate alteration
)
to whom the camera belonged
by looking up the camera id in a database on a computer
of the date and time at which the pictures were taken, and the reliability of th
e date/time
if any frames were rearranged
if any frames from different sources were put together
by
overlaying text on the video image
which can turned on or off by control panel or computer interface
transmitting information to a secondary image display device (computer, TV, proj
ector)
showing a copy of the image, with authentic or non-authentic areas highlited, di
splayed, or removed
which could re-order frames that were transmitted out of order
store authentication data in a computer file
with the digitized image
or with tracking numbers for synchronizing with the video stream
store authentication data on an analog storage device
magnetic tape
with the authentication data stored digitally alongside the video stream