8/27/2014

Automatically Generated C Test Cases Earn a Solid Return on Investment

print | close

Automatically Generated C Test Cases Earn a Solid Return on

Investment
Electronic Design
Adnan Hamid
Wed, 2014-08-27 12:09

In the history of functional v erification for complex chips, increasing automation has replaced tedious and
ex pensiv e manual effort. Register transfer lev el (RTL) languages hav e replaced the manual generation of
zeroes and ones, and testbench languages then automated it. RTL constructs and then flex ible scoreboards
replaced results checking v ia wav eforms. And, standard assertion and cov erage formats hav e replaced
inline hardware description language (HDL) code to check for interesting corner cases. The latest step in
this process is happening now, as automatically generated C test cases replace handwritten tests for
embedded processors.

Related Articles
Interview: Adnan Hamid Addresses Trends In Chip Verification
The Verification Flow Can Enable Horizontal Reuse
The Fundamentals of Thread Visualization for Test Case Understanding and Debug
T esting the Lim its of the UVM
Constrained-random stimulus was a big step forward, and the technique was standardized as the Univ ersal
V erification Methodology (UV M). Howev er, the UV M tries to v erify deep behav ior in a complex design
purely by manipulating the chip inputs. This can be likened to pushing on a rope, try ing to make the far
end of the rope go around corners and into small spaces with limited control. V erification teams hav e begun
to wonder if there is a way to pull on the rope and somehow guide the input stimulus from the desired
outcomes.
A related limitation of the UV M arose when sy stem-on-chip (SoC) designs began appearing. The defining
characteristic of SoCs is that they include one or more embedded processors. These processors ty pically
program the v arious intellectual property (IP) blocks and control the flow of data around the chip. Since the
power of the SoC resides in its processors, try ing to v erify such a chip without lev eraging the processors is a
challenge. Try ing to set up an embedded processor to do useful work from the chip inputs is the ultimate
pushing on a rope problem and one that few SoC v erification teams attempt.
Figure 1 shows a small but representativ e SoC, one that might be found in a digital camera. It includes two
embedded processors and a v ariety of peripherals and memories connected to the processor bus. It can
take an image from the camera, display it on the screen, and sav e it to an SD card in raw (bitmap) or JPEG
compressed format. It also can retriev e a prev iously sav ed photo from the SD card, conv ert back from JPEG
http://electronicdesign.com/print/eda/automatically-generated-c-test-cases-earn-solid-return-investment

1/6

8/27/2014

Automatically Generated C Test Cases Earn a Solid Return on Investment

if necessary , and show the image on the screen. All of the IP blocks are programmed by the processors to
cooperate and create scenarios that reflect real use cases for the camera.

The most common UV M-based solution to v erify ing such a chip is to remov e the processors, replacing
them with a model for the processors bus. The result is a traditional UV M testbench in which a UV M
v erification component (UV C) replaces the processor bus. The UV M approach calls for the UV Cs, each of
which includes a sequencer to generate constrained-random stimulus, to be coordinated by a v irtual
sequencer (Fig. 2). A scoreboard keeps tracks of the results and may gather cov erage metrics as well.

http://electronicdesign.com/print/eda/automatically-generated-c-test-cases-earn-solid-return-investment

2/6

8/27/2014

Automatically Generated C Test Cases Earn a Solid Return on Investment

The problem with this approach is that its hard to replicate processor bus cy cles for realistic user cases
without an embedded processor to actually run code. For ex ample, to JPEG-encode and store a new photo,
the processors must:
Program the camera to receiv e an image from the testbench and transfer it to memory v ia direct
memory access (DMA)
Program the photo processor to read the raw image from memory v ia DMA, conv ert it to a JPEG with
the selected quality and file size, and write it to memory v ia DMA
Program the SD card controller to read the JPEG image from memory v ia DMA and write the result to
the testbench
Giv en definitions of the chips registers and memory , its easy to imagine writing a C routine that performs
this sequence. Its harder to imagine sending sequences to the bus UV M to produce the same bus cy cles.
Most of the time, the SoC v erification team includes a few embedded programmers who write test code to
run on the processors in simulation. This is a manual and tedious process for a single processor and
v irtually impossible when multiple processors or ev en multiple threads on one processor are inv olv ed.
Humans are not good at thinking and programming in parallel.
The other problem is that neither the UV M nor any other constrained-random methodology prov ides
guidance on how to connect the embedded processors with the testbench (Fig. 3). The user scenario of
storing a new photo has two additional steps. Somehow the camera UV C needs to know when to supply an
image to the design so the camera block can grab it and store it. Similarly , the SD card UV C needs to know
http://electronicdesign.com/print/eda/automatically-generated-c-test-cases-earn-solid-return-investment

3/6

8/27/2014

Automatically Generated C Test Cases Earn a Solid Return on Investment

when to receiv e the JPEG image, compare it to the ex pected result, and model store it on the SD card.

V erification teams who want their manual C tests to interact with the UV Cs then must create a non-standard
mechanism to coordinate the code and testbench. This takes effort and, quite often, embedded
programmers write only C tests that manipulate the internal data paths without sending data on or off chip.
Clearly , this leav es a large gap in the SoC v erification effort and reduces realistic user scenarios that can be
run. A test might v erify that the photo processor can properly conv ert bitmaps to JPEG, and v ice v ersa, but
this is not an end-to-end use case.
Bey ond the UVM with C T est Case Generation
As happened during the prev ious bottlenecks in the ev olution of chip v erification, the answer for SoCs is to
automate a process that had been manual. In fact, technology ex ists today to automatically generate
sophisticated C test cases that can run on embedded processors in simulation. Figure 4 shows the Breker
solution where test cases are generated from graph-based scenario models that document the SoCs
v erification space and the teams v erification plan. The test cases are compiled, loaded into SoC memory ,
and run just like handwritten tests or production software.

http://electronicdesign.com/print/eda/automatically-generated-c-test-cases-earn-solid-return-investment

4/6

8/27/2014

Automatically Generated C Test Cases Earn a Solid Return on Investment

This automated approach offers sev eral adv antages. The test-case generator keeps track of parallel threads
and programs, so the C tests it produces can run with multiple threads on multiple processors. The
scheduler is aggressiv e at mov ing user test cases from thread to thread, processor to processor, and
memory region to memory region. This ensures thorough ex ercise of paths between all processors and all
allowable memory regions, which is ex cellent for helping to v erify the SoCs cache and memory coherency
scheme.
The solution includes a simulation run-time element that communicates with the UV Cs. Along with the C
code, the generator produces an ev ents file with a list of ev ery thing that has to happen in the testbench
during the test case run. When the C code gets to a point where testbench coordination is needed, it sends a
simple message with an ev ent ID v ia a memory -mapped mailbox . The run-time element looks up this ID in
the ev ents file and takes the appropriate action. This mechanism prov ides sy nchronization between the
running test case and the UV Cs with no user effort at all.
In the prev ious ex ample of sav ing an image, when the test case is ready to grab the image from the camera
port, it will send a message and the camera UV C will respond. Likewise, when the JPEG image is ready to be
sent to the SD card UV C, another message triggers this action as well as a comparison of observ ed and
ex pected results. To minimize embedded memory usage, input data and ex pected results are stored in the
simulator and not in the SoC. Likewise, all comparisons happen in the simulator to max imize the
performance of the embedded processors and the SoC.
The C test cases can be generated from a single scenario model not just for simulation, but also for
acceleration, emulation, FPGA prototy ping, and ev en actual silicon in the lab. Each stage of generation can
be tuned for best results in the target platform. Graph-based scenario models are also reusable from IP
blocks through complete chips. Graphs are simply instantiated to form a scenario model for the nex t lev el
http://electronicdesign.com/print/eda/automatically-generated-c-test-cases-earn-solid-return-investment

5/6

8/27/2014

Automatically Generated C Test Cases Earn a Solid Return on Investment

in the design hierarchy . Finally , automatic analy sis of the graphs rev eals any unreachable portions and
generates the test cases needed for cov erage closure on the remainder.
Assessing Return on Inv estm ent
As chip v erification has adv anced v ia automation of prev iously manual steps, some amount of new effort is
required for each enhancement. It might be a new language to learn, new knowledge to acquire, or a new
methodology to embrace. For automated generation of C test cases, the new effort is the specification of
graph-based scenario models. Any v erification team considering the adoption would and should ask how
this new effort compares to the gain in productiv ity , improv ement in design quality , reduction in time-tomarket, and increased likelihood of first-silicon success.
Fortunately , scenario models are not hard to dev elop. They look like a standard data flow diagram for the
SoC, although flipped so results are on the left and inputs are on the right (Fig. 4, again). The generator then
can start with each possible outcome, trace a path to determine how to make that outcome happen, and
produce a test case following the path. This pulling on the rope approach makes it easier to v erify deep
design behav ior. Importantly , there is no new language to learn. Scenario models are specified in standard
C/C++ with only a few ex tensions based on Backus-Naur Form (BNF), another standard language.
Repeated ex perience has shown that a v erification engineer or embedded programmer can start writing a
scenario model after only a day of training and generating interesting cases in well under a week. Thus,
sy ntax is not a barrier, but the user does hav e to accurately capture the chips v erification intent, and the
knowledge required is almost the same knowledge needed to write manual C tests. In the digital camera
ex ample, any one working on the v erification team knows how to program each IP block to do its job and
how to string the blocks together in user case scenarios.
For about the same amount of effort as handwriting a test or two, a user can produce a scenario model that
can generate an unbounded number of test cases, each one far more complex than what could be written by
hand. One leading cell-phone SoC company reduced the number of v erification team members writing
embedded tests from 1 0 to two because of the increase of productiv ity with test case automation. The other
eight engineers were not laid off. They were redirected to work on production software and applications
that would add v alue to the final product.
Conclusions
The ev olution of chip v erification has been the process of replacing ex pensiv e manual work by automated
processes. The nex t step for the industry is eliminating the need to hand-code tests for v erification or
prototy pe bring-up and the problem of pushing on a rope.
Leading SoC teams today are automatically generating multi-threaded, multi-processor, self-v erify ing C test
cases that stress the design much more thoroughly than manual methods. The minimal inv estment is
compensated many times ov er by better productiv ity , a shorter v erification schedule, and increased design
quality .

Source URL: http://electronicdesign.com/eda/automatically-generated-c-test-cases-earn-solid-returninvestment

http://electronicdesign.com/print/eda/automatically-generated-c-test-cases-earn-solid-return-investment

6/6