Black Box Software Testing

2004

Academic Edition

PART 22 -- SCRIPTING: AN INDUSTRY WORST PRACTICE

Alternate title: PROCEDURALISM: THE ENEMY OF GOOD TESTING
Cem Kaner, J.D., Ph.D.
Professor of Software Engineering
Florida Institute of Technology
and
James Bach
Principal, Satisfice Inc.

Copyright (c) Cem Kaner & James Bach, 2000-2004

This work is licensed under the Creative Commons Attribution-ShareAlike License. To view a copy of this
license, visit http://creativecommons.org/licenses/by-sa/2.0/ or send a letter to Creative Commons, 559
Nathan Abbott Way, Stanford, California 94305, USA.
These notes are partially based on research that was supported by NSF Grant EIA-0113539 ITR/SY+PE:
"Improving the Education of Software Testers." Any opinions, findings and conclusions or
recommendations expressed in this material are those of the author(s) and do not necessarily reflect the
views of the National Science Foundation.
Black Box Software Testing

Copyright

2003

Cem Kaner & James

Scripting

COMPLETE SCRIPTING is favored by people who believe that

repeatability is everything and who believe that with repeatable
scripts, we can delegate to cheap labor.
Here's the kind of detail you often find in scripts
1 ____PulldowntheTaskmenu
2 ____SelectFirstNumber
3 ____Enter3
4 ____Enter2
5 ____Pressreturn
6 ____Theprogramdisplays5
The idea (the fantasy) is that the script specifies the test so
precisely that the person following it can be a virtual robot, doing
what a computer would do if the test were automated.

Black Box Software Testing

Copyright

2003

Cem Kaner & James

Scripting:Anotherscriptformat
Step
#

Check What to
?
do

What to
see

Design
notes

1.

____

Task menu
down

This starts the

blah blah test,
with the blah
blah goal

Pull down
task menu

Observation
notes

ThisistheformatthatI(Kaner)usedtouseaWordStar,longago
(backwhenIthoughtscriptingwasagoodidea).

Black Box Software Testing

Copyright

2003

Cem Kaner & James

Scripting:Perceivedbenefits
Proponents of highly scripted tests believe that scripting offers several
benefits:
Repeatability:Differenttesters,followingthesamescript,willdothesame
thingsandachievethesameresults.Thisprovidesalevelofprocesscontrol
fortesting.
Scalabilityofexpertise:Oneexperttesterwritesthescriptsandjunior
testersexecutethem.Weleverageourbestpeople.
Abilitytoaddinexpensivestaff:Thejuniorscostlessthanthesenior.
Supportsoutsourcedtesting:Cheaplaborfarawaycanfollowthescripts.
Abilitytoaddstafflateintheproject:thescriptsprovidestructureforthe
testerwhoisunfamiliarwiththeproduct.Iftheprojectschedulefalls
behind,wecanaddtestersandkeepthembusywithouttoomuchtraining
cost.
Training:Thescriptactsas"trainingwheels"forthenewtester.After
severalmonthsoffollowingawiderangeofscripts,thenewtesterwill
havelearnedbyexamplealotabouttheprogramandhowtotestit.

Black Box Software Testing

Copyright

2003

Cem Kaner & James

So,whatswrongwithscripting?

Black Box Software Testing

Copyright

2003

Cem Kaner & James

Letstryanexample:Asimplearithmeticexercise
Problem

We have a vat of marbles and three jars, with the

following capacities

Obtain this
many marbles

21

100

Black Box Software Testing

127

Copyright

2003

Cem Kaner & James

Asimplearithmeticexercise
Problem

We have a vat of marbles and three jars, with the

following capacities

Obtain this
many marbles

21

127

100

14

163

25

99

Black Box Software Testing

Copyright

2003

Cem Kaner & James

Asimplearithmeticexercise
Problem

We have a vat of marbles and three jars, with the

following capacities

Obtain this
many marbles

21

127

100

14

163

25

99

142

121

Black Box Software Testing

Copyright

2003

Cem Kaner & James

Asimplearithmeticexercise
Problem

We have a vat of marbles and three jars, with the

following capacities

Obtain this
many marbles

21

127

100

14

163

25

99

142

121

18

43

10

Black Box Software Testing

Copyright

2003

Cem Kaner & James

Asimplearithmeticexercise
Problem

We have a vat of marbles and three jars, with the

following capacities

Obtain this
many marbles

21

127

100

14

163

25

99

142

121

18

43

10

23

49

20

Black Box Software Testing

Copyright

2003

Cem Kaner & James

10

TheArithmeticExercise
What we see is that people adapt to procedures.
Once in the mindset of a set procedure
peoplearelikelytomissorignoreblatantlyobvious
informationthattellsthemthat:
a different procedure would be more efficient (in this
case)
this procedure is not applicable to this situation
something else happened while they were following
a procedure, that isnt within the scope of
consideration of this procedure

Black Box Software Testing

Copyright

2003

Cem Kaner & James

11

Thiscontrastswithacommonassumption
The assumption
Whensomeoneisfollowingastandardprocedure,theyllkeep
theireyesopenandnoticeunusualthings,noticevariation,et
cetera

Black Box Software Testing

Copyright

2003

Cem Kaner & James

12

Thiscontrastswithacommonassumption
The assumption
Whensomeoneisfollowingastandardprocedure,theyllkeep
theireyesopenandnoticeunusualthings,noticevariation,et
cetera
Have you ever noticed?
Whathappenswhenyoudrivetoanewplace
from a map, or in some other way, making your own way
with a navigator who says, turn left next light etc

Ifyoutrytogothereagain,doyouremembertheroute?

Black Box Software Testing

Copyright

2003

Cem Kaner & James

13

Thiscontrastswithacommonassumption
The assumption
Whensomeoneisfollowingastandardprocedure,theyllkeep
theireyesopenandnoticeunusualthings,noticevariation,et
cetera
The contrast
Theexamplessuggeststhereverse,thatpeoplegetlockedinto
procedures,focusmoreonfollowingtheinstructionsthanonthe
systembeingworkedthroughthoseinstructions.
My assertion
Whenpeopleembraceprocedures,theyoftenbecomeautomatons.

Black Box Software Testing

Copyright

2003

Cem Kaner & James

14

Thiscontrastswithacommonassumption
The assumption
Whensomeoneisfollowingastandardprocedure,theyllkeep
theireyesopenandnoticeunusualthings,noticevariation,et
cetera
Inattentional Blindness
Flatlycontradictsourhappyassumption.
seeSightsUnseen,
http://www.apa.org/monitor/apr01/blindness.html
seethedemonstrationsat
http://viscog.beckman.uiuc.edu/djs_lab/demos.html
seethediscussionsathttp://psyche.cs.monash.edu.au/v7/
seetheargumentthatitislackofexpectationratherthanlackof
attentionathttp://psyche.cs.monash.edu.au/v7/psyche706
braun.html
Black Box Software Testing

Copyright

2003

Cem Kaner & James

15

AnotherProceduralismExample
HowdoFornixFimbria Study compared normal rats with rats with
fornix-fimbria lesions that destroyed the
LesionsAffectOneWay
hippocampus.
ActiveAvoidance
Behavior?
Through a series of pilot studies, the
experiment simplified and simplified until it
was finally a big white box (with an
CemKaner,Bob
electrical grid) for a floor and a smaller
Osborne,HarveyAnchel,
black box, with a metal door between
MarkHammer&
them.
AbrahamH.Black
Rats were placed in the white box, given a
signal, and had to move to the black box
McMasterUniversity
within 5 seconds or get shocked.
Normals appeared to know where they
86thAnnualConvention
were and where they wanted to go. The
oftheAmerican
lesioned rats appeared to have learned a
PsychologicalAssociation
fallible procedure for escaping.
Toronto,Canada,August <See the paper, and/or the demonstration
in class.>
28,1978
Black Box Software Testing

Copyright

2003

Cem Kaner & James

16

Scripting:Thetesters'tourbus
Scripting is the Greyhound Bus of software testing:

Just relax and leave the

thinking to us.
Tothenovice,thetestscriptisthewholetour.Thetestergoesthroughthe
script,starttofinish,andthinkshesseenwhatthereistosee.
The novice follows the script.

Totheexperiencedtester,thetestscriptisatourbus.Whensheseessomething
interesting,shestopsthebusandtakesacloserlook.
The skilled tester breaks away from the script.
The skilled tester can also notice problems that the script writer didn't
anticipate, and make comparisons that the script doesn't call for. An
automated test is very different from a human test,
In effect, the script hides what the skilled tester is doing.

One problem with a bus trip. Its often pretty boring, and you might
spend a lot of time sleeping.
Black Box Software Testing

Copyright

2003

Cem Kaner & James

17

Scriptingdoesn'tliveuptoitspromise

Notrepeatable:Differentpeoplefollowingthesamescriptdoandnoticedifferent
things.
Whatexpertiseareyouscaling?Thepersonwhowritesthetestsimaginesasubset
ofthepossibleissuesand,bythenatureofscripting,islimitedtoasmallsetoftests
thatwillberepeatedoverandover.SeeourdiscussionofGUIlevelregressionthe
repeatedtestslackpower.Worse,thescript'sfocusonthesameexpectedresultevery
timesteersthescriptuserawayfromnoticingotherdimensionsof(mis)behavior.
AllofthedisadvantagesofGUIlevelcapture/replayautomation:hardtomaintain,
nomodularity,expensivetocopewithUIchanges,notlikelytoexposemanybugs.
Itmissesthecriticaladvantageofautomation:Ifyou'rerunningabunchoftests
thatareunlikelytoyieldmuchinformation(becauseyou'verunthembefore),atleast
withautomatedtests,youcanminimizethetimeahumanspendsrunningthem
(additionally,theautomatedtestscanbebetterdesignedformaintainability).
Nocommunicationofanalysis:Scriptsvaryonthisissue,butmostfocuson
procedurewithoutexplainingthepointofthetest.Neitherthetesterwhofollowsthe
scriptnortheonewhomaintainsitknowswhatwasinthemindofthetestdesigner.
Ineffectivewaytoeducate:Stepbystepproceduralinstructionisapoorwaytoteach
someoneabouttheprogramorabouttesting.It'snothowpeoplelearn.

Black Box Software Testing

Copyright

2003

Cem Kaner & James

18

Scripting
Chip Groeder created scripts as pseudocode, in preparation for
test automation.
Won the Best Paper award at STAR 97
But he found most (88%) of the bugs during creation of the
scripts (my experience too).
Maybe the bug-finding benefit lies in structured exploration,
rather than fully-specified regression.

Black Box Software Testing

Copyright

2003

Cem Kaner & James

19

NeverDoANYkindofScripting?
Checklists (as distinct from scripts) have their place. For
example, think of releasing a product:
Manyverydifferenttasks
Allofthemmustbedone
Thetaskisrarelydone,somanystepsmaybeforgotten.
I prefer to tell testers what to test (what issues to cover), not
how to do the tests. Teaching people how is a matter of
training, not something that I record time and time again in the
test plan. A checklist will sometimes be the right way to present
the list of issues.
I think the level of detail is sufficient if I can successfully pass
the section to a reasonably experienced tester who is a little
familiar with the program and be confident that she can figure
out what the test cases are and how to run them.

Black Box Software Testing

Copyright

2003

Cem Kaner & James

20