Effectiveness of Online Assessment
Dr. Denise Woit Dr. David Mason
Ryerson University
Toronto, Ontario
{dwoit,dmason}@ryerson.ca
Abstract
For five academic years we have engaged in an on -going
study of the effectiveness of online assessment of student
programming abilities for introductory programming
courses in Computer Science. Our results show that online
evaluation can be implemented securely, efficiently, and can
result in increased student motivation and programming
efficacy; however, unless onli ne components are integrated
throughout the course evaluations, student competence will
be underestimated. Our data reveals disadvantages of online
evaluations, but also shows that both students and faculty
benefit when online evaluations are implemented
appropriately.
1. Introduction
We were motivated to begin online evaluation of our
students because we hoped to improve student programming
efficacy and measure it more accurately. Through
discussions with senior students, T.A.s and other instructors,
and through our personal observations that actual
programming skills of some students were much poorer than
predicted by their course work scores —especially
assignment and laboratory exercises—we became concerned
that copying of these exercises was a significa nt problem
among our first year students. It appeared plausible that
students could obtain excellent assignment and laboratory
marks through little or no practical work of their own, and
could manage to concatenate enough memorized code
fragments on writte n examinations to achieve a “good”
grade in the course through part -marks. We sought the
ability to clearly identify those students who had not
engaged in the amount of practical course work we
expected. Furthermore, we hoped that our improved
evaluation techniques would motivate students to participate
more earnestly in the practical components of the course.
2. Related Work
Our speculation and anecdotal evidence concerning cheating
on assignment and laboratory work among our students is in
keeping with past and recent international studies. In
Sheard’s recent study [12] of 287 undergraduate
students—137 in Computer Science and Software
Engineering at Monash University, and 150 in
Information Technology at Swinburn University —
students report it accepta ble to copy a majority of an
assignment from a friend, to submit a friend’s assignment
from a past running of the course, to hire a person to write
an assignment for them, or to resubmit an assignment
from a previous subject in a new subject. More
specifically, 34%/28% said they had personally copied the
majority of an assignment from a friend, and 53%/38%
said they personally knew someone who did this. They
reported 52%/42% personally collaborated on an
assignment meant to be completed individually, and
75%/63% personally knew someone who did this. Also,
77/79% said they would do nothing if they observed
another student cheating on an assignment. Results from
this survey support our own anecdotal evidence and study
data.
Sheard [12] found their results “b roadly in agreement”
with similar studies. For example, a study [11] of 791
undergraduates found that 90% would not report cheating
by others, and in a more recent study [3], 49% -86%
agreed with this notion. Several studies report a high
number of studen ts who admit to cheating. A study of
422 students [10] reports 92% admit to being involved in
incidents of “academic misconduct”, while in studies
[9,5] of 943 and 500 undergraduates, 88% and 90%,
respectively, admit to cheating. In another study [6] of
380 undergraduates, 54% reported they cheated, while
only 1% reported being caught cheating.
Since we began our online evaluation studies, a number of
other educators have reported similar studies, although on
a smaller scale. Chamillard [2] reports incorpo rating 2
online “lab practica” into course work for approximately
500 students enrolled in an introductory Computer
Science course at the U.S. Airforce Academy. Califf [1]
reports instituting an online “laboratory exam” in CS1
over two years, with 200 -280 students per year. English
describes an end -of-year online exam for 64 students in a
first year programming course [4]. In the Chamillard and
Califf studies, the online tests were aimed at assessing
only students’ practical skills, and were given in addit ion
to the customary course tests and exam. In the English
study, the online evaluation constituted the course final
exam, and contained multiple -choice and short -answer
questions in addition to programming questions. As
described more fully in the sequel , these studies generally
support our own findings about the effectiveness online
evaluations; they also hint at the weaknesses of online
evaluations revealed by our data.
Study Implementation
Over a total of five academic years we have studied
incorporating online components into our first year language
course. This course is administered alongside CS2, and its
main objective is to impart practical skills in Unix shell
usage, in Unix shell programming, in the procedural
paradigm, and in the C language, all of which are essential
in future courses. When our study began, approximately 150
students were enrolled in this course; the enrolment has
increased over the years to approximately 210 in the most
recent year.
We employ a multiple -case study design, with each trial is
studied and analyzed independently, as well as in relation to
the others. The studies each include an online final exam;
however, they vary in whether laboratory assignments are
marked or voluntary, and in the frequency of other online
evaluations, which ranged all the way from having no other
online components at all to having online weekly quizzes
worth a majority of course marks. We assessed our results in
relation to student efficacy, student stress, student
motivation, instructor evaluation time and satisfaction.
We differentiate among our four individual case studies
according to their configurations of evaluations and
laboratory assignments as follows:
-Partially marked laboratories, no midterm. (PML-N)
-Voluntary laboratories with online midterm, (VL-M)
-Marked laboratories with online midterm, (ML-M)
-Online weekly laboratory quizzes, no midterm, (WQ-N)
Figure 1. Online final exam marks
3. Study Results
Student online final exam marks in each study, depicted in
Figure 1, form a starting point for our comparative analysis;
we also evaluate data from a variety of other sources: from
other quantitative benchmarks of student performance, from
student and instructor surveys, and from student, T.A., and
instructor anecdotal evidence. We incorporate results from
similar studies where appropriate [1, 2, 4].
3.1 Partially Marked Laboratories With No Midterm (PML-N)
The most dismal performance occurred with partially
marked laboratory exercises and no midterm. In this s tudy
students were given laboratory exercises to work on each
week, with T.A.s available during their regularly scheduled
laboratory sessions. We selected a portion of every second
laboratory to be marked, so that a total of 6 submissions
were made over the term. Student survey results show that
students generally did not attempt the voluntary
laboratory exercises, but worked merely on the questions
that were to be marked. Although the voluntary
laboratory exercises accounted for 90% of assigned
coursework, the majority of students reported allocating
them just 0 -25% of the total time they spent working on
laboratory exercises.
Evidence also suggests that a significant amount of
copying occurred. Student survey results show that for
marked laboratory exer cises, 87% of students, at some
point, gave their completed questions to other students
before the due date, while 72% had used other students’
completed questions to help them with their own. In fact,
48% of students reported giving away their assignments
frequently, and 38% reported using others’ assignments
frequently. Incredibly, students reported that online
evaluations motivated them to cheat less on their
submitted laboratory questions.
Further evidence of the copying problem lies in the
comparison between student laboratory grades and final
exam grades, shown in Figure 2. Since the final exam
questions were taken directly from the marked laboratory
exercises (some with slight modification), it is reasonable
to expect similar grades for the final exa m and the marked
laboratory questions. However, as shown in Figure 2,
85% of students received A -level grades and none failed
in the marked laboratory component, whereas in the final
exam only 21% of students obtained A -level grades and
30% failed.
The re sults from Figure 1 show this study has the largest
number of failures and D -level grades, and the smallest
number of A -level grades. Clearly, this study was not
successful from the point of enhancing student efficacy
and motivation in the practical components of the course.
Figure 2. Final exam vs. laboratory marks (PML-N)
3.2 Voluntary Laboratories with Midterm (VL-M)
The next -to-worst results occurred when laboratory
exercises were completely voluntary, but an online
midterm w as included (VL -M). Evidence from student
and T.A. surveys suggests students did not devote time to
the voluntary laboratory exercises and thus were very
unprepared for the online midterm test. Figure 3 shows
the distribution of online midterm marks for this study
compared to the online final exam marks. A majority of
students report their poor performance on the midterm
subsequently motivated them to learn enough practical
course material to achieve a higher mark in the online final
exam. A majority also report that familiarity with the online
testing process, obtained via the midterm, also helped them
achieve a higher mark on the final exam.
Figure 3. Midterm vs. final exam marks (VL-M)
It is interesting to compare the midterm marks of these
students (VL-M) with the final exam marks of the students
in the “worst” group (PML -N), since these are the first
online evaluations of the term for each group. Both marks
are generally poor; however, students in VL -M got a
“second chance” b ecause this was the first of their two
online evaluations.
Comparing data from these two studies (PML -N and VL -
M), we conclude that there is little difference between
having completely voluntary and mostly voluntary
laboratory exercises, because our studen ts tend to ignore
voluntary assignments. Our results, and corroborating
results from Califf [1] and English [4], show a limitation of
online testing is that students will tend to perform poorly on
the first online evaluation. Therefore, it is essential to have
at least one online evaluation prior to the final exam because
it is this evaluation that motivates students to take the
practical aspects of the course more seriously and provides
them experience working in the online testing environment.
3.3 Marked Laboratories with Midterm (ML-M)
The second -best student performance occurred when all
laboratory exercises were marked, and an online midterm
test was given. Figure 1 shows that 40% of students
achieved a final exam mark in the A -range, and 22% failed.
Final exam grades in this study were a marked improvement
over those of PML -N and VL -M. Comparing these three
studies, we note that even though students may cheat, it
appears preferable to require practical work to be submitted
frequently, as opposed to rarely or not at all.
Students and instructors were pleased with the results of this
study, but the instructors were not satisfied. We noted that
with online midterm and final evaluations, a substantial
number of students, 30% -40%, now achieved marks in the
A-range, but the same number also failed the course or
achieved D -range marks. We found it especially
discouraging that the failing and marginal students often did
poorly on exam questions that were taken directly from their
laboratory exercises. Survey r esults showed cheating was
reduced but not eliminated by online evaluations, and this
was clearly evident among the marginal and failing
students. In fact, in ML -M, of those students who
received F-level or D-level grades on the final exam, 77%
had achieve d an A -level grade on marked laboratory
exercises, 17% a B -level grade, and 6% a C -level grade.
Not surprisingly, these were the students who felt least
that online evaluations motivated them to stop cheating.
We hoped to target these marginal students wi th our WQ-
N study.
3.4 Online Weekly Laboratory Quizzes With No
Midterm (WQ-N)
This study included 12 online quizzes (one per week) as
well as the online final exam. Laboratory exercises were
voluntary; however, at the end of each week, portions of
the laboratory exercises for that week were selected for an
online quiz. Because the scheduled laboratory time was
devoted to the online quiz each week, we scheduled 12
hours per week of free in -laboratory T.A. consulting time.
A more detailed description of th is study is presented in
[14].
Given our previous success with enhancing motivation by
employing online evaluations, we predicted even more
success with this study. We expected that students would
be motivated to complete and understand the weekly
laboratory exercises, and to utilize the scheduled T.A.
hours. We suspected that perhaps students would not
immediately realize that doing and understanding the
laboratory work was essential, but that after failing one or
two initial quizzes, they would reform an d begin the
diligent laboratory work we expected. We were wrong.
Most students failed miserably on the first four quizzes;
on quizzes 5 and 6 about half were passing, but with very
poor marks. On the midterm quiz, the average grade was
in the D -range. Evi dence showed that students had not
been working on their laboratory exercises at all. We
obtained this evidence by asking concerned students
submit all of their “rough” laboratory work, and finding
none had any work to submit. T.A records showed only a
handful of students had used the consulting service during
the first half of the term.
In order to motivate students to begin working on their
laboratory assignments, we decided to count only the last
6 quizzes toward their final mark. The students worked
diligently in the second half of the term, heavily utilizing
T.A. support. Only 7% of students obtained an overall
failing grade in the quizzes, and 42% achieved an A -level
grade. Figure 4 shows students’ overall quiz grades as
well as their final exam grade s. We believe quiz grades
are a good indication of students’ practical knowledge
because of the high correlation between quiz marks and
final exam marks.
In Figure 1, the final exam marks in this WQ -N study are
compared to those of the other studies. Note that the
implementation of online weekly quizzes dramatically
reduced failure rates. Students who would have otherwise
failed moved into the C - and D -ranges. Students from the
D- and C -ranges moved into the A - and B -ranges. In the
WQ-N study, students ach ieved a mark approximately
one full letter grade above those in other studies, and
because of our secure, online evaluations, we were certain
that these students deserved their grades. When all 4 studies
are compared, it appears unnecessary to require stud ents to
submit laboratory assignments for grading when weekly
online quizzes of laboratory content are employed.
However, lacking such frequent online evaluations, it is
preferable to require frequent submission of laboratory
assignments.
Figure 4. Quiz and final exam marks (WQ-N)
4. Discussion of Results
Results of student surveys from the studies show that
students believe conventional tests are more likely to pass or
assign higher marks to students who lack practical skills in
the course. They think online tests are a better indicator of
their practical skills, and that tests of these skills could not
be achieved as well by conventional evaluations. They feel
that student marks on online evaluations are a good
indication of whether or not students cheated on marked
laboratory assignments. A majority report that online tests
motivate them not to cheat on marked laboratory
assignments, and motivate them to attend class and
laboratory sessions. We found a s light positive correlation
between students’ marks and their motivation not to cheat.
The poorer the student mark, the less motivated they feel not
to cheat on marked laboratory exercises, although they are
still more motivated than with conventional tests
Interestingly, survey results are similar among the various
studies. There are no significant correlations between a
student’s course mark and any survey question.
Student/instructor opinions and test results from our studies
lead us to conclude that ha ving an online final exam, with at
least one prior online evaluation, has the advantage of
bringing students to a higher standard, of reducing the level
of cheating and copying, and of encouraging students to
attain the practical skills expected of them in the course. Our
results show that students are most likely to attain the
practical skills expected of them when the course contains
frequent, online evaluations. When the only online
evaluations are the midterm and final exam, students’
practical skills a re better when laboratory exercises are
graded frequently.
Because these studies took place in different years, we
hypothesised that students might perform increasingly well
on the laboratory exercises, and thus online tests, as the
years progressed, because they would have access to an ever
increasing set of solutions from students of previous years.
Even if students do have access to solutions, it does not
appear that this results in higher test scores. The most
recent study was PML -N, in which students showed the
lowest overall performance of all studies. The second -last
study was WQ -N, in which students showed even lower
performance than PML -N in the first half of the term.
Student performance, however, did increase to the overall
best of the studies in the last half of the term. It is clear
that this resulted from students working on the exercises
themselves; it is unclear whether or not these students did
have solutions and, if so, the extent to which this
contributed to their ability to do the exercis es themselves.
We plan further study in this area.
Through anecdotal evidence and student surveys, we
found that student stress was a significant factor in our
studies. Both Califf and Chamillard [ 1,2] noted this as
well. Chamillard reports that in future classes, the number
of online evaluations will be increased from 2 to 3 partly
to reduce the “stress associated with taking each
practicum.” In the first year of the Califf study, they
found that students “felt major stress taking the exam”
because student s had to write complete programs from
scratch and debug them, but they had little experience in
these areas. In the subsequent year, the instructors
modified the laboratory exercises so that these topics were
more thoroughly covered, and instituted severa l short
quizzes in these areas during the term. The authors report
students were “much less stressed” in the second year,
and achieved higher grades in the year-end online test.
Our data shows students believe they would feel less
stress on the online fina l exam if they had the “practice”
of an online midterm. Following this logic, we expect
students in our WQ-N study to suffer the least online final
exam stress, since they had 12 previous online tests that
term; we expect the most online final exam stress to have
occurred in PML-N, when students had no practice before
the online final exam. Survey results from the PML -N
study show that a majority of students found our online
final exam more stressful than traditional exams. There
was also a significant neg ative correlation between (a)
. stress felt and (b) believing their mark was higher because
the test was online (vs. traditional). Unfortunately, we
cannot directly compare this with QW -N because that
survey did not include questions relating to stress.
However, other data is consistent with our expectations:
Approximately 15% of students in the PML -N study
complained to the instructor that they did not work to
their potential because of the stress of the online test,
while there were no complaints in the WQ -N study.
Anecdotal evidence concurs: In the WQ -N study, none of
the students appeared particularly stressed, while in the
PML-N study about 25% of the students appeared
stressed during the final exam, and about 5% seemed to
panic at some point. Our eviden ce, as well as that of
Califf and Chamillard [1, 2], implies that in order to
mitigate the variable of stress in our studies, we must
provide at least one online evaluation before the final one.
5. Study Environment
We attempted to control extraneous variab les in our
studies by having similar online testing and laboratory
environments, similar test questions, similar laboratory
assignments, and by having the same instructor mark all
online evaluations for all studies.
The online test environment for the WQ -N, ML-M, and VL-
M studies was identical and text -based. In the PML -N
study we moved to a GUI environment matching the
students’ usual setting. The effects of using a matching
environment are unclear, as environmental results are
conflated with other effect s. Student opinion, however, is
that having the matching environment is preferable.
The online test questions varied slightly, but were “the
same” in that they were taken from the students’ laboratory
exercises, some with slight modification. The laborator y
exercises were almost identical in all the studies.
6. Automatic Evaluation
Automatic marking in this course is especially valid because
it is simply a practicum course, and we do not grade on
elements such as style, algorithms, or documentation, as
would a CS1 course. We are concerned mainly with code
being configured correctly and producing correct output. In
the WQ-N study, all of the online quizzes were marked in a
fully automated test harness, while the online final exam
was marked manually. We bel ieve the high correlation
between these marks, shown in Figure 4, is evidence of the
accuracy of our automatic marking harness.
Even when marking “manually” the instructor employed a
semi-automated test harness, which executed students’ code.
When the test harness reported correct output, the code then
required only a brief, visual inspection, which reduced
overall marking time. Programs that did not pass all tests, or
did not compile, were marked in the normal fashion,
although online. Other studies with o nline evaluation
reported similar reductions in marking time [1, 2, 4]. In
other work we more fully report on our methods of online
evaluation [7, 13, 14], and on the online marking
environment we developed [8, 15].
8. Conclusions
Students and instructor s agree that online testing of
students’ practical skills provides a more accurate measure
of student efficacy. This opinion is supported by the data we
have collected over five academic years, comparing student
performance on online tests in a variety of scenarios. The
data shows that it is imperative to provide more than one
online evaluation per session; this mitigates the effects of
student stress, it allows the students to “practice” before the
final test, motivates them to acquire the expected skills, and
results in a more realistic score on the final online
evaluation. Students best attain the expected skills when
many online evaluations are incorporated into the course,
and in this case, traditional, non -online evaluations are
unnecessary. Instructo rs and students believe that our
online tests hold students to a higher standard and motivate
them to strive to achieve a higher level of practical
competency in our course.
References
[1] Califf, M.E. and Goodwin, M. Testing skills and
knowledge: Introdu cing a laboratory exam in CS1.
SIGCSE Bulletin, 34.1 (2002), 217-221.
[2] Chamillard, A. T. and Joiner, J. K. Using Lab
Practica to Evaluate Programming Ability. SIGCSE
Bulletin, 33.1 (2001), 159-163.
[3] Cole, S. and McCabe, D.L., Issues in academic
integrity., New Directions for Student Services,
Jossey-Bass Publishers, (1966), pp. 67-77.
[4] English, J. Experience with a computer -assisted
formal programming examination. Proc. 7 th Annual
Conference on Innovation and Technology in
Computer Science Educat ion (2002, Aarhus,
Denmark) 51-54.
[5] Graham, A.M., Monday, J., O’Brien, K. and Steffen,
S., Cheating at small colleges: an examination of
student and faculty attitudes and behaviours, Journal
of College Student Development, 35 (1994) 255-260.
[6] Haines, V.J., Diekhoff, G.M., LaBeff, E.E. and Clark,
R.E., College cheating: Immaturity, lack of
commitment, and the neutralizing attitude, Research
in Higher Education, 25 (1986) 342-354.
[7] Mason, D. and Woit, D. Integrating Technology into
Computer Science E xaminations. SIGCSE Bulletin,
30.1 (1998), 140-144.
[8] Mason, D., Woit D., Abdullah, A., Barakat, H., Pires,
C. and D’Souza, M. Web -based evaluation for the
convenience of students, markers, and faculty. Proc.
N.A.Web’99 Conference, (October 1999).
[9] Newstead, S.E., Franklyn -Stokes, A. and Armstead,
P., Individual differences in student cheating, Journal
of Educational Psychology, 88 (1996) 229-241.
[10] Roberts, P., Anderson, J. and Yanish, P., Academic
misconduct: where do we start?, Northern Rocky
Research Association, Jackson, Wyoming, (1977).
[11] Schemo, D.J., Degree of dishonour. The Age ,
Melbourne, 2001, pp. 16.
[12] Sheard, J., Dick, M., Markham, S., Macdonald, I. and
Walsh, M. Cheating and plagiarism: Perceptions and
practices of first year IT s tudents. Proc. 7 th Annual
Conference on Innovation and Technology in
Computer Science Education (2002, Aarhus,
Denmark) 183-187.
[13] Woit, D. and Mason, D. Lessons from On -line
Programming Examinations. SIGCSE Bulletin, 30.3
(1998), 157-259.
[14] Woit, D. and Mason, D. Enhancing student learning
through on -line quizzes. SIGCSE Bulletin, 32.1
(2000a), 367-371.
[15] Woit, D. and Mason, D. Evaluation Methods for In-
Context, Consistent Comments and Hierarchical
Marking. In Bruce Mann (Ed.), Chapter 13,
Perspectives in web course management. (2000b).
Canadian Scholars’ Press. ISBN 155130-143-1