2012 12th IEEE International Conference on Advanced Learning Technologies

A Long-Term Evaluation of Educational Animations of Functional Programs

Jaime Urquiza-Fuentes, J.Angel Vel zquez-Iturbide a Laboratory of Information Technology in Education LITE Rey Juan Carlos University, Madrid, Spain {jaime.urquiza,angel.velazquez}@urjc.es the experiment and its results. Finally in the section VI we draw our conclusions. II. R ELATED W ORK Program animation is not a new research eld. It is partially driven by a rm belief from researchers on that animations can help students in learning programming concepts. However, existing experiments do not clearly support this intuition. Hundhausen et al. [2] conclude that the more active the educational use the better the learning outcomes, but they also highlight that there are areas that need further investigation, e.g. the use of narratives and textual contents within animations [5]. The Engagement Taxonomy [1] classies different educational uses of animations based on students engagement. It identies six different engagement levels: Nonviewing, Viewing, Responding, Changing, Constructing and Presenting. These levels are sorted by students engagement with animations, being Non-viewing the lowest level and Presenting the highest one. This taxonomy proposes that the more the engagement the better the learning outcomes. Here we briey describe those levels involved in our study. The Non-viewing level represents the typical scenario without the use of animations. In the Viewing level the students can play animations, changing their direction, pace or abstraction level. If the animation provides multiple views, switching among them is also considered in this level. Finally, the Constructing level requires from students to construct explanatory animations of program/algorithms behavior. Students need neither special construction tools nor to program and execute the program/algorithm. This section surveys related studies that evaluate the educational impact of constructing and viewing animations. A. Experiments with Animation Construction We have found ve studies related to animation construction. Stasko [6] designed assignments where students had to construct their own animations. He detected that students dedicated more time to study the algorithms for which they had constructed animations. Urquiza-Fuentes et al. [7] made a comparative study with vision tasks. Students who constructed animations used the WinHIPE IDE, while the others just viewed the animations produced by the instructor with the same IDE. Both groups of animations
26

AbstractIn this paper we study two different approaches to using program animations with educational aims: their construction by students a constructivist and active approach and their vision a less active approach. In addition, we compare both approaches to a traditional teaching methodology where animations are not used. We have conducted a longterm evaluation with functional program animations using an existing IDE with visualization features called WinHIPE. Our results are mixed. Students have a positive opinion about the animations, either viewing or constructing them. While the viewing approach improves some aspect of knowledge acquisition, both approaches decrease the fail-rate. Finally, the construction approach improves the students attitude towards the subject. Therefore, both uses of program animations should be integrated in the teaching methodology. Keywords-program visualization; engagement; evaluation;

I. I NTRODUCTION Learning to program is one of the most difcult issues in computer science curricula. Program animations, animations for the rest of the paper, are one of the existing approaches to overcome this problem. But despite of its potential, it is not widely used by computer science educators. One of the main reasons, highlighted in [1], is the lack of evidence of their educational benets. One of the most signicant contributions in eld recommends a more constructivist approach in their educational use [2]. Other studies propose to increase students engagement with animations [1]. But despite of the literature supporting the educational use of animations, results are not clear regarding either the way of using them or their key features to be effective educational tools [3]. In this work we investigate the benets of two different approaches: students construction of animations (an active, constructivist and highly engaging educational task) and students vision of animations (a less active approach). In addition, we compare the previous approaches with a traditional approach where animations are not used (the typical approach for the rest of the paper). To this end we use a previously developed tool called WinHIPE [4], a programming environment with visualization capabilities. The rest of the paper is structured as follows. In section II we survey the educational experiences about viewing and constructing animations. The section III explains both, vision and construction approaches. The sections IV and V detail
978-0-7695-4702-2/12 $26.00 2012 IEEE DOI 10.1109/ICALT.2012.50

had additional textual explanations. The results showed that students who constructed animations were longer involved with the algorithm. As a consequence, they also improved their knowledge acquisition. The following three experiments studied animation construction and presentation tasks. Hundhausen [8] compared two different tools: a well-known algorithm visualization tool, and artifacts selected by the students, ranging from slides to crafts. This observational study detected improvements in the students attitude who used their own artifacts. Based on these results, a tool to construct algorithm animations was developed and compared again with the construction artifacts selected by students [9]. In this experiment, improvements in programming performance were detected on students who worked with the new tool. Finally, H bscher-Younger et al. [10] made a comparative u study with vision tasks. In this study, a number of volunteers developed and presented animations for a set of algorithms. They were allowed to choose the construction utilities. Then, all the students, animation developers included, had to evaluate those animations this can be considered as the vision task. The results showed that the students who developed and evaluated the animations acquired more knowledge than those who just evaluated the animations. The construction utilities play an important role in construction tasks. Thus, most studies allow students to choose the construction utilities or provide them with carefully designed interfaces for this task. In addition, the use of textual contents is present in most of the studies. We must also remark that we have only found two comparative studies. Although both studies delivered positive results, their representativeness can be questioned due to their experimental design. One of them involved only fteen subjects [7]. The other [10] seems to use groups of participants biased in terms of students motivation prior to the experiment. While the evaluation (viewing) tasks were compulsory, construction and presenting tasks were voluntary, therefore students who constructed animations were probably more motivated than those who just evaluated others animations. B. Experiments with Animation Vision We have found ve experiments for this approach. All of them detected educational improvements in terms of knowledge acquisition. Most of them compare vision tasks with, at least, the non-viewing approach. Lawrence [11]s study is the only one without a comparison: it just detected improvements when animations were used with textual labels. Kann et al. [12] made a comparative study among different educational uses of animations. However, they only detected signicant improvements between the viewing and non-viewing approaches. Crosby et al. [13] detected improvements when using multimedia materials made up of visualizations and textual contents. Kehoe et al. [14] detected improvements when using animations in an environment that

simulated homework. Thus, students used animations with narrative contents to complete assignments without time limit. Finally, Kumar [15] showed that using visualizations within the feedback provided by a tutoring system improves knowledge acquisition. Furthermore, students who received feedback with animations and textual contents obtained better results than those who received feedback with just animations. In summary, narrative contents and textual explanations are used in all the experiments but one. Therefore, they seem to be features of educational tools that make animations effective, even though they deserve more research [2]. III. E DUCATIONAL A NIMATIONS WITH W IN HIPE This section summarizes the features of educational animations produced with WinHIPE [4] and their construction process; further details can be found elsewhere [16]. A. Design and Contents of the Educational Animations Our educational animations are web-based materials. Consequently, they can be easily shared, reused and published. In addition, they can be used either in theoretical sessions, lab sessions or at home. These animations consist of four components: a description of the problem, an explanation of an algorithm that solves the problem, the source code that implements the algorithm, and the program animation. The pace and direction of program animations can be controlled with a typical VCR interface. Besides, animations can simultaneously display one or two consecutive steps for better understanding. Fig. 1 shows an example of an educational animation about simple recursive data types (computing the sum of a list of integers), with all its contents visible and showing two consecutive steps in the program animation. We provide four different web formats to view our educational animations, the user can change among them anytime.

Figure 1.

An educational animation of the SumList program

27

Two of them, called Framed1 and Framed2, are inspired by general principles of web pages usability [17]. They allow the user to see, respectivelly one or simultaneously two of the four contents of the educational animation selected by the user. The third web format, called Plain, is an HTML page navigable by means of a local index and the scroll bar, where all the contents are available. Finally, the fourth format, called Star, is intended to enhance exibility of user interaction. Here the user selects the contents that will be simultaneously visible and distributes them in the screen, Fig. 1 is an example of this web format as well. B. Producing Educational Animations with WinHIPE We have paid special attention to achieve an effortless animation production process in WinHIPE. An effortless approach allows the students to produce animations focusing on educational tasks rather than on technical issues. Thus, animation construction can be used as an active educational task which could improve students learning outcomes. The construction of our educational animations is divided into two phases: generation of the program animation (the graphical representation of the execution of a program) and generation of the educational animation (the web-based materials). The phase of program animation generation starts with the user typing the source code of the program and ends with a sequence of discrete visualizations of some execution steps of the program. The execution model of the functional programming paradigm is term rewriting. In this model, running a program consists in a process of rewriting a starting term, the main expression, step by step, until no rewriting can be done. The basis of program animations in WinHIPE are the graphical representation of these execution steps, also called visualizations. The contents of visualizations can be pretty-printed text and/or graphical representations of data structures, i.e. binary trees and lists. These visualizations are automatically produced by WinHIPE. Then, the user selects which visualizations will form part of the nal program animation. She/he can easily complete this task with a specially designed interface, also integrated within the IDE. This interface provides simultaneously the user with a global view of the execution process and a detailed view of a specic execution step selected by the user. Finally, the program animation is displayed in WinHIPE as a sequence of discrete visualizations. Therefore, the generation of the program animation is quite similar to the program development process (edit-compile-execute), where additional tasks are assisted by specially designed interfaces. The educational animation is also generated through the user interface provided by WinHIPE. Thus, the user must provide the information corresponding to textual contents (title, problem description, and algorithm explanation) and look contents (appearance on the web).

Notice that user interaction is limited to building the program animation, explaining the problem and its solution, and specifying the web look of the animation. Any additional processing is automatic. A nal, automatic step generates the textual contents (HTML code) and the web look information (CSS code) and adds them to the program animation (images corresponding to the visualizations). Therefore we have an effortless educational animation construction process totally integrated into the development environment. IV. E VALUATION In this evaluation we investigate the long-term effects of the previously described treatments. A. Subjects and Experimental Design The participants where 132 novel students enrolled in a Programming Languages Foundations course placed in the rst year of CS degrees. The evaluation took place during the second half of the course, which was focused on the functional programming paradigm. Students participation was almost voluntary, and incentive based. Students can slightly improve their grades (0.25 out of 10) only if they pass the course exam. We have discarded the students who have previous knowledge about the functional paradigm. Our study has two treatment conditions for the viewing (VG, n=50) and constructing tasks (CG, n=40) and one control condition for the typical approach tasks (TG, n=42). Students were randomly assigned to the treatments/control conditions. Thus, the independent variable is the engagement level used in each group. The dependent variables are: knowledge acquisition, measured with the term exam grades, attitude towards the subject, measured with the exam taking rate, and opinion about the animations, measured with a questionnaire. B. Tasks and Protocol At the beginning of the course, students were informed about the evaluation. Previously to the treatments application, students got familiar with the WinHIPE programming environment. Also they were trained in the facilities provided by WinHIPE related with the tasks they have to complete, view or construct animations. The treatments were applied during the sessions corresponding to the topics Prex and inx operators, User dened data types and Recursive data types. For each topic, the protocol consisted in two parts: theoretical sessions, and a lab session. Theoretical sessions of all groups followed the typical teaching methodology, but sessions of VG and CG were enhanced with the use of animations for examples and simple exercises. The lab sessions of each group consisted different types of tasks but using the same problems/cases, all of them selected from the course book. The students in the control group (TG) worked on solving problems proposed by the teacher. They were allowed to use

28

WinHIPE without its visualization facilities. The students attention were focused on understanding the description of the problem and coding the solution. The main task of the vision group (VG) was viewing animations constructed by the teacher. For each topic they have a list of animations in a web page. They had to view them ensuring that they understood them. The students attention were focused on understanding the description of the problem and its solution using the animation and the textual explanations. Finally, the main task of the construction group (CG) was constructing animations following the construction treatment described in the previous section. They had available through the web all needed materials: the description of the problem and the source code. Construction tasks consisted in studying the program so they can select appropriate input data, execute the program and generate the animation using the visualization facilities provided by WinHIPE. Actually, the students effort were focused on producing a representative animation choosing suitable input data and selecting the important execution steps and writing a correct explanation of the solution. The three groups ended their lab sessions working on an exercise sheet, equal for the three groups, with theoretical questions and problems. In summary, the students participated in three evaluative sessions, completing three groups of tasks and three exercise sheets. At the end of the last lab session, students from all the groups were asked to ll in the opinion questionnaire. V. R ESULTS OF THE E VALUATION In this section we detail the results of the experiment. We have studied the existing differences among the three groups. We have used ANOVA and t-student tests for normal data sets and, Kruskal-Wallis X 2 and Mann-Whitney U for nonnormal data sets. A. Knowledge Acquisition Knowledge acquisition was measured in terms of the grades of the term exam. We have analyzed these data from two points of view, scalar and categorical. From a scalar point of view, the grades were standardized to the range [0,1]. The term exam grade (TEG) was divided into two parts, labs exercises (L, 25%) and theoretical exercises (75%). Theoretical exercises consisted of a theoretical question (T, 33.3%) and a problem (P, 66.6%). The following formula was used to compute TEG(Term Exam Grade): T EG(L, T, P ) = 0, 25 L + 0, 75 ((T + 2 P )/3) No signicant differences among the groups have been detected, see table I. However, differences for the theoretical question seem to be almost signicant p = .057 and near to signicant for the term exam grade p = .084. A detailed analysis of both cases shows two signicant differences between the VG and TG groups, both in favour of VG, regarding the theoretical question (20.39%, U = 565.5, p = .028) and the term exam grade (12.72%, t(61) = 2.028, p = .046).

Table I S CALAR ANALYSIS OF TERM EXAM GRADES Average grades for CG VG TG .5486 .65 .525 .7122 .7233 .5194 .3954 .3707 .2808 .5129 .5287 .4015 Signicant differences test among groups 2 (2, 95) = 1.77, p = .412 2 (2, 95) = 5.74, p = .057 2 (2, 95) = 3.92, p = .141 F (2, 92) = 2.528, p = .084

L T P TEG

Scalar grades were mapped to four categorical grades: failure, fair, good and excellent. Table II shows their distribution. We have detected signicant differences of these grades, 2 (6, 95) = 13, 408, p = .037. The most important difference is the number of failures, less than 47% in the CG and VG groups and more than 61% in the TG group. B. Students Attitude towards the Subject We measured this variable with the exam-taking rate for each group. Table III shows the exam-taking rates for all the students enrolled in the subject and for those who participated in the evaluation. While the rates for all the students are similar in the three groups, i.e. the rates are independent from the groups (p = .649), the rates for participants are almost dependent from the groups (p = .057). A detailed study shows that there exist signicant differences between CG and TG (U = 1223, p = .022) of 20.71%, and almost signicant differences between CG and VG (U = 1348.5, p = .054) of 17%, always in favor of CG. C. Students Opinion about the Animations We collected students opinion about animations with a questionnaire. This questionnaire was only applicable to the groups that used animations. It consisted in ve statements following a Likert scale of ve values (being 5 the best and 1 the worst). The statements were: (1) animations have helped me in understanding the concepts, (2) animations are easy to construct, (3) animations are easy to use (play/view) and (4) animations are useful. Note that statement (2) was only answered by students from CG while statement (3) was
Table II C ATEGORICAL ANALYSIS OF THE TERM EXAM GRADES T EG CG VG TG Failure [0.0, 0.5) 43.20% 46.40% 61.10% Fair [0.5, 0.7) 35.15% 25.60% 22.20% Good [0.7, 0.9) 21.65% 14.00% 16.70% Excellent [0.9, 1.0] 0.0% 14.00% 0.0%

Table III E XAM - TAKING RATE ANALYSIS All the students 65.5% (78/119) 60.8% (73/120) 60.4% (90/149) 2 (2, 388) = .865, p = .649 Just participants 85.00% (34/40) 68.00% (34/50) 64.28% (27/42) 2 (2, 132) = 5.73, p = .057

CG VG TG

29

Table IV A NSWERS TO THE QUESTIONNAIRE % of answers in [4-5] Help understanding Ease of construction Ease of use Usefulness CG 86,2% 100% NA 83,4% VG 100% NA 100% 100% Total (CG & VG) 93% 100% 100% 91,5%

[2] C. Hundhausen, S. Douglas, and J. Stasko, A meta-study of algorithm visualization effectiveness, J. Visual Lang. Comput., vol. 13, no. 3, pp. 259290, 2002. [3] J. Urquiza-Fuentes and J. A. Vel zquez-Iturbide, A survey of a successful evaluations of program visualization and algorithm animation systems, ACM TOCE, vol. 9, pp. 9:19:21, 2009. [4] C. Pareja-Flores, J. Urquiza-Fuentes, and J. A. Vel zqueza Iturbide, WinHIPE: an IDE for functional programming based on rewriting and visualization, ACM SIGPLAN Not., vol. 42, no. 3, pp. 1423, 2007. [5] R. Mayer, Multimedia Learning. Cambrige Univ. Press, 2008. [6] J. Stasko, Using student-built algorithm animations as learning aids, SIGCSE Bull., vol. 29, no. 1, pp. 2529, 1997. [7] J. Urquiza-Fuentes and J. Vel zquez-Iturbide, An evaluation a of the effortless approach to build algorithm animations with WinHIPE, Electronic Notes in Theoretical Computer Science, vol. 178, pp. 313, 2007. [8] C. Hundhausen, Integrating algorithm visualization technology into an undergraduate algorithms course: Ethnographic studies of a social constructivist approach, Comp. Educ., vol. 39, no. 3, pp. 237260, 2002. [9] C. Hundhausen and J. Brown, Designing, visualizing, and discussing algorithms within CS1 studio experience: Empirical study, Comp. Educ., vol. 50, no. 1, pp. 301326, 2008. [10] T. H bscher-Younger and N. Narayanan, Dancing hamsters u and marble statues: characterizing student visualizations of algorithms, in Proc. of the 2003 ACM Symposium on Software Visualization. New York, NY, USA: ACM Press, 2003, pp. 95104. [11] A. Lawrence, Empirical studies of the value of algorithm animation in algorithm understanding, Ph.D. dissertation, Dep. of Computer Science, Georgia Inst. of Technology, 1993. [12] C. Kann, R. Lindeman, and R. Heller, Integrating algorithm animation into a learning environment, Comp. Educ., vol. 28, no. 4, pp. 223228, 1997. [13] M. Crosby and J. Stelovsky, From multimedia instruction to multimedia evaluation, J. Educational Multimedia and Hypermedia, vol. 4, pp. 147162, 1995. [14] C. Kehoe, J. Stasko, and A. Taylor, Rethinking the evaluation of algorithm animations as learning aids: an observational study, Int. J. Hum.-Comput. St., vol. 54, no. 2, pp. 265284, 2001. [15] A. Kumar, Results from the evaluation of the effectiveness of an online tutor on expression evaluation, SIGCSE Bull., vol. 37, no. 1, pp. 216220, 2005. [16] J. Vel zquez-Iturbide, C. Pareja-Flores, and J. Urquizaa Fuentes, An approach to effortless construction of program animations, Comp. Educ., vol. 50, no. 1, pp. 179192, 2008. [17] J. Nielsen, Designing Web Usability. New Riders Pub., 1999.

only answered by students from VG. Table IV summarizes the students answers to these questions. VI. C ONCLUSION In this study we have analyzed the educational impact of two uses of functional program animations, vision and construction tasks. Both of them were compared to a typical teaching methodology where animations are not used. Taking into account the Learning by Doing theory, Hundhausen et al.s [2] conclusions and the Engagement Theory [1], construction tasks should have overcome the other two approaches. Our results only support partially this assumption. We have considered knowledge acquisition from two points of view: scalar and categorical. From the scalar point of view, only the vision approach outperformed the typical approach, although grades from the construction approach were quite similar to the vieweing one. From the categorical point of view, the use of animations, either viewing or constructing, decreased the fail-rate more than 14% with respect to the typical approach. Students from the construction approach showed to have better attitude towards the subject than the rest of students. Their exam-taking rate (85%) is signicantly higher than the one from the typical aproach (64.28%). In addition, the students exam-taking rate from the viewing approach (68%) was quite similar to the typical approach. Finally, the students opinion about the animations is clearly positive. Most of the students (between 83.4% and 100%) who used animations think that: animations helped them to understand concepts, animations are easy to use and construct and animations are useful. All of them are desirable effects on students learning process, therefore both approaches, viewing and constructing, should be integrated in the teaching methodology. ACKNOWLEDGMENT This work was supported by the II Teaching Innovation Projects programme of the Rey Juan Carlos University and project TIN2008-04103/TSI of the Spanish Ministry of Science and Innovation. R EFERENCES
[1] T. Naps, G. R ling, V. Almstrum, W. Dann, R. Fleiso cher, C. Hundhausen, A. Korhonen, L. Malmi, M. McNally, S. Rodger, and J. Velzquez-Iturbide, Exploring the role of visualization and engagement in computer science education, SIGCSE Bull., vol. 35, no. 2, pp. 131152, june 2002.

30