Getting to Know your Student in Distance Learning Contexts

Claus Zinn and Oliver Scheuer

DFKI German Research Centre for Articial Intelligence Stuhlsatzenhausweg 3, 66123 Saarbrcken, Germany, u Email: claus.zinn@dfki.de, Phone: +49 681 3025064

Abstract. Good teachers know their students, and exploit this knowledge to adapt or optimise their instruction. Teachers know their students because they interact with them face-to-face in classroom or one-to-one tutoring sessions. They can build student models, for instance, by exploiting the multi-faceted nature of human-human communication. In distance-learning environments, teacher and student have to cope with the lack of such direct interaction, and this must have detrimental effects for both teacher and student. In this paper, we investigate the need of teachers for tracking student actions in computer-mediated settings. We report on a teachers questionnaire that we devised to identify the needs of teachers to make distance learning a less detached experience. Our analysis of the teachers responses shows that there is a preference for information that relates to student performance (e.g., success rate in exercises, mastery level for a concept, skill, or method) and analysis of frequent errors or misconceptions. Our teachers judged information with regard to social nets, navigational pattern, and historical usage data less interesting. It shows that current e-learning environments have to improve to satisfy teachers needs for tracking students in distance learning contexts.

Motivation

There is an increasing number of e-learning systems that aim at adding computermediated instruction to traditional classroom teaching, one-to-one tutoring, and individual learning. The use of such systems can be benecial for teacher and learner, but it can also lead to frustration for both of them. Learners may not prot from e-learning tools because of incorrect or sub-optimal use, lack of guidance or discipline, the complexity of the software etc. Teachers may have problems to judge whether learners make proper use of the software, or achieve learning gains because of, or despite of, e-learning software. In traditional settings, good teachers know their students, in terms of their mastery levels, motivation, learning progress, learning styles, and misconceptions. Good teachers can then exploit this information to adapt or optimize their instruction and to timely intervene in the learning process when problems occur. The importance of student monitoring is essential. Cotton, for instance,

surveyed the literature in educational research and found that this discipline identied the practice of monitoring student learning as an essential component of high-quality education [1]. In face-to-face teaching instructors can build their student model by direct questioning, observations (e.g., overall behaviour, visual cues as facial expression and gesture), and communications (e.g., linguistic clues, intonational pattern). This is hard, if not impossible, in computer-mediated learning systems, which are far less perceptive to aective and motivational aspects of student behaviour and the subtleties of human-human communication. It seems that the often proclaimed advantage of e-learning, learning any time, any place, turns to a major handicap when it comes to student tracking. One-to-one human-human tutoring is costly, however, and one-to-many classroom instruction leaves little time to teachers to care for individual student needs. There is thus a justication for e-learning systems to complement humanhuman instruction with (one-to-one) computer-mediated learning. In fact, where human resources are scarce, intelligent e-learning systems can open new possibilities. They could monitor student activity by keeping track of all system-user interactions. A sophisticated analysis and use of such data can then provide valuable information to teachers, say, to better understand their (distant) students, and to point them to their students at risk.1 The analysis process as well as the presentation of the results can be mechanised by software tools. Our goal is to develop a student tracking tool for teachers in the context of ActiveMath, a web-based, user-adaptive learning environment for mathematics [http://www.activemath.org], and iClass, an intelligent cognitive-based open learning system [http://www.iclass.info]. The requirements analysis unfolds into two aspects. First, from the vast amounts of raw system-learner interaction data, we need to identify and extract information that is valuable for teachers. Second, from the various possible presentations for extracted information, we need to identify intelligible visualisation methods that allow teachers to interpret student actions eectively and eciently. To investigate the rst aspect, we have devised a teachers questionnaire.2 In this paper, we present the questionnaire, an analysis of the results, and derived requirements for a student tracking tool.

2
2.1

Related Work
Computer-based learning

There is a plethora of e-learning systems, ranging from general content management systems to domain-dependent intelligent tutoring system, using dierent
1

Results from data analysis can also be presented to course designers to improve the content of instruction, to learners for self-reection, and to system developers, say, to increase the usability of an e-learning systems graphical user interface. The questionnaire is still available on-line and accessible via the URL http://www.activemath.org/questionnaire/questionnaire.html as we continue to welcome teachers contributions to student tracking tool requirements.

architectures, having various purposes, and oering dierent services and tools. There are standalone systems that run on your local machine; others follow a network-based client/server architecture; some use web-browsers, others have proprietary graphical user interfaces. Learning systems may support self-study, collaborative (peer-to-peer) learning, organized learning in supervised groups, web-based homework, or combinations thereof. Some systems emphasise services for students, others target teachers and course designers. The features and limitations of student tracking software depend on the characteristics of the underlying e-learning systems. The overall architectural setting (local/network, browser/proprietary GUI etc.) has a major impact on logging mechanisms and the exploitation of log data. A local installation, for instance, excludes a centralized log data collection; and a standard web-browser requires sophisticated (client-side) scripting to get hold of tracking data. In this paper, we ignore technical constraints of existing e-learning systems, and take a user-centered approach to requirements analysis. Given a set of common learning activities in e-learning contexts such as reading content material (provided as text, (graphical) illustrations, movies, animations, sound etc.), communicating with others (using email, discussion forums, chat etc.), practicing and assessing (say, answering ll-in-the-blanks questions, multiple-choice questions, puzzles, teacher-evaluated assignments etc), we need to identify those which analyses yield potentially valuable learner information for teachers. 2.2 Student Tracking Approaches

There is a wide range of student tracking approaches and implementations. Commercial course management systems like WebCT, for instance, track: the date of the rst and most recent sign-on for each student in the course; the usage of the conference tool (number of articles read, number of postings and follow-ups to measure of involvement and contribution to the course); the use of various other tools (e.g., notes, calendar, e-mail); and visits to the content material (page visits, view times) to monitor course usage, usefulness, and quality. Gathered data is then visualised in tabular form or with bar charts [2, 14]. In [7], Mazza et al. critisise visualisations in tabular format as often incomprehensible, with a poor logical organisation, and as dicult to follow. As a result, they say, tracking data is very rarely used by distance learning instructors. The information visualization approach followed by CourseVis [7] and its successor, the graphical interactive student monitoring system GISMO [8], uses graphical encodings of data (generated by course management systems) that are adapted to human cognition and that allow teachers to capture important relations and facts easily. The graphical representations provided are usually interactive and can be modied by the user (e.g., rotatable 3D graphs). In [4], Hardy et al. present a toolbox to track, analyse and display learners interactions within a virtual learning environment. Student tracking data include, for instance, amount of time a user spends in dierent parts of system, number of accesses to the course material by date, or the navigational route taken through the contents. Whereas Mazza et al. have the main motivation to

help instructors to build a mental model of their students, Hardy et al. emphasise that the provided information can be used as a valuable input when evaluating the eectiveness of the e-learning materials and framework. Merceron and Yacef pursue a data mining approach to student activity tracking [10]. Their tool for advanced data analysis in education (TADA-Ed) oers, for instance, an association rule algorithm that can be used to identify sets of mistakes that often occur together, or a decision tree algorithm to predict nal exam marks given students prior usage behaviour. Usage of the tool requires basic knowledge in data mining (and machine learning as its technological basis), and therefore, TADA-Ed is a tool that is primarily directed at educational researchers rather than professional teachers. Some tutoring systems have a proprietary student tracking tool that depends on the domain of instruction. The LISTEN Reading Tutor, for instance, helps children learning to read [11]. The system displays stories, sentence by sentence, which children then have to read aloud. LISTENs logging mechanism stores the childrens utterances as well as other student-system interaction events into a relational database. The LISTEN student tracking tool processes interaction events by means of SQL queries and presents the results to the educational researcher [12]. The results are hierarchically structured: if the time interval of one interaction is contained in the time interval of another interaction of the same student, then there is a parent-child relation between the two interactions. A browser then shows the resulting tree; browsing is facilitated by expanding and collapsing tree nodes, or by dening interaction types lters. The tool is intended for qualitative case analysis. A related strand of research deals with the visualization of learner models. Student modelling is a discipline in articial intelligence which analyses the computer-tracked student behaviour and information provided by the student himself to build an internal representation (the systems beliefs about the student). This representation can then be used to adapt courses to the needs, goals, and preferences of individual students and to generate personalized feedback and help. The resulting model can also be visualised. The VisNet system visualises learner models that are based on Bayesian Belief Networks (BBN) [3]. VisNet is intended for engineering and tuning BBNs and for engineering and maintaining BBN representations of learner models. In [15], Zapata-Rivera and Greer describe the VisMod system, another system for visualizing Bayesian learner models. The inspectible learner models provided by this system should also benet instructors and learners: Using ViSMod it is possible to inspect the model of a single learner, and compare several learners by comparing their models, navigate throughout the model changing focus according to learners and teachers interests, and use several sources of evidence to animate the evolution of a new model. Given the large variety of student tracking systems and their rich sets of features, it is time to keep stock and ask teachers (in part, experienced with the use of such systems) to state their interests and requirements for the tracking and analysis of student activity in e-learning systems.

Teachers Questionnaire

The questionnaire resulted from a multi-step process. First, candidate information units for student tracking was gathered. Then, to evaluate the candidates usefulness as well as their description in terms of understandability and completeness, feedback was requested from a few teachers via email. Their answers were exploited to make questionnaire formulations less technical, and to add or elaborate explanations were necessary. The revised questionnaire was then encoded into hypertext and published on-line (in both English and German), together with a PHP script that stored submitted forms into a MySQL database. Distant instructors were pointed to the questionnaire by personal contacts, hyperlinks from various web-sites3 , and postings with a call for participation in two highly frequented discussion forums for e-learning: the Distance Education Online Symposium (DEOS-L)4 and the Edtech Discussion list5 . 3.1 Questionnaire Description

The questionnaire consisted of three parts. The introductory part motivated and explained its purpose. The second part, organised thematically, proposed actual candidates for student tracking information. Teachers were asked to evaluate those candidates with regard to interestingness on a scale with the values not at all interesting, not very interesting, fairly interesting, and very interesting. The value I am not sure was available to cover cases where participants could not assess the interestingness of a given proposal. Teachers were also asked to give qualitative feedback with regard to their preferences or to the appropriateness of our proposals. In other text input elds, they could contribute their ideas to improve or complement our candidate proposals for student tracking. The third part aimed at proling those who participated in lling-out the questionnaire; this last part was adopted, with some slight modications from [5]. The prole included, for instance, participants role, instructional setting, and e-learning experience. Fig. 1 display a shortened version of the questionnaires second part. 3.2 Questionnaire Results

At the time of writing, 49 teachers participated in the study. Their votes on our proposals for tracking student activity were computed with the following weights:6 very interesting (weight 5), interesting (4), not very interesting (2), and not at all interesting (1). The score value for a given proposal resulted from the addition of all votes with their corresponding weights, and a subsequent
3 4 5 6

http://www.activemath.org, ccel.dfki.de, and www.saarlernnetz.de. http://www.ed.psu.edu/acsde/deos/deos.asp. http://www.h-net.org/ edweb/list.html. Question category 11 (presentational issues) and 12 (free text input eld for preferences, comments etc.) were not scored. They were also omitted in Fig. 1.

(1) Usage & Activity times (1.1) Amount of time the learner spends with the system (per session) (1.2) Number of learner actions with the system (per session) (1.3) Number of sessions (1.4) History of past usage (2) Course coverage (2.1) Percentage of available material read (2.2) Percentage of available exercises tackled (2.3) History of past percentages (3) Learning Content (3.1) Amount of time spent per concept/skill/method/competency (3.2) Number of learner activities per concept/skill/method/competency (3.3) History of past learning contents (4) Activity types (4.1) Amount of time per activity type (4.2) Activity type distributions (4.3) History of past activity patterns (5) Student assessment/aptitude (5.1) Overall success rate (5.2) Overall cancellation rate (5.3) Number of steps done (for multi-step exercises) (5.4) Ratio of correct to incorrect steps (for multi-step exercises) (5.5) Number of learner help requests (5.6) Replay of exercises (in discrete steps) (5.7) List of n most frequent diagnosed mistakes and misconceptions (6) Learning gains (6.1) Mastery level for each concept/skill/method/competency (6.2) History of past mastery levels (for each concept/skill/method/competency) (7) Learning interests (7.1) List of accessed (and potentially) read course material (7.2) List of teaching materials underlying concepts/skills/methods/competencies (7.3) List of most frequently looked-up terms (when a dictionary is available) (7.4) List of most frequently trained concepts/skills/methods/competencies (8) Learners navigational style (8.1) Learner classication (8.2) History of past learner classication (9) Groups (9.1) Allow manual denition of groups and computation of group results (9.2) Allow automatic group clustering (10) Communication tools (10.1) Number of communication actions (per tool) (10.2) Ratio of social activities to all activities (10.3) Number of thread initiations (10.4) Number of follow-up postings (10.5) Social Nets

Fig. 1. Questionnaire (simplied form without explanations). The full version is available at http://www.activemath.org/questionnaire/questionnaire.html.

linear transformation that yielded a scale from 0 to 100. Note that votes of the form I am not sure did not enter the score. We have summarised our results in two tables and gures: Tab. 1 shows the top-ten proposals with the highest scores, and Tab. 2 shows the ten proposals with the lowest scores. Fig. 2 and Fig. 3 show the distributions for the ten best and ten worst proposals, respectively. We will discuss the results in Sect. 4.
Votes 2 3 0 0 0 1 3 3 1 4 5 4

Nr 5.1 6.1 2.2 5.7 2.1 7.1 3.1 5.2 3.2 4.2

Proposal Overall success rate (for exercises) Mastery level for each concept, skill, method, competency Percentage of available exercises tackled List of n most frequent diagnosed mistakes and misconceptions Percentage of available material read List of accessed (and potentially) read course material Amount of time spent per concept, skill, method, competency Overall cancellation rate Number of learner activities per concept, skill, method, competency Activity type distributions

0 3 8 2 5 2 5 4 7 4 6

1 0 0 0 1 0 1 1 0 1 2

Score 93 92 92 90 87 86 85 85 83 82

12 34 13 28 15 32 11 31 16 28 12 28 20 23 14 24 11 28 11 26

Table 1. Highest scored proposals: 0 I am not sure (or no vote), 1 not at all interesting, 2 not very interesting, 3 fairly interesting, 4 very interesting.

3.3

Questionnaire Participants Proles

In total, 49 teachers, representing 10 countries, participated in the study (USA: 63%, United Kingdom: 12%, Germany: 8%, and others).7 The most frequently taken roles were instructor (91%), instructional designer (67%), and course coordinator (37%). Most of the participants were involved in 2-5 online courses (63%), with maximum course sizes of 20-50 students (50%). Most participants supervised learners in higher education: 76% worked with undergraduates, and 43 % worked with postgraduate students. Two thirds (65%) of the participants are engaged in pure distance courses, the remaining third (35%) work in blended learning scenarios. The main proportion of participants were rather experienced with an involvement of 5 years or longer in online courses (62%). A wide range of learning systems was used: 22 dierent systems were explicitely stated, headed by the commercial ones from WebCT (47%) and Blackboard (42%).
7

The nationality was derived from IP addresses and email addresses (where supplied).

5.1 6.1 2.2 5.7 2.1 7.1 3.1 5.2 3.2 4.2

40

20 Not at all interesting/ Not very interesting (in %)

20

40 60 Fairly interesting/ Very interesting (in %)

80

100

Fig. 2. Distributions of high-score proposals

Participants used the following e-learning services or tools: content material (98%), email (93%), discussion forums (87%), chat tools (59%), and glossary (28%). Other tools (given in free text input) included video-conferencing, portfolios, eld investigations, wikis and blogs. Participants used the following assessment techniques: teacher-evaluated assignments (96%), quizzes, e.g., multiple-choice and ll-in-the-blanks questions (76%), analysis of discussion forum contributions (61%), group work (59%), simple usage statistics (33%), and analysis of log les (28%). Other assessment techniques (given in free text) included portfolio, student self-evaluation, automatic essay scoring, peer assessment and self assessment, end of course paper, and student homepages graded by required criteria.

Discussion

Approval rates for individual proposals (measured as the proportion of participants stating fairly interesting or very interesting) ranged from 63% to 100%, with an average value of 85%. Thus, all proposals were met with considerable approval. This also corresponds to some qualitative feedback: I believe you have covered a great range of features., I think all those data should be

9 Votes 2 3 7 7 9 5 7 10 8 7 11 14

Nr 1.2

Proposal

0 4 5 10 6 8 7 7 5 6 8

1 1 1 0 2 2 0 2 4 3 1

Score 76 76 75 75 73 71 70 70 68 62

Number of learner actions with the system (per session) 8.1 Learner classication (navigational style) 5.6 Replay of exercises (in discrete steps) 4.3 History of past activity patterns 9.2 Allow automatic group clustering 2.3 History of past percentages (course coverage) 3.3 History of past learning contents 10.5 Social Nets 10.2 Ratio of social activities to all activities 8.2 History of past learner classication (navigational style)

18 19 18 12 20 15 18 18 16 10 16 18 18 16 17 14 14 17 19 10

Table 2. Lowest scored proposals: 0 I am not sure (or no vote), 1 not at all interesting, 2 not very interesting, 3 fairly interesting, 4 very interesting.

available to the teacher, In principle all proposals are well thought-out and justiable ... (translated from German) and From the point of view of someone who likes investigating data to nd patterns my answer to the majority of the questions is very interesting . Although all proposals were assessed as at least fairly interesting by a large majority of participants, the computed scores reveal clear trends and favorites. 4.1 Proposal Ratings

Our teachers considered performance-related student tracking information particularly interesting. All of the top four entries in Tab. 1 address this issue. This high interest is not surprising as student assessment is a major teacher task. When teachers correct homework assignments or student exams, then notions like overall success rate and percentage of available exercises tackled are crucial to obtain a mark; the way students solve an exercise, or fail to solve it, gives teachers valuable information about students level of mastery for exerciserelated concepts, skills, and methods. It is worth to note that teachers voted list of n most frequent diagnosed mistakes and misconceptions as particularly interesting (rank 4). The identication of typical student errors helps assess mastery levels, and also informs teachers remedial feedback. Some anectodal email-based feedback said: For me, it would be more interesting to get feedback on the type of error; or in abort situations, feedback describing where pupils failed.8
8

Translated from German: Es waere fuer mich interessanter, Rueckmeldung ueber die Art der Fehler zu erhalten bzw. bei Abbruechen, an welchen Stellen die Schueler (innen) scheitern.

10

1.2 8.1 5.6 4.3 9.2 2.3 3.3 10.5 10.2 8.2

40

20 Not at all interesting/ Not very interesting (in %)

20

40 60 Fairly interesting/ Very interesting (in %)

80

100

Fig. 3. Distributions of low-score proposals

On fth and sixth position of the interestingness scale, we have student tracking information with regard to reading. Proposal 2.1 focuses on the percentage of available material read, whereas proposal 7.1 oers an insight into a students learning interest by listing all of the course material that the student has (potentially) read. It could be that teachers with frequent reading assignments gave high rating for these proposals; those teachers may welcome tools to check whether students carried-out their reading assignments in time, or even explored material of their own choosing. On positions 7-10, we nd information that only e-learning systems could provide. The amount of time (rank 7) and the number of activities (rank 9) can be used to estimate how much eort a student invests. The overall cancellation rate (rank 8) may indicate a students perseverance in tackling problems. Together with other top-ranked proposals, this information can be used to derive a students pace through the course material.9 The activity type distribution (rank 10) shows how students divide their time; it may indicate whether stu9

Using this data in isolation is problematic; for instance, students may not read texts that were given to them, or they may start a large number of activities without being properly involved or motivated.

11

dents systematically evade certain activities, or whether they spend too much time on other, potentially distracting, tools. Some teachers assign less importance to these lower-ranked pieces of information. One teacher considers teaching as an amalgam of dierent activities. For this teacher, the way the student divides his/her time over these activities is of less concern.... Another teacher wrote the following comment: As teacher, I am interested in what stuck [with the students]. The way [towards understanding] is only of interest if there are problems in achieving a learning goal, and to analyse the causes for this10 . Another teacher agrees: [...] often problems, misconceptions, negative emotions, and the like are more important to a facilitator than controlling the actual pathways (which are hidden inside the learners and partially take place outside the learning environment). This view is reected by our ranking; navigational pathways and replay of learner actions received low scores. Also, historical information of student-system interaction in terms of past activity pattern, past percentages of course coverage, and past learner classication obtained a lower rating. From the questionnaires proling section, we know that the assessment techniques usage statistics and log data analysis were only marginally used. The rather low rate for this kind of data indicates that current student tracking tools do not address instructors needs. 4.2 Preferences, Appropriateness, and Completeness

In question 12, we asked teacher to rank given student tracking proposals as well as to comment on their appropriateness and completeness. Preferences. Only 9 teachers from a total of 49 teachers stated their preferences. Two participants gave very general statements (I believe you have covered a great range of features, I think all of those data should be available...]). The ranking of the remaining seven teachers is quite diverse, ranging from proposals that appear in the top-ten list to medium-scored proposals (e.g., analysis of online time, amount of social activity) and low-scored ones. The small sample of 9 teachers, thus, only partially represents the 49 votes on our candidate proposals, which may be explained by the diverse teacher backgrounds: The level of the learner has much to do with the answers given. We teach only graduate students pursuing masters or doctoral degrees. Our interests would be far dierent from those dealing with K-12 or undergraduates.
10

Translated from German: Als Lehrer interessiert mich, was haengengeblieben ist. Der Weg dahin ist nur dann von Interesse, wenn es Probleme bei der Lernzielerreichung gibt, um Ursachen zu analysieren.

12

Appropriateness. We received controversial feedback with respect to the appropriateness of our candidate proposals for student tracking. Whereas one teacher sees online time as a valuable piece of information (This gives me information on the time spent on dierent subjects, so this is a valuable addition), another one questions its usefulness (On-line time barely tells me whether a student really worked through the content of the e-learning system)11 . We obtained similar controversial remarks with regard to activity times. One teacher believes that the use of activity times indicates which activities are time-consuming or not.; another teacher considers teaching as an amalgam of dierent activities. For this teacher, [T]he way the student divides his/her time over these activities is of less concern [...]. There is no perfect proler, since learning, as one teacher put it, takes place within and outside virtual learning environments. There is thus a danger that student tracking tools may mislead teachers to interpret the share of all activities performed in the learning environment for the whole picture. According to this teacher, some of our questions and proposals sound like implying the existence of perfect learner classication, when student models can only be based on imperfect beliefs. Completeness. Less than a quarter of the participants stated an opinion on the completeness of our proposals for student tracking. One teachers found that the number of metrics is sucient. Another teacher is more sceptical: Im not sure any question can capture the essence of learning cognition and application- I think you might get some sense of it through a small group of questions. In the completeness text eld, teacher proposed to track: the number of times and duration of students viewing help materials, student inactivity, and metacognitive indicators (e.g., condence and prudence values derived from learner answers). One teacher would like to detect potential cheaters, and proposes to check whether students go online and take their tests at the same time. 4.3 Related Studies

The designers of the CourseVis system also devised a teachers questionnaire to inform the formulation of system requirements for the tracking of distant students [6]. Their empirical study covered three aspects: social aspects (studentstudent, student-tutor, student-system, and vicarious interactions), cognitive aspects (e.g., student performance), and behavioural aspects (e.g., course attendance, students pace). In total, 98 distance learning instructors participated in the CourseVis study, most of them involved in university courses. The ndings are as follows (in parentheses: the share of participants who voted for a feature as extremely interesting or very interesting): participation in discussions (66%), participation in
11

Translated from German: Die Onlinezeit sagt kaum etwas darueber aus, ob tatsaechlich im bzw. mit dem System inhaltlich gearbeitet wurde.

13

group work (67%), overall performance (82%), and performance for certain domain concepts (63%). From the behavioural aspects, course attendance, access of and time spent for course material, performance for evaluation proofs, and participation in discussions earned the highest assessment. There is high agreement between both studies as all teachers favoured performance-related student tracking data. In the CourseVis study, 82% of all teachers rated the feature overall performance as extremely interesting or very interesting. In our study, overall success rate came out rst. There is also agreement on the importance of mastery levels. The CourseVis category performance for certain domain concepts got a high approval rate of 63%, where our feature mastery-level for concepts/skills... took second place in our top-ten table. The relation between mastery levels and diagosed mistakes/misconceptions (our rank 4) was also drawn in the CourseVis study: The importance of this information was also stressed by participants in the discussions we had. Almost all instructors wanted to use this information in order to identify and remedy learners common misconceptions in their courses The CourseVis proposal to track students progressing within the course schedule yielded high ratings as well. This is in line with the rating of our teachers regarding percentages of exercises tackled (rank 3), and percentages of available material read (rank 5).

Conclusion

Good teachers know their students. In the words of Cotton [1]: Research comparing eective and ineective teachers cites the existence and use of a systematic procedure for keeping and interpreting data on student performance as a notable dierence between these groups. Student monitoring is thus a crucial ingredient to optimize instruction and to make learning a success. Teachers (and students) will only prot from the increasing market for e-learning products and services, when proling tools help them getting to know their students in distance-learning contexts. The participants of our study know this, of course. When we asked them how they would you use student tracking information (Question 13.9), their top-three choices were: to respond to specic individuals in an appropriate way (83%); to adapt teaching to individuals or groups of learners (73%); and to identify and remedy common misconceptions (73%). It is therefore not surprising that teachers have a preference for pedagogicallyoriented measures (overall success rate, mastery levels, typical misconceptions, percentage of exercises tackled and material read), which are, naturally, easy to interpret. On the other hand, teachers nd ne-grained statistics of log data (replay of exercises, histories of past course coverages and activities, navigational

14

style) too cumbersome to inspect or too time-intensive to interpret, and social data (social nets, ratio of social activities to all activities) of less interest. Our ranking of student tracking proposals, however, shows no clear winner as the distance between any adjacent entry in Tab. 1 and Tab. 2 is at most three points. The separation of ranked proposals into two tables (top-scores, lowscores) is therefore articial; a student tracking tool that will only contain our top-ten proposals will disappoint teachers with other preferences. Teachers, with their many dierent backgrounds, roles, instructional settings, and visualisation preferences should therefore be given an easy-to-use conguration mechanism to personalise their student tracking workbench. Also, given the rich e-learning experience of many of our participants, it would be an enormous advantage if a student tracking tool could interface to many e-learning systems, thus providing a uniform student proling application across multiple e-learning platforms. A common, standardised format for log data would facilitate the construction of a multi-platform student tracking tool. The adequate logging, ltering, and analysis of student log data is obviously constrained by the underlying e-learning system, which may or may not have the capabilities to properly log semantically rich student actions. Unfortunately, current state-of-the-art e-learning system only provide quite shallow data. They track accesses to teaching material (say, to compute course coverage), time-related information (say, to compute the number and duration of tutorial sessions), and they have simple student assessment methods where students are asked to answer closed questions (e.g., multiple-choice questions and ll-in-the-blank questions). This information makes it hard, if not impossible, to identify a students conceptual understanding of the course, say, in terms of mastery levels or student misconceptions. As one teacher said: Presumably, the pure number of successful steps will not tell me much, apart for closely-guided exercise solutions, which, however, tell me very little about [learners] conceptual understanding.12 Teachers found tracking information on the students mastery level for each concept, skill, method, or competency as well as the amount of time spent and the number of learner activities invested to acquire such level particularly interesting. The provision of this information, however, is only possible if content material, exercises, tool use etc is semantically annotated with adequate meta-data. The requirements of our teachers are partially met by intelligent tutoring systems [13]. The ActiveMath system [9], for instance, has a semantic annotation of learning content, interactive exercises that allow students to solve non-trivial, multi-step problems, and domain reasoners that support the system to perform deep analyses of student answers in terms of correctness and misconceptions. Such systems are expensive to build, but will help students to better learn, and with appropriate student activity tracking tools teachers to better getting to know their distant students.
12

Translated from German: Da sagt mir die Anzahl der erledigten Schritte vermutlich wenig aus, wenn es nicht streng gefuehrte Aufgabenloesungen sind, die dann aber wieder weniger ueber das entwickelte Begrisverstaendnis aussagen.

15

Acknowledgements. We are grateful to Erica Melis for initial work on the questionnaire, and for publicizing its availability. The work presented in this paper is supported by the European Community under the Information Society Technologies programme of the 6th FP for RTD project iClass contract IST-507922.

References
1. K. Cotton. Monitoring student learning in the classroom. Northwest Regional Educational Laboratory, U.S. Department of Education, 1988. School Improvement Research Series (SIRS), http://www.nwrel.org/scpd/sirs/2/cu4.html. 2. M. W. Goldberg. Student participation and progress tracking for web-based courses using webct. In Proc. 2nd N.A.WEB Conf., Fredericton, NB, Canada, 1996. 3. J. Greer, J. D. Zapata-Rivera, C. Ong-Scutchings, and J. E. Cooke. Visualization of bayesian learner models. In Proceedings of the workshop Open, Interactive, and other Overt Approaches to Learner Modelling AIED 1999, 1999. 4. J. Hardy, M. Antonioletti, and S. Bates. e-learner tracking: Tools for discovering learner behavior. In The IASTED International Conference on Web-base Education, Innsbruk, Austria, 2004. 5. R. Mazza. Using Information Visualisation to Facilitate Instructors in Web-based Distance Learning. PhD thesis, University of Lugano, 2004. 6. R. Mazza and V. Dimitrova. Informing the design of a course data visualisator: an empirical study. In 5th International Conference on New Educational Environments (ICNEE 2003), pages 215 220, Lucerne, 2003. 7. R. Mazza and V. Dimitrova. Generation of graphical representations of student tracking data in course management systems. In IV 05: Proceedings of the Ninth International Conference on Information Visualisation (IV05), pages 253258, Washington, DC, USA, 2005. 8. R. Mazza and C. Milani. Gismo: a graphical interactive student monitoring tool for course management systems. In T.E.L. Technology Enhanced Learning 04 International Conference, Milan, 2004. 9. E. Melis, G. Goguadze, M. Homik, P. Libbrecht, C. Ullrich, and S. Winterstein. Semantic-aware components and services of activemath. British Journal of Educational Technology, 37(3), 2006. 10. A. Merceron and K. Yacef. Tada-ed for educational data mining. Interactive Multimedia Electronic Journal of Computer-Enhanced learning, 7(1), 2005. 11. J. Mostow, G. Aist, P. Burkhead, A. Corbett, A. Cuneo, S. Eitelman, C. Huang, B. Junker, M. B Sklar, and B. Tobin. Evaluation of an automated reading tutor that listens: Comparison to human tutoring and classroom instruction. Journal of Educational Computing Research, 29(1):61117, 2003. 12. J. Mostow, J. Beck, H. Cen, A. Cuneo, E. Gouvea, and C. Heiner. An educational data mining tool to browse tutor-student interactions: Time will tell! In AAAI 2005 Workshop on Educational Data Mining, 2005. 13. D. Sleeman and S. J. Brown, editors. Intelligent Tutoring Systems. Computers and People Series. Academic Press, 1982. 14. WebCT. WebCT VistaT M 3.0 Designer and Instructor Reference Manual, April 2004. Technical Communications, http://www.webct.com. 15. J. D. Zapata-Rivera and J. Greer. Externalising learner modelling representations. In Workshop on External Representations of AIED: Multiple Forms and Multiple Roles. International Conference on Articial Intelligence in Education AIED 2001, pages 7176, 2001.