2015 IEEE 15th International Conference on Advanced Learning Technologies

An Affective Learning Environment for Java

Ramn Zatarain Cabada, Mara

Luca Barrn Estrada, Francisco
Gonzlez Hernndez, Ral
Oramas Bustillos.
Instituto Tecnolgico de
Culiacn
rzatarain@itculiacan.edu.mx
II. JAVA SENSEI
Abstract We present Java Sensei, an Intelligent Learning THE Java Sensei ILE consists of seven components
Environment (ILE) for learning Java programming. The ILE
(figure 1): Affective Framework, Student or User Module,
is formed by an affective tutoring system working in a Web
environment. The tutoring system was implemented under Pedagogical Agent, Adaptation Module, Teaching
different learning methodologies like problem-solving for the (Pedagogical) Module, Expert Module and the ITS
pedagogical module, knowledge space for the expert module, (Intelligent/Affective Tutoring System) Engine. Next, we
and overlays for the student module. The main contribution of describe each of the components.
this work was the integration of cognitive and affective
information in a behavior graph which is used by the
pedagogical agent to show emotion and empathy to the
student.

Keywords: Intelligent Learning Environments, Intelligent

Tutoring Systems, Affective Computing, Programming
Languages, Neural Networks, Fuzzy Systems.

I. INTRODUCTION
The teaching and learning of computer programming is a
universal challenge for all colleges and universities in any
country. Some factors that contribute to this challenge are:
the teaching methods employed by the instructor, the study Figure 1. Architecture of ILE JavaSensei
methods employed by the student and his/her abilities and
attitudes, the nature of the art of programming, and the
psychological influence that the student suffers from society
[1-4]. Affective Framework: In the Affective Module we created
Many researchers have proposed and developed an emotional framework based on AutoTutor [9] where we
methodologies and tools to help students learn programming. implemented a fuzzy logic system and a neural network
For example the ACM has SIGCSE (Special Interest Group back propagation for emotion recognition. The emotional
on Computer Science Education) where issues related to the framework represents a student's emotional and cognitive
development, implementation and/or evaluation of programs model and is composed of input and output variables for the
and curricula are discussed [5]. fuzzy logic system. In Table 1 we can see each variable of
Today, in the field of Artificial Intelligence, Intelligent the emotional framework and next a brief description of
Tutoring Systems (ITS) try to simulate a human tutor to each one.
provide personalized instruction taking into account not only
cognitive aspects of students [6] but also affective elements Table 1. Emotional Framework.
[7].
This paper presents the implementation of an Intelligent Student Module Tutor Actions
Learning Environment (ILE) for learning Java programming. Current Emotion Feedback
This tool, named Java Sensei, attempts to create a flexible Previous Emotion Empathetic and emotional
and interactive web-based environment that considers the responses
cognitive and affective states of students. Global Student Skill Facial expression
The paper is organized as follows: section 2 describes the Quality of Current Intervention
design and implementation of system components. Results Answer
and application testing are shown in section 3 and
conclusions and future work are discussed in Section 4.

978-1-4673-7334-0/15 $31.00 2015 IEEE 350

DOI 10.1109/ICALT.2015.53
Student (User) Module: The student model represents the
cognitive and emotional part of the student. It consists of
two emotional and two cognitive aspects.
Current Emotion: The current emotion of the student when
answering an exercise. The emotion is obtained through
the emotion recognition application. The possible values
are: Happy, Angry, Sad, Neutral, and Surprise.
Previous Emotion: The student previous emotion.
Global Student Skill: The proficiency of the student
regarding the course.
Quality of Current answer: Is the number of errors that the
student makes in one exercise.
Tutor Actions.
The actions of the tutor represents the emotional response of
a human tutor and represent the output of the fuzzy logic
system that is used by the component of the Pedagogical
Agent and the Example-Tracing Engine. Both will be
explained later.
Feedback: This is a message that will give a positive,
negative or neutral perspective to the student regarding
their progress during the year. It represents the feedback
that gives you a human tutor when you want to know if
you're right or wrong in your academic progress.
Empathetic and emotional responses: These are phrases
received by the student depending on their emotional state.
The sentences will depend on the student's current
emotional state and aims to give the feeling of
understanding that gives a human tutor. This is not an
input variable for the fuzzy logic system, since the value
of a facial expression is used.
Facial expression: This is the facial expression of the
Pedagogical Agent.
Intervention: This value represents whether the
Pedagogical Agent must perform an intervention or not on
the student.
Fuzzy System and Neural Netwkork
The fuzzy logic system seeks to represent the way a
human tutor acts when he/she faces various scenarios
related with cognitive and emotional situations of a student
[10]. These situations have to do with the predefined
actions.
The fuzzy system uses the emotional values obtained by
the neural network which recognizes the affective state of
the student when he/she works with the ILE Java Sensei.
The fuzzy logic system works with four input fuzzy
variables and three output ones. A total of 144 fuzzy rules
were constructed. To define these rules we use Fuzzy
Control Language, a standardized language for creation of
fuzzy rules. The implementation was done with jFuzzyLogic
[11].
351
The emotion recognition system was built in three steps:
the first one was an implementation to extract features from
face images in a corpus used to train the neural network.
The second step was the implementation of the neural
network. We used the Java-based algorithms implemented
in Weka [12] to implement classification by using a neural
network (feed-forward method). The third step integrated
extraction and recognition into a fuzzy system, which is part
of the ITS. For training and testing the neural network, we
used the corpus RAFD (Radboud Faces Database), which is
a database with 8040 different facial expressions that
contains a set of 67 models including men and women.
Once the emotional state is extracted from the student, the
emotional state is sent to the fuzzy system. Figure 3 shows
the steps followed in the recognition and affective feedback
from the system. In the first step the ILE receives the data
and the student's collection of images from the browser
which are later sent to the ILE. In the second step the ILE
starts a cycle with several student faces being sent to the
neural network, and decides, at the end of the cycle, the
current emotion. This current emotion is obtained based on
the most frequent student emotion. In the third step the ILE
send the updated student model to the fuzzy system which
perform the fuzzy rules to output the tutor actions. In the
last step, the tutor actions are sent to the browser to build the
actions of the pedagogical agent.
Pedagogical Agent: The pedagogical agent is responsible
for transmitting messages that a human tutor would
perform. For this, it has been provided with both facial
expressions and dialogues that can communicate to the
student. These expressions are based on facial expressions
investigated by Ekman [13]. Positions of the eyebrow,
mouth, and eye opening are taken into account. These
positions are entered as parameters in the FaceGen Modeller
[14] software. In Figure 2 we observe the four facial
expressions made by the pedagogical agent of Java Sensei.
These emotions were based on the pedagogical agent
working with AutoTutor [10].
Figure 2. Pedagogical Agent of JavaSensei
 Figure 3. Affective Recognition and Feedback
Adaptation Module: The ILE must be able to adapt to
student [8]. The ILE collects the rating produced by the
student concerning the exercises and resources that the
student visits. The rating of one exercise is measured using
Likert scale [15] where one indicates that you totally
disagree with the exercise and five indicates that you totally
agree. At the beginning all the exercises are rated with a
value of two. We used a recommendation system created
with Apache Mahout [16]. This tool helps to find the next
problem the student must solve. The system implements the
K-Nearest Neighbor (k-NN) approach and the Pearson
Correlation. As more users are obtained, the rating system
generates the best recommendations and thereby it is more
adapted to the changes.
Pedagogical Module: The system uses the pedagogical
model known as "problem solving" [6] where the student
learns as he solves problems with a certain structure. The
system uses three types of strategies for solving problems.
Type 1 is used to evaluate theoretical concepts with true-
false exercises. Type 2 questions present a complete
program or piece of code and asked the student to determine
the output of the code. Type 3 is a combination of the two
previous types and allows you to create more complex
exercises involving "n" number of steps to reach the
solution.
Expert Module: This model contains expert knowledge
that the student wants to learn and practice. The system
represents the knowledge using JSON files (a variation of
XML files). Knowledge representation uses the knowledge
352
space theory [20]. The expert model represents six basic
skills that students must master to be modeled by a graph
representing the Knowledge Space. These skills are:
Introduction to Java, Variables and calculations, selection,
iteration, methods, and arrays.
ITS Engine: An ILE combines hypermedia techniques with
other methods used by ITS. We applied Example-Tracing
[17, 18] in order to implement the ITS in Java Sensei.
Example-Tracing ITS have the advantage of providing step-
by-step guiding to the students and providing with multiple
strategies to the problem solution, including optimal or sub-
optimal solution or miss-conception management. Example-
Tracing exercises can be easily adapted with the
pedagogical model implemented with problem solving
and they are represented by a graph that is traversed by the
student (figure 4).
One of the main contributions of our work is the integration
of cognitive and affective information in a behavior graph
which is used by the pedagogical agent to show emotion and
empathy to the student. Figure 4 shows part of a behavior
graph.
Next, we describe the algorithm to implement the affective-
example-tracing [19]:
a) The ILE load the JSON file with the corresponding
exercise.
b) From the JSON file, a behavior graph is created and the
initial step is executed.
c) As students traverse the graph (by clicking with the
mouse), different options can be performed and different
paths are traversed. The options are:
i. Initial step (IS)
ii. Error step (ES)
iii. Optimal step (OS)
iv. Sub-optimal step (SOS)
v. Final Optimal step (FOS)
vi. Final Sub-optimal step (FSS)
d) Every step triggers two events:
i. Actual Emotion Evaluation.
ii. Tutor Action Evaluation. Figure 5. Affective Example-Tracing

III. RESULTS AND COMPARISONS

The system has been tested with students in our research lab.
Next, we show (figure 6) a small session of a student
browsing in the ILE. To access the ILE, the student must
use a browser with a PC, laptop, or mobile device (e.g. a
smartphone), and a Facebook account allowing the use of the
Web cam. The top part of the example shows a problem with
the student making a mistake, at the end of the code. In the
bottom part of the problem the student finish successfully the
exercise.

Figure 4. Behavior Graph

Every time a student chose some of the options in the graph,

the engine call the fuzzy logic system to get the tutor action
(feedback, intervention, etc.) and to get the emotion
information so the pedagogical agent can build the graph.
Figure 5 illustrate an example. At the left we can observe
how the student solved the problem in an optimal way. At
the right the student made a mistake.

Figure 6. Session of a Student

353
 For the evaluation of our system we took as reference the REFERENCES
seven features that distinguish a modern ILE or ITS from [1] Matthasdttir, . How to teach programming languages to novice
traditional instructional systems Woolf [6]. Table 2 shows students? Lecturing or not. in International Conference on Computer
the system evaluation with a metric of Likert values 0 to 5, Systems and Technologies-CompSysTech. 2006.
where 0 represents the features that will be added in future [2] Jenkins, T. On the difficulty of learning to program. in Proceedings of
versions or are not covered, 1 to 4 represents an incomplete the 3rd Annual Conference of the LTSN Centre for Information and
Computer Sciences. 2002.
feature and 5 is a complete feature. The system was
compared against ITSs ASK-ELLE [21], JavaTutor [22], [3] Gomes, A. and A.J. Mendes. Learning to program-difficulties and
solutions. in International Conference on Engineering Education
SQLT-Web [23] y ELM-ART [24]. ICEE. 2007.
[4] Bernard, M. and E. Bachu, Enhancing the Metacognitive Skill of
Table 2. Comparison against other ILE or ITS systems. Novice Programmers Through Collaborative Learning, in
Metacognition: Fundaments, Applications, and Trends. 2015,
Characteristic Java ASK- Java SQLT- ELM-
Springer. p. 277-298.
Sensei ELLE Tutor Web ART
[5] SIGCSE. 2015 [cited 2015]; Available from: http://www.sigcse.org/.
Generality 3 3 1 1 1
Student 3 3 3 3 3 [6] Woolf, B.P., Building Intelligent Interactive Tutors. 2009, Amherst,
Model Massachusetts, Estados Unidos: Morgan Kaufmann Publishers.
Expert Model 5 5 5 5 5 [7] Picard, R.W., Affective computing. 2000: MIT press.
Mixed 2 2 1 1 1 [8] Brusilovsky, P. and E. Milln. User models for adaptive hypermedia
Initiative and adaptive educational systems. in The adaptive web. 2007.
Interactive 5 5 5 5 5 Springer-Verlag.
Learning [9] D'Mello, S., R.W. Picard, and A. Graesser, Toward an affect-sensitive
Instructional 2 3 3 3 3 AutoTutor. IEEE Intelligent Systems, 2007(4): p. 53-61.
Model [10] DMello, S., et al. AutoTutor detects and responds to learners
Self- 2 1 1 1 1 affective and cognitive states. in Workshop on Emotional and
improvement Cognitive Issues at the International Conference on Intelligent
Total 22 22 19 19 19 Tutoring Systems. 2008.
assessment [11] Cingolani, P. and J. Alcal-Fdez, jFuzzyLogic: a java library to
design fuzzy logic controllers according to the standard for fuzzy
control programming. International Journal of Computational
According to the total assessment, Java Sensei achieves a Intelligence Systems, 2013. 6(sup1): p. 61-75.
score of 22 points, tied with ASK-ELLE, and with a higher [12] Hall, M., et al., The WEKA data mining software: an update. ACM
SIGKDD explorations newsletter, 2009. 11(1): p. 10-18.
score than the others ILEs/ITSs. We still are working to
[13] Ekman, P., Facial Expressions. 1999.
include Mixed Initiative and Self-improvement features.
[14] Inversions, S., FaceGen modeller (Version 3.3)[computer software].
Toronto, ON: Singular Inversions, 2008.
IV. CONCLUSIONS AND FUTURE WORK [15] Likert, R., A method of constructing an attitude scale. Scaling: a
sourcebook for behavioral scientists. Chicago: Aldine, 1974: p. 233-
The ILE has been implemented to be acceded from 243.
different platforms (operating systems, web browser, [16] Owen, S., et al., Mahout in action. 2011: Manning .
computers, etc.) with open source code in programming [17] Aleven, V., et al., A new paradigm for intelligent tutoring systems:
languages and software libraries like Java, Javascript, Example-tracing tutors. International Journal of Artificial Intelligence
MongoDB, JSON, jFuzzyLogic, Apache Mahout, Weka, in Education, 2009. 19(2): p. 105-154.
jQuery, jQueryMobile among others. Right now, a group of [18] Aleven, V., et al. The cognitive tutor authoring tools (CTAT):
preliminary evaluation of efficiency gains. in Intelligent Tutoring
programming Java professors is starting to test the ILE with Systems. 2006. Springer.
students from our university (Instituto Tecnolgico de [19] Kumar, R., et al. Comparison of Algorithms for Automatically
Culiacn). For future work, we want to include multimodal Building Example-Tracing Tutor Models. in Educational Data Mining
affect recognition, including identification of emotion by 2014. 2014.
text, voice, and EEG devices. We also want to work with [20] Doignon, J. P., & Falmagne, J. C. Knowledge spaces. 1999. Springer.
emotions more learning centered (engagement, boring, [21] ASK-ELLE (Haskell Tutor): Gerdes, A. (2012). Ask-Elle: a Haskell
frustration, etc.). Last we are developing a new version Tutor. PhD thesis, Universiteit Utrecht.
oriented to more collaboration work among students and [22] JavaTutor: Wallis, M. (2011). JavaTutor-A Remotely Collaborative,
authoring new exercises by instructors. Real-Time Distributed Intelligent Tutoring System for Introductory
Java Computer Programming-A Qualitative Analysis.
ACKNOWLEDGMENT [23] SQLT-Web: Mitrovic, A. (2003). An intelligent SQL tutor on the
web. International Journal of Artificial Intelligence in Education,
The work described in this paper is fully supported by a 13(2), 173-197.
grant from the DGEST in Mxico. [24] ELM-ART: Brusilovsky, P., Schwarz, E., & Weber, G. (1996,
January). ELM-ART: An intelligent tutoring system on World Wide
Web. In Intelligent tutoring systems (pp. 261-269). Springer Berlin
Heidelberg.

354