With a Little Help From My Friends: An Empirical Study of
the Interplay of Students' Social Activities, Programming
Activities, and Course Success
Adam S. Carter and Christopher D. Hundhausen
Human-centered Environments for Learning and Programming (HELP) Lab
School of Electrical Engineering and Computer Science
Washington State University
Pullman, WA 99164 USA
+1 509-335-6602
cartera@wsu.edu, hundhaus@wsu.edu
ABSTRACT
Computing education researchers have become increasingly
interested in leveraging log data automatically collected within
computer programming environments in order to understand
students' learning processes and tailor instruction to student needs.
While data on students' programming activities has been positively
correlated with their learning outcomes, those data tell only part of
the story. Another part of the story lies in students' social activities,
which, according to social learning theory, can also be predictive of
students' learning outcomes. In order to gain further insight into how
computing students' learning processes influence their learning
outcomes, we present an empirical study that explores the interplay
of students' social activities, programming activities, and course
outcomes in an early computing course. By analyzing log data
collected through a programming environment augmented with a
social networking-style activity stream, we found that answers to
questions posed through the activity stream were positively
correlated with students' ability to make programming progress, and
their eventual success in the course. Based on our findings, we
present recommendations for the design of pedagogical
environments to support a more social programming process.
Categories and Subject Descriptors
D.2.8 [Metrics]: Performance measures, Process metrics, Product
metrics.
K.3.1 [Computer Uses in Education]: Collaborative learning;
K.3.2 [Computer and Information Science Education]:
Computer science education.
General Terms
Design, Experimentation, Human Factors.
Keywords
Learning analytics, Predictive models of student performance and
achievement, Educational data mining, Activity streams, Social
networking.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for
components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to
post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions fromPermissions@acm.org.
ICER '16, September 08-12, 2016, Melbourne, VIC, Australia
© 2016 ACM. ISBN 978-1-4503-4449-4/16/09…$15.00
DOI: http://dx.doi.org/10.1145/2960310.2960322
1. INTRODUCTION
In recent years, the ease with which learning process data can be
collected, coupled with the availability of low-cost, high-power
machines to store and process such data, has led to an explosion in
the field of learning analytics [29]. Computing education
researchers, for instance, have collected data on students'
programming processes, and have used those data to identify
programming difficulties and how they are resolved [2]. Similarly,
researchers have created predictive models that relate students'
programming behaviors to course outcomes [1, 7, 27]. While these
models have achieved only modest predictive power, they do pave
the way toward a future in which computing educators, with
minimal effort and involvement, can leverage continuous updates
on individual student learning processes and achievement.
Even though programming activities figure prominently in learning
success, any approach that attempts to predict student success based
solely on programming activities may overlook other factors
critical to students' success or failure. For example, according to
social learning theory, it makes sense to consider students' social
activities: how they interact with their peers during the
programming process. Indeed, past research provides evidence that
a student's social behavior is a significant predictor of learning
outcomes in computing courses [12]. This suggests that a broader
picture of student achievement might be obtained by examining the
interplay of a student's social behavior and programming activities.
In an effort to observe students' social and programming behaviors
as they work on course programming assignments, we have
developed a social programming environment (SPE), which
augments a traditional IDE with features commonly found in social
networking software [8]—most prominently, an activity stream in
which students can asynchronously communicate about their
ongoing programming activities. Our SPE provides an empirical
foundation for studying the interplay of social behavior,
programming behavior, and student achievement in computing
courses. As a starting point for such an investigation, we propose
three basic research questions:
RQ1: What kinds of programming questions do students pose
within an SPE?
RQ2: How does programming behavior influence social behavior
and vice versa?
RQ3: What is the relationship between students' social
participation and course grades?
In this paper, we present an empirical study that uses log data
collected within our SPE to address these questions. We found that
students who pose questions, receive a suggestion, and make a
follow-up post were significantly more likely to have made positive
progress towards correct programming solutions. Furthermore, the
scores received for programming solutions produced by these
students were significantly more likely to exceed the class average.
Our study makes two key contributions to the computing education
literature. First, while others have studied the relationship between
students' programming activities and course outcomes, this paper
presents the first-ever empirical study that uses automatically
collected log data to explore the interplay between students'
programming activities, social activities, and course outcomes. As
such, it presents a more rounded attempt to characterize the
relationship between computing students' learning processes and
learning outcomes. Second, our study contributes evidence-based
recommendations for the design of future pedagogical
environments to support a more social computer programming
process in early computing courses.
2. BACKGROUND AND RELATED WORK
This research draws on two lines of related research work that (a)
has used students' programming data to predict their learning
outcomes, and (b) has explored the influence of students’ social
activities on their learning outcomes. Guided by social learning
theory, this research also builds on our prior effort to build a social
programming environment to study the interplay of programming
and social activities. Below, we review these lines of related and
foundational work.
2.1 Using Programming Data to Predict
Learning Outcomes
A large body of educational research has explored the extent to
which various learner variables are able to predict learning
outcomes or future learning behaviors. These variables include the
learner's background [e.g. 6, 16], prior knowledge [e.g. 6, 19],
cognitive abilities [e.g. 23], time-on-task [e.g. 24] and learning
attitudes [e.g. 5]. While this line of research shares our interest in
drawing associations between students and course outcomes, it
differs from our work in that it is based on written surveys
administered periodically, rather than on continuous learning
process data automatically collected through a learning
environment.
A more recent line of research has focused on leveraging a
continuous stream of data in order to identify patterns of learning
associated with positive learning outcomes. As such, it falls within
the emerging areas of educational data mining and learning
analytics [3, 25], which, in many STEM fields, have been used to
gain insights into the processes that underlie student learning, and
ultimately to better tailor instruction. A foundational idea is to build
learner models that infer learners' background knowledge, learning
strategies, and motivations from learning process data [19]. In turn,
such models are used to adapt instruction to learner needs.
Duval and Verbert [26] categorize two approaches for the
application of learning analytics: one that focuses on identifying
patterns of behavior, and another that focuses on deriving
interventions aimed at improving the learning process. In
computing education, the former approach has been used to
describe compilation behavior [e.g. 2, 15] and to identify behaviors
associated with eventual success or failure in a course [e.g. 1, 7, 14,
28]. In computing education, the latter approach (developing
learning interventions) has been less explored, although there are
some notable exceptions [e.g. 10, 20]. In both cases, researchers
have yet to seriously capitalize on data generated from students'
online social interactions.
2.2 The Influence of Social Activities on
Learning Outcomes
Students' online social interactions have been widely studied. For
example, researchers have examined the impact of social
participation on course performance [12] or on particular skills [e.g.
metacognition see 21]. Alternatively, researchers have created
online social environments to facilitate pedagogically beneficial
activities such as peer code reviews [11] or to generate and critique
test questions [9]. In these cases, researchers correlate usage of the
system with a proxy for course performance—often an exam or
final grade [e.g. 11, 18]. In contrast to prior work, the research
presented in this paper directly studies the influence of students'
unstructured question-asking behavior on their future coding
behaviors.
2.3 Social learning Theory
This work is predicated on the theoretical frameworks of social
learning. Situated Learning Theory [17] holds that participation in
a community of practice, which involves both the observation of
others and actual participation in community activities, facilitates
learning. Furthermore, Bandura's self-efficacy theory [4] posits that
students develop a positive sense of their own programming
abilities (so-called self-efficacy) by being able to observe the
activities of their peers, by being able to evaluate themselves
relative to their peers, and by engaging in their own performances.
Resonant with this theory, Rosson et al. [22] identified strong
positive correlations between a number of attitudinal variables,
including self-efficacy, and a learner's orientation towards the
computing discipline. Both of these social theories of learning
suggest that learners may have difficulties making progress if they
are forced to program in isolation from a broader community, as
has traditionally been the case in computing education. This is
especially true with the type of individual programming
assignments that are common in early computing courses.
2.4 Social Programming Environments
Guided by these theories, our research lab has been investigating
the impact of social programming environments (SPEs; see Figure
1, next page) on learning processes and outcomes in early
computing courses. In previous work, we found that SPEs can have
a positive impact on students' sense of community [8], and that
regular participation in such an online social environment is
positively correlated with course outcomes [12]. Furthermore, we
found that data collected within our SPE can be used to construct
predictive models of student success [7]. While our SPE has
allowed us to collect the data necessary to explore the interplay
between programming behavior, social behavior, and student
performance within the context of early computing courses, we do
not claim that an SPE is a necessary component for facilitating
online social discourse within a classroom. Indeed, in past research,
we have studied computing students’ social discourse within a
variety of online environments, including Facebook and more
traditional learning management systems [12, 13].
3. STUDY METHODOLOGY
3.1 Participants
Participants in this study were enrolled in the spring 2014 offering
of CS2 at Washington State University. Focusing on the C and C++
programming languages, this 15 week course had three 50-minute
 Figure 1: The OSBLE+ social programming environment
lectures and one 170-minute lab period per week, three exams (two
midterms and a final), and seven individual programming
assignments. The course enrolled 140 students, 129 of whom
finished the course and received a grade. 108 of these students (100
men, 8 women) consented to releasing their learning process data
(automatically collected within the SPE) and course grades data for
this study.
3.2 Data Collection, Sampling, and Analysis
Method
Our SPE (see [8]) was used to collect students' online social activity
and programming behaviors. The SPE collected 21,952,494 points
of interaction and 10,720 discussion posts. In an attempt to make
the data manageable, we employed a principled approach to
sampling the content. To this end, we randomly sampled 10
students who received each of the five possible course grades (A,
B, C, D, and F). While we were able to sample 10 students from
each of the A, B, and C levels, only four students were available
who received D's in the course, and only two students were
available who received F's. Therefore, our sample could include
only 36 students, instead of the 50 that we targeted.
Having chosen our sample, we next identified the 2,352 instances
in which students in our sample were involved in some kind of
social activity—either a post or reply on the SPE's activity stream.
Since our research questions related to the interplay of social and
programming behaviors, we opted to focus only on social activities
(posts and replies) that had to do with programming activities. This
refinement yielded 461 posts. Finally, in order to focus our study
even more intently on our research questions, we further pruned the
sample to include only posts and replies that were part of a thread
that was initiated by a programming question. This resulted in a
final sample of 93 threads.
For each of the 93 threads of social activity in our sample, we
carried out the analyses described in Table 1 in order to provide a
foundation for addressing our research questions.
In order to categorize question content (#1 in Table 1), we extended
a previously used content coding scheme (see [13]), which we
present in Table 2. To verify the validity of this coding scheme for
the present analysis, the two authors independently coded a 20%
sample of the corpus (n = 19 posts), attaining an overall agreement
of 88% (0.83 kappa). Having established a sufficiently high level
of inter-rater reliability for this coding scheme, the first author
coded the remaining posts.
Table 1. Analyses of Programming Questions
# Description of Analysis Relevant RQs
1 Categorize question content RQ 1
2 Determine whether question related to question RQ 2, RQ3
author’s current build state
3 Determine whether responses to question were RQ 2, RQ3
posted
4 Determine whether programming suggestions to RQ 2, RQ3
address question were posted
5 Determine whether question author explicitly RQ 2
acknowledged suggestions (with, e.g., “thank
you”)
6 Determine whether question author resolved the RQ 2
question in a future build
 Table 2. Programming Question Content Categories In order to increase the likelihood that any progress made by
question authors was influenced by their interaction in the
Category Description Example
corresponding activity stream thread, we considered subsequent
COMPILE Question relates to an “After I debug my builds that occurred within a reasonable time—two days—of the
issue encountered code, I got this error,
during program question author's last post to the thread. However, given that some
can anyone explain it to
compilation. me? thank you Error 1 students compiled their code infrequently, we required that a
error LNK2019: minimum of five compilations be considered. In some cases,
unresolved external examining a minimum of five compilations led us to consider
symbol […]”
compilations beyond the two-day window.
IDE Question relates to the “Okay, does anybody else
operation of Visual consistently get the
Two caveats come with the analysis approach just described. First,
Studio problem where cin and because it requires evidence that (a) programming questions be
cout are underlined with related to programming context, and (b) programming progress fall
red and VS gives you the within a reasonable time after related programming questions are
error "cout is
ambiguous"?” answered, our analysis approach may be seen as overly
conservative. For example, it would have been possible for students
IMPLEMENTATION Question asks for tips “If both the player and
on how to best to have asked a related coding question before they started coding
the computer draw the
implement an algorithm same type of hand(say their solution. Likewise, it would have been possible for students
or function. This is two pairs), who wins?” to have made positive strides towards a correct solution outside of
often, but not always,
related to the
our two-day, five-build window. Thus, it is possible that our
requirements of a given analysis approach failed to identify some relationships between
lab or assignment. programming and social behavior.
LANGUAGE Question asks about the “Does anyone know if the Second, just because we are able to find evidence of positive
C/C++ language, or strtok() keeps the old
about a programming string or does it fully erase programming progress that closely follows related social activity,
issue related to the it?” this does not mean that such progress was made because of the
misunderstanding of social activity; indeed, such an assumption would fall prey to the
syntax.
post hoc, ergo proctor hoc fallacy. Clearly, our study is
RESOURCES Question requests What did you guys use correlational, not causal. These two caveats should be borne in
external programming to create your UML
resources or tips. mind in interpreting the results that follow.
diagrams?

RUNTIME Question relates to an Does anyone know what 4. RESULTS

issue encountered "vector subscript out of
during the runtime range" means and to fix
execution of a student's it?
code
In order to determine whether a question was related to the student's
current build state (#2 in Table 1), we examined the build that came
immediately before or immediately after the question was posed.
We considered these builds to be related to the question only if we
could find clear evidence connecting the question to the state of the
code in the build. For example, if a student's question asked about
substrings, the student's previous or subsequent build would need
to contain code that uses substrings, attempts to use substrings, or
calls a function that uses substrings. In cases in which both builds
occurred more than a day before or after a question was posed, we
coded the question as not related to the current build state.
Whether a given question had responses (#3 in Table 1) could be
gleaned directly from a transcript of the activity stream. If at least
one of the responses to a question suggested concrete action that
the question author might take to address the question, we
concluded that a suggestion was present (#4 in Table 1). Evidence
of a question author's acknowledgment of a suggestion (#5 in Table
1) came in the form of any follow-up post by the author indicating
that the author had read the post (e.g., "Thank you" or "What
about…").
Finally, in order to determine whether the question author altered
his or her code in a future build so as to resolve the original question
posed (#6 in Table 1), we looked for clear evidence that the
student's code had moved towards a resolution of the issue. For
example, if the student asked a question about how to resolve a
syntax error with using substring in her program, the error
mentioned in the question would need to be resolved in a
subsequent build.
Using the categories of Table 2, Figure 2 presents a high-level
breakdown of the content of the 93 questions in our sample. In this
figure, the bars represent the percentage of questions that fall into
each content category; all bars add up to 100%. The shading of each
bar indicates the proportion of questions in each content category
that were related to the question author's immediate coding context.
As this figure suggests, a strong majority of the questions (74%)
focused on IMPLEMENTATION (46%) or LANGUAGE (28%). The least
common questions focused on COMPILE (4%) and IDE (4%) issues.
Moreover, we see that questions having to do with RUNTIME,
COMPILE, and LANGUAGE issues were most commonly associated
with a student's active coding solution. In contrast, we see that none
of the questions in the RESOURCES category was related to an active
coding solution. This makes sense given that, by definition,
RESOURCE questions ask for external programming resources or
tips.
4.1 Relationship between Suggestions
Received and Programming progress
82 of the 93 questions (88%) received a response; Slighter fewer
(75, or 81%) received a response containing a programming
suggestion to address the question. Did question authors'
participation in the activity stream relate to their subsequent ability
to make programming progress? To explore this question, Figure
3 considers the extent to which we found evidence that question
authors made positive progress when they (a) did not receive a
suggestion, (b) received a suggestion but did not acknowledge the
response, or (c) received a suggestion and acknowledged the
response. This figure illustrates a clear increase in success rates:
only 38% of question authors made progress without a suggestion,
versus a success rate of 85% when authors received and
acknowledged a suggestion from another student.
According to chi-squared test, there was a significant association
between the type of feedback received (no suggestion, suggestion 50%
+ no acknowledgement, suggestion + acknowledgement) and
40%
whether or not a future build demonstrated progress towards a
correct solution (2(2) = 7.80, p = 0.02). A post-hoc z-test revealed 30%
a significant difference (p < 0.05) between question authors who 20%
did not receive a suggestion and those who received and
acknowledged a suggestion. The middle condition—question 10%
authors who received a response but did not acknowledge the
0%
response—did not differ significantly from the other two
categories.
4.2 Relationship between Social Activity and
Course Outcomes
We next consider the relationship between discussions anchored in
code and course outcomes, in order to further explore whether Related Not Related
talking about code was a general indicator of success within the
course. We begin by examining the relationship between the
Figure 2: Percentage of questions in each content
number of programming questions posed by a student and both the
category, broken down according to whether or not they
student's final grade, and the average programming assignment
were related to immediate coding context
grade. We failed to find a significant relationship between the
number of questions posed and the student's final grade (F(1,18) =
100%
0.2, p = 0.66) or assignment average (F(1,18) = 0.2, p = 0.66). Next,
we compared students who posted coding questions to those who 80%
did not. Again, we did not find a significant relationship between
these groups and final grade (F(1,105) = 0.03, p = 0.87) or 60%
assignment average (F(1,86) < 0.01, p = 0.96).
40%
The data indicate that simply posting a question about code bears
no relationship to a student's overall class performance. However, 20%
it is possible that receiving programming suggestions on a given
assignment might have positively impacted the grade received for 0%
Without suggestion With suggestion With suggestion and
that assignment. To investigate this possibility, we examined the 23 acknowledgement
instances in which a student asked a question, received a
suggestion, and acknowledged the suggestion with a response. For Figure 3: Percentage of future builds that demonstrate
each instance, we compared the grade the student received on that progress based on social feedback received
assignment to the average grade of students that did not receive help
on the assignment. Figure 4 depicts this relationship. Out of these
23 observations, only three observations were below the class
average. When taken as a whole, we see that the average score
received by this group was 10% higher (88% vs 78%) than the class
average. A two-sample t-test with equal variance not assumed
found this difference to be statistically significant (t(23) = 2.65, p
< 0.01).
5. VIGNETTES
The prior section discovered statistically-significant positive
relationships between participation in online discussions about
code, coding progress, and course outcomes. In this section, we
consider a series of vignettes, transcribed directly from the activity
stream data collected in our study, that illustrate the interplay
between student's coding behavior and social interactions. In
exploring these vignettes, we see how rich discussions centered
around coding can positively impact a student's coding progress.
Likewise, these vignettes give us insights of what might happen Figure 4: Grades of students who received help on
when these kinds of discussions fail to materialize. assignment compared with those that did not for a given
assignment
5.1 Vignette 1: Multiple Inheritance
The concept of inheritance in object oriented programming has to compile when inheriting from a single parent, but cannot figure
recently been introduced in class and James1 would like to use out the syntax for multiple inheritance. He asks for help with the
inheritance in his game of Battleship. James would like to build a issue on the OSBLE+ activity stream:
subclass that inherits from multiple parents. He can get his program

1All names used in these vignettes are pseudonyms.

James: How do I make a class that is composed of two other
classes? This is giving me an error: class Player : public
Board : public Stats
Seeing James' question, Sharon provides a suggestion on how to
accomplish multiple inheritance:
Sharon: Never done it myself, but this might help [url]… Try class
Player : public Board, public Stats
James sees Sharon's question, modifies his class definition as
suggested. His project compiles and he reports back his progress:
James: That's it. Crisis averted!
Timothy, who is also toying with multiple inheritance, sees James'
conversation and confirms Sharon's suggestion:
Timothy: This works. I have for instance, 5 different boat classes,
all of which are inherited from a mothership parent
class…
Once again, James thanks Timothy for his confirmation. In the final
post of the conversation, Steven, an upperclassman, offers a
cautionary message regarding multiple inheritance:
Steven: While you are all correct and multiple inheritance does
work in c++. It can lead to a lot of errors, and stuff like
the diamond problem…
Less than three hours after posing the question, James was back on
track.
5.2 Vignette 2: Understanding Requirements
Jessica has started work on her homework assignment, but her
unfamiliarity with the game of poker is preventing her from making
progress on how to properly implement a required function. Unable
to make headway, she uses OSBLE+ to ask the class how about
how the function should operate:
Jessica: In highCard(), it returns one card which has the highest
rank and the highest suit, right? I just want to make sure
since I am unfamiliar with [poker]
Stacie responds first:
Stacie: Your function wants to return the highest rank over the
highest suit. However, if you had, say 2 jacks in your
hand, it would return the Jack of Spades, before the Jack
of hearts. But say you had an Ace no matter what the suit
is it would return the ace over the jacks. Hope this makes
sense.
Fred, who is also working on his poker assignment, asks a
clarifying question:
Fred: Isn't high card just for a no hand combination? Meaning
2 jacks would make a pair, high card only returns in the
instance of you have no hand right?
Barry responds to Fred:
Barry: It would only be relevant if you had a jack and the dealer
had a jack. Suits are only there to be compared if there is
a tie if both have the same high card ...and flushes of
course.
Fred replies to Barry:
Fred: I took the said suit function out, but I'm putting it back in
[…]
To close out the conversation, Jessica thanks the discussion
participants:
Jessica: Thanks guys
Jessica continues to work diligently, and a day later, she has a
working implementation of her function. However, not only did
Jessica help herself, she also helped Fred, who might otherwise
have been too shy to ask a question.
5.3 Vignette 3: Learning a New Concept
In the latest homework assignment, Beau is required to implement
a basic factory pattern for creating different employee types. As this
concept was recently introduced, Beau has no experience writing
factories and is thoroughly confused. He uses OSBLE+ to reach out
for help:
Beau: Any ideas to write studentFromString, staffFromString,
facultyFromString? How do those function help the
fromString funtion?
Justin responds with an explanation of how the factory pattern
works:
Justin: The fromString function determines whether the
employee is student, staff, or faculty then calls
studentFromString, staffFromString, facultyFromString
respectively. In those functions it sets all the information
it gets from the string then returns it back to the
fromString function. The fromString function then
returns that.
Apparently, other students are, like Beau, also struggling to
understand how to implement a factory. Jessica writes:
Jessica: So a majority of the code should be in the Factory and
not in main? […] What I'm having trouble with is
bringing it back into main [partial code snippet]
Steven responds that in his case, the majority of his code is indeed
inside the factory. James follows this with a code snippet of how he
"brought it back into main":
Stephen: I just use: Employee *employ; and then in the while loop,
just write: employee = factory.fromString(temp);
cout<<employee->toString()<<endl; Hopefully, it can
help you.
Sandwiched between the discussion between Justin, Jessica, and
Steven, Beau thanks the students for their suggestions. At the time
Beau originally posed his question, he had not written any factory-
related code. Less than a day later, Beau had implemented a fully-
operational factory.
5.4 Vignette 4: Unacknowledged Question #1
Like Beau from Vignette 3, Rachel is having trouble implementing
a factory for her homework assignment. Also like Beau, Rachel
decides to seek help on OSBLE+. Unfortunately, she accesses
OSBLE+ a day after Beau's discussion. Therefore, the help she
seeks has fallen from the first page of the activity feed. Unaware of
the previous discussion, Rachel makes her own post asking for
implementation strategies:
Rachel: I'm not sure what I should do in employeeFactory class,
anyone can give me some ideas? Thank you.
Unfortunately, Rachel's post does not receive the same attention as
Beau's post; she receives no responses. Without help, Rachel has to
go it alone. Over the course of the next two days (the maximum
window of observation), Rachel makes progress towards correctly
implementing a factory. However, at the end, she has still not
implemented a fully working solution.
5.5 Vignette 5: Unacknowledged Question #2
Sean encounters an issue related to IFNDEF/ DEFINE preprocessor
directives in his code. For some reason, large chunks of his code
are being categorized as an "inactive preprocessor block."
Confused, he poses his question on OSBLE+:
Sean: So, in one of my header files I accidentally hit some key
that made everything within that ifndef an "Inactive
Preprocessor Block" (when I minimize it says that)... so,
what key did I hit and how do I reactivate it?
Unfortunately, nobody responds to his post. In an effort to solve his
problem, Sean alters his preprocessor directives in a way that fixes
his immediate error, but also introduces a potential bug that may
cause issues in the future.
5.6 Vignette 6: Unacknowledged Response
Jon would like to create an array of pointers to use in his homework
implementation, but he cannot figure out how to properly initialize
the data structure. Looking for guidance, Jon poses his question on
OSBLE+:
Jon: I'm trying to use an Employee ** employee1 in my main
function to hold all the employees in my .csv file.
However there is no way of initializing that. So how
would I hold an array of employees in main so I can
access them whenever I want (i.e. for the paycheck)?
Tyler responds with a code snippet demonstrating how he had
initialized his array of pointers:
Tyler: I did employee1 = new Employee*[100] to initialize my
array of employees
Jon never responds. Inspecting Jon's code, one finds that he appears
to have implemented a different solution strategy. Future builds of
Jon's solution do not include code related to 2D array initialization.
6. DISCUSSION
We organize our discussion of the results around our original
research questions.
6.1 What Kinds of Programming Questions
Do Students Pose Within an SPE?
Based on the content coding scheme presented in Table 2, nearly
three-quarters (74%) of the programming questions posed within
the OSBLE+ activity stream related to either language or
implementation. The remaining questions had to do with runtime
(10%), resources (8%), IDE (4%) and compilation (4%) issues.
Given that this study focused on a CS2 course, it is perhaps
unsurprising that so few questions had to do with IDE and
compilation issues, which students at the CS2 level likely had
already grappled with in previous computing courses. However, we
found it somewhat surprising that such a small percentage of
questions related to runtime issues (e.g., unexpected runtime
behavior or runtime exceptions), which are notoriously difficult.
We speculate that social interaction in an activity stream could
provide a strong basis for addressing such issues.
6.2 How Does Programming Behavior
Influence Social Behavior, and Vice
Versa?
As illustrated in Figure 2, we found that at least half of all
programming questions relate to the author's most recent program
solution at the time the question was posed. Questions related to
language, compile, and runtime issues were most commonly
associated with the question author's current programming
solution. This makes sense, as each of these question types is likely
to arise as these issues are encountered by students. In contrast, not
a single question about coding resources was related to the student's
active programming solution. Again, this makes sense: students
who are looking for programming resources are probably still
formulating the problem and have not yet transitioned to writing
actual code.
The vignettes presented in the previous section illustrate some ways
in which social activity can influence coding behavior. In cases in
which questions generated a healthy back-and-forth between their
authors and other students, authors were likely to ultimately fix
their issues. Furthermore, the vignettes illustrate the way in which
discussions can attract students with similar issues. In this respect,
we can see social activity as a mechanism for disseminating
knowledge and creating bonds. The results of a previous study (see
[8]), in which students using our SPE exhibited a significant
increase in sense of community, provide additional empirical
support for this observation.
6.3 What Is the Impact of Social Participation
on Students' Ability to Successfully
Complete Their Assignments?
Our quantitative results indicate that social behavior can have a
significant influence on future coding behavior. When we
examined conversations that contained a suggestion and an
acknowledgement by the post's author, we saw the likelihood of the
author making progress towards a correct solution increase
significantly. Without a suggestion, only 38% of question authors'
future compilations demonstrated evidence of progress. With an
unacknowledged suggestion, this figure increased to 63%. And this
number increased to 85% when the author received and
acknowledged a suggestion. This result suggests that students who
notice a helpful suggestion are likely to incorporate the suggestion
into their own code, and that by doing so, they increase their
chances of developing a correct solution.
In addition to helping students make positive solution progress,
asking questions and receiving help also led to statistically-
significant improvement in homework grades—at least for students
who asked a question, received a suggestion, and acknowledged the
suggestion. This finding suggests that asking for and receiving help
on an assignment not only leads students towards a correct solution
in the immediate sense; it also increases the likelihood of ultimately
receiving a high grade on an assignment.
7. DESIGN IMPLICATIONS
Having addressed our original research questions, we next consider
the implications these results might have for future pedagogical
environments that aim to support individual programming
assignments in early computing courses. Given the positive
association between social interaction and programming behavior
we identified, we believe that providing a space for students to
discuss programming problems should be a key feature of any such
pedagogical environment. Moreover, the results presented here
provide guidance on how such pedagogical environments should be
designed to support social discussions. We consider three specific
design implications below.
First, as illustrated by our quantitative analysis, code-centric social
interaction becomes more effective when a student receives a
suggestion. The impact becomes significant when the question
author acknowledges the suggestion. This finding implies that a
pedagogical environment should specifically highlight unanswered
questions. For example, a revised version of our SPE might have a
section of the activity feed dedicated to unanswered questions.
Second, to ensure that those who ask for help actually read answers
to their questions, an effective pedagogical environment needs to
somehow bring potential solutions to the attention of question
authors. To this end, the environment might contain a notifications
area that is updated whenever a new solution has been submitted to
a question. Along these lines, it might be helpful to incorporate a
mechanism for both marking a potential solution and for marking
the "best" solution, similar to what is currently provided by
StackOverflow (2012).
Third, as highlighted by Vignettes 3 and 4, students may pose
similar questions. Due to the ephemeral nature of an activity feed,
students could easily be unaware that their current question has
already been asked and answered. This implies the need for
students to be made aware of prior questions. Such awareness could
be provided through, for example, an ability to search past
questions, perhaps using hashtags common on social media sites.
Alternatively, the environment could recommend other posts based
on the content of the question (e.g. "The following posts may be
related to your current question"). Yet another approach could
leverage the continuous nature of data logging to relate
conversations. For example, for a question related to a compiler
error, the pedagogical environment might be able to automatically
suggest a discussion that was created by another student under
similar coding circumstances.
8. LIMITATIONS
It is important to underscore three limitations of the research
presented in this paper. First, while we were able to identify when
students incorporated suggestions into their homework, our data
does not indicate how students actually used online discussions to
solve their programming issues. Understanding the process by
which students incorporate feedback into their solutions would
require us to augment our log data with retrospective interview or
survey or survey data—an important direction for future research.
Second, it is possible that our study suffers from a sampling bias. It
is possible that the sample of students selected for this study are not
representative of the entire dataset. Alternatively, the inclusion of
more A, B, and C students in our sample may have resulted in
artificial significance findings. We can address the first threat by
increasing the number of students in our sample; however,
addressing the second threat would require us to gather more data.
Lastly, while we established inter-rater reliability for the content
coding scheme (see Table 2), inter-rater reliability was not
established for the other judgments required to perform the data
analyses described in Table 1, item 6. We will need to address this
shortcoming in future research.
9. CONCLUSION
In this paper, we have empirically investigated the relationship
between programming activities and programming-focused activity
stream discussions within the context of individual programming
assignments in an early computing course. We began with a
quantitative analysis that showed that questions posed by students
are likely to be related to a current coding issue. We also discovered
that students who received and acknowledged a suggestion to their
questions were significantly more likely to fix their issues.
Furthermore, these students were likely to turn in assignments
whose scores were significantly higher than the class average.
These findings were elucidated through a series of vignettes that
illustrated how programming discussions can influence eventual
success, how programming discussions can positively impact other
students, and what happens when such discussions do not occur.
The work presented in this paper provides a compelling foundation
for future work. First, we would like to broaden our analysis by
expanding our selection criteria to include programming questions
that occur as a sub-thread of another post. Second, we would like
to explore how programming discussions impact students other
than the question author. This includes both students who
participate in the discussion and students who merely "lurk." Third,
we would like to explore factors that contribute to question asking
and answering. For example, do students with high social capital
receive more responses? Alternatively, are successful students
more likely to ask for and receive help? Does this create a divide
between students who are capable of clearly formulating
programming questions and those who are not? Finally, we would
like to explore the impact of the design modifications suggested by
the results of this study, in order to further harness the potential for
online social interaction to positively influence learning outcomes
in early computing courses.
10. ACKNOWLEDGMENTS
This project is funded by the National Science Foundation under
grant no. IIS-1321045. Carla DeLira assisted with the coding of the
activity stream logs. We are grateful to the participants in the Social
Analytics Workshop at ICER 2015 for their suggestions on how to
pursue the analyses presented in this paper.
11. REFERENCES
[1] Ahadi, A. et al. 2015. Exploring Machine Learning Methods
to Automatically Identify Students in Need of Assistance.
Proceedings of the Eleventh Annual International
Conference on International Computing Education
Research (Omaha, NE, USA, 2015).
[2] Altadmri, A. and Brown, N.C.C. 2015. 37 Million
Compilations: Investigating Novice Programming Mistakes
in Large-Scale Student Data. Proceedings of the 46th ACM
Technical Symposium on Computer Science Education
(Kansas City, MO, USA, 2015), 522–527.
[3] Baker, R.S.J. and Siemens, G. 2014. Educational data
mining and learning analytics. The Cambridge Handbook of
the Learning Sciences. Cambridge University Press. 253–
274.
[4] Bandura, A. 1997. Self-efficacy: the exercise of control.
Worth Publishers.
[5] Bergin, S. et al. 2005. Examining the role of self-regulated
learning on introductory programming performance. Proc.
2005 ACM International Computing Education Research
Workshop. ACM Press. 81–86.
[6] Bransford, J. et al. eds. 1999. How people learn: Brain,
mind, experience, and school. National Academy Press.
[7] Carter, A.S. et al. 2015. The Normalized Programming State
Model: Predicting Student Performance in Computing
Courses Based on Programming Behavior. Proceedings of
the Eleventh Annual Conference on International
Computing Education Research (2015).
[8] Carter, A.S. and Hundhausen, C.D. 2015. The Design of a
Programming Environment to Support Greater Social
Awareness and Participation in Early Computing Courses.
J. Comput. Sci. Coll. 31, 1 (Oct. 2015), 143–153.
[9] Denny, P. et al. 2011. PeerWise: Exploring Conflicting
Efficacy Studies. Proceedings of the Seventh International
Workshop on Computing Education Research (Providence,
Rhode Island, USA, 2011).
[10] Hartmann, B. et al. 2010. What would other programmers
do: suggesting solutions to error messages. Proc. 28th
Conference on Human Factors in Computing Systems.
ACM. 1019–1028.
[11] Hundhausen, C.D. et al. 2011. Online vs. face-to-face
pedagogical code reviews: An empirical comparison.
Proceedings 2011 SIGCSE Symposium on Computer
Science Education. ACM Press. 117–122.
[12] Hundhausen, C.D. et al. 2015. Supporting programming
assignments with activity streams: an empirical study.
Proceedings of the 46th ACM Technical Symposium on
Computer Science Education (New York, 2015), 320–325.
[13] Hundhausen, C.D. and Carter, A.S. 2014. Facebook me
about your code: An empirical study of the use of activity
streams in early computing courses. Journal of Computing
Sciences in Colleges. 30, 1 (2014), 151–160.
[14] Jadud, M.C. 2006. Methods and tools for exploring novice
compilation behaviour. Proce. Second International
Workshop on Computing Education Research. ACM. 73–
84.
[15] Jadud, M.C. and Dorn, B. 2015. Aggregate Compilation
Behavior: Findings and Implications from 27,698 Users.
Proceedings of the Eleventh Annual International
Conference on International Computing Education
Research (Omaha, NE, USA, 2015).
[16] Jeske, D. et al. 2014. Learner characteristics predict
performance and confidence in e-Learning: An analysis of
user behavior and self-evaluation. Journal of Interactive
Learning Research. 25, 4 (2014), 509–529.
[17] Lave, J. and Wenger, E. 1991. Situated Learning:
Legitimate Peripheral Participation. Cambridge University
Press.
[18] Luxton-Reilly, A. et al. 2012. The Impact of Question
Generation Activities on Performance. Proceedings of the
43rd ACM Technical Symposium on Computer Science
Education (Raleigh, NC, USA, 2012).
[19] Ma, W. et al. 2014. Intelligent tutoring systems and learning
outcomes: A meta-analytic survey. Journal of Educational
Psychology. 106, 2007 (2014), 901–918.
[20] Mujumdar, D. et al. 2011. Crowdsourcing suggestions to
programming problems for dynamic web development
languages. CHI ’11 Extended Abstracts on Human Factors
in Computing Systems (Vancouver, BC, Canada, 2011),
1525–1530.
[21] Pifarre, M. and Cobos, R. 2010. Promoting metacognitive
skills through peer scaffolding in a CSCL environment.
Computer-Supported Collaborative Learning. 5, (2010).
[22] Rosson, M.B. et al. 2011. Orientation of Undergraduates
Toward Careers in the Computer and Information Sciences:
Gender, Self-Efficacy and Social Support. ACM
Transactions on Computing Education. 11, 3 (Oct. 2011),
1–23.
[23] Schunk, D.H. 2012. Learning theories: An educational
perspective. Merrill Prentice Hall.
[24] Slavin, R.E. 2011. Educational psychology: Theory and
practice. Pearson Education.
[25] U.S. Department of Education, Office of Educational
Technology 2012. Enhancing Teaching and Learning
through Educational Data Mining and Learning Analytics:
An Issue Brief.
[26] Verbert, K. and Duval, E. 2012. Learning Analytics.
Learning and Education. 1, 8 (2012).
[27] Watson, C. et al. 2014. No tests required: comparing
traditional and dynmaic predictors of programming success.
Proceedings of the 45th ACM Technical Symposium on
Computer Science Education (2014), 469–474.
[28] Watson, C. et al. 2013. Predicting Performance in an
Introductory Programming Course by Logging and
Analyzing Student Programming Behavior. Proceedings of
the 2013 IEEE 13th International Conference on Advanced
Learning Technologies (2013), 319–323.
[29] Wise, A.F. 2014. Designing Pedagogical Interventions to
Support Student Use of Learning Analytics. Proceedings of
the Fourth International Conference on Learning Analytics
And Knowledge (New York, NY, USA, 2014), 203–211.