Computers in Human Behavior 77 (2017) 11e18

Contents lists available at ScienceDirect

Computers in Human Behavior

journal homepage: www.elsevier.com/locate/comphumbeh

Full length article

The effects of teaching programming with scratch on pre-service

information technology teachers' motivation and achievement*
Osman Erol a, *, Adile Aşkım Kurt b
a
Mehmet Akif Ersoy University, Turkey
b
Anadolu University, Turkey

a r t i c l e i n f o a b s t r a c t

Article history: The aim of this study is to examine the effect of programming instruction with Scratch on student
Received 6 November 2016 motivation and their programming achievements. The study group consisted of 52 sophomore students
Received in revised form attending the Department of Computer Education and Instructional Technologies of Mehmet Akif Ersoy
6 August 2017
University's Faculty of Education, Turkey. Participants were randomly divided into two groups in order to
Accepted 10 August 2017
Available online 11 August 2017
have 26 students in both the test group and the control group. During the first seven weeks of the study,
it is aimed that the students will understand programming logic and learn basic programming structures.
For this purpose, participants in the test group were instructed using Scratch, whilst in the control group,
Keywords:
Programming
flowcharting and problem-solving activities were conducted as per the curriculum. During the second
Scratch seven weeks of the study, C# programming language instruction was conducted using the same method
Flowcharts for both the test and control groups. Achievement Test and Motivated Strategies for Learning Ques-
Motivation tionnaire were utilized as data collection tools in the study, and a 3
2 (measurement time x groups)
factorial design was employed. Study findings revealed that programming achievement scores for both
the test and control groups increased at the end of the whole process; however, the increase was
significantly different in favor of the test group at the end of the whole process. It was observed that
motivation scores decreased in the control group, while the test group's scores increased.
© 2017 Published by Elsevier Ltd.

1. Introduction
Especially in basic courses for beginners in programming, the
objective is to teach the students both the basic concepts of pro-
gramming and programming logic in order to prevent potential
problems experienced in advanced programming courses, where
coding instruction is delivered. In this process generally daily lan-
guage and flowcharts, which are designated as so-called codes, are
the tools utilized to teach programming logic by developing algo-
rithms. The fact that algorithm instruction is such an abstract
process causes beginner programmers to only comprehend pro-
gramming logic at a low level and then fail to transfer this
knowledge to other programming languages in the future
*
This study is the summary of the doctoral dissertation titled “The Effects of
Teaching Programming with Scratch on Pre-Service Information Technology
Teachers' Motivation and Achievement” written by Osman Erol and supervised by
Adile Aşkım Kurt.
* Corresponding author.
E-mail addresses: oerol@mehmetakif.edu.tr (O. Erol), aakurt@anadolu.edu.tr
(A.A. Kurt).
(Kinnunen & Malmi, 2008; Kurland, Pea, Clement, & Mawby, 1986).
Problems experienced in algorithm and programming instruction
could lower student motivation in programming courses and the
resulting negative attitudes could negatively affect student
achievement related to programming (Anastasiadou & Karakos,
2011; Lahtinen, Ala-Mutka, & Ja €rvinen, 2005). However, there are
many tools that can be employed as a means for solving this
problem through the teaching of algorithm and basic programming
such as Raptor, Flint, Code.org, Scratch, and Alice. General charac-
teristics of these software are to materialize the algorithm process,
provide rapid testing and editing possibilities to the user (Baldwin
& Kuljis, 2000; Cooper et al., 2006; Maloney, Resnick, Rusk,
Silverman, & Eastmond, 2010). Some of these, such as Raptor,
Flint, and FCPro, are just simple flowchart creation software and
lack design capabilities (Cooper, Dann, & Pausch, 2003). However,
on the other hand, tools that are capable of designing games and
stories such as Alice or Scratch, make it possible for students to
design substantial and unique content; rendering the process more
attractive. One of the most frequently used tools lately in visual
programming of game design is Scratch (Resnick et al., 2009).

http://dx.doi.org/10.1016/j.chb.2017.08.017
0747-5632/© 2017 Published by Elsevier Ltd.
12 O. Erol, A.A. Kurt / Computers in Human Behavior 77 (2017) 11e18
Scratch has been used worldwide to teach algorithm and pro-
gramming logic to students of different ages, from primary school
to college (Maloney et al., 2010), due to its simple interface and drag
and drop structure. Scratch is an effective tool that teaches pro-
gramming logic and is generally preferred by beginner pro-
grammers rather than learning a mere programming language. In
addition, Scratch enables the design of games, stories and anima-
tions thanks to its design-oriented structure, which in turn makes
learning more fun. As these tools employ game design in pro-
gramming teaching, the game design process could also help
learners to comprehend programming logic. Digital game design, in
programming instruction is based on Papert (1991) constructionist
approach. According to Papert (1980), the most effective learning
occurs when individuals are allowed to design and create
(construct) a product instead of instructing the knowledge directly.
Learning takes place through experience while constructing these
products (Harel & Papert, 1991; Kafai, 2006; Mishra & Girod, 2006).
This practice increases the learner's motivation and enables better
learning; thus it could be possible for beginners in programming to
gain increased motivation and a positive attitude towards pro-
gramming. Furthermore, using Scratch, student motivation for
programming courses could be improved, which would in turn
increase their achievement (Malan & Leitner, 2007). Also, learners
who understood programming logic could transfer the acquired
knowledge to the learning of other programming languages (Wolz,
Leitner, Malan, & Maloney, 2009). Regarding these, it could be
argued that the effect of tools and game design processes such as
Scratch on the instruction of programming logic and programming
languages should be scrutinized in connection with learner
motivation.
A review of the existing literature revealed numerous studies
conducted using Scratch in programming instruction. One such
study was conducted by Rizvi, Humphries, Major, Jones, and Lauzun
(2011) with college students. In their study, the effect of game
design activities performed in basic computer lessons on students'
programming achievement in advanced programming lessons was
examined. At the end of the two-year long study, it was observed
that Scratch increased students' attitudes towards programming
and that their achievements were better than those in the group
that did not receive Scratch training. The authors stressed that
game design activities with Scratch developed algorithmic-
thinking skills of the students and that these skills were then
transferred to a programming language. In another study, Westcott
(2008) analyzed the effect of programming instruction using
Scratch on student achievements in Cþþ programming. In the test
group, Cþþ code instruction was integrated within game design
activities using Scratch. Study findings demonstrated no significant
difference between the test scores that included basic structures
such as variable definitions, but there was a significant difference
favoring the test group that included more complex and difficult
programming structures such as decision making and loops. In a
study conducted by Leiva and Salas (2013) with college students,
the effect of game design activities using Scratch on students'
programming achievements were indicated. At the end of the
study, findings showed that game design activities conducted with
the test group increased student course satisfaction and that their
academic success was more comparable to that of the control
group. However, in several studies it was determined that game
design activities conducted with visual tools such as Scratch or
Alice increased the programming achievements of the students,
their attitude, and their motivation for programming (Bishop-Clark,
Courte, & Howard, 2007; Cooper et al., 2003; Howland & Good,
2015; Malan & Leitner, 2007; Nikou & Economides, 2014;
Çagıltay, 2007). Hence, the main objective of the current study is
to analyze the effect of programming instruction with Scratch on
the motivation and programming achievements of pre-service in-
formation technologies teachers. Based on this objective, the
following research question has been investigated:
 Is there a significant difference between the group that learned
with Scratch and the group that learned with flowchart in
participant motivation and their academic achievements?
2. Methodology
The current study utilized the pretest-posttest experimental
model with a control group. The dependent variables of the study
were achievement in programming and motivation, while the in-
dependent variable was instructional method (Scratch/Flowchart).
The research design is presented in Table 1.
2.1. Study group
Participants of the study consisted of 52 sophomore students
taking Programming 1 course at the Department of Computer Ed-
ucation and Instructional Technologies of Mehmet Akif Ersoy Uni-
versity during the fall semester of the 2014e2015 academic year.
Participants were divided into test and control groups, each con-
sisting of 26 students. Convenience sampling method was applied
and participants were randomly assigned to the two groups. Equal
probability assignment of the participants to the test and control
groups was obtained through random selection. Participant de-
mographic characteristics (gender and type of high school gradu-
ated) for both groups are presented in Table 2.
Overall, 48.07% of the participants were female and 51.93% were
male. Analysis of the type of high school that participants gradu-
ated from demonstrated that 73.07% of participants (i.e. from both
groups) graduated from vocational or technical high schools, while
26.93% graduated from general high schools. In the vocational and
technical high schools, there are programming courses in the
computer science department, but not in the general high school.
Accordingly, having experience with programming could be an
advantage for graduates of vocational and technical high schools
participants. In addition, gender was an influential variable on
attitude and success in programming, especially girls were more
unsuccessful in programming (Goode, Estrella, & Margolis, 2006;
Kelleher, Pausch, & Kiesler, 2007). Therefore, while distributing
the participants into groups, attention is paid to balance the gender
and the type of high school graduated. After the placement, par-
ticipants were informed about which group (Scratch or Flowchart)
they were assigned to. However, participants did not know
whether their group was the experimental or the control group. In
addition, similar activities were applied by different methods in
each group before the groups were merged for their subsequent
programming language training. The same instructor was
employed throughout the whole process.
2.2. Data collection tools
The first tool the study utilized the Motivated Strategies for
Learning Questionnaire (MSLQ), as developed by Pintrich, Smith,
Garcia, and McKeachie (1993) and adapted to Turkish by
Büyüko €ztürk, Akgün, Kahveci, and Demirel (2004). The question-
naire includes two subscales; motivation and learning strategies.
There are 31 items and six sub-dimensions in the motivation sub-
scale, while there are 50 items and nine sub-dimensions in the
learning strategies subscale. These subscales can be applied sepa-
rately, and accordingly, for this current study only the motivation
subscale was employed. In the 7-point, Likert-type scale, each item
 O. Erol, A.A. Kurt / Computers in Human Behavior 77 (2017) 11e18 13

Table 1
Research design.

Groups Selection Pretest Process Posttest Process Posttest 2

Test R ATPre MSLQPre PIS ATPost MSLQPost CPLI ATPost2 MSLQPost2

Control R ATPre MSLQPre PIF ATPost MSLQPost CPLI ATPost2 MSLQPost2

R Random selection.
PIS Programming Instruction with SCRATCH.
PIF Programming Instruction with Flowcharts.
CPLI C# Programming Language Instruction.
ATPre Achievement Test e Pretest.
MSLQPre: Motivated Strategies for Learning Questionnaire e Pretest.
ATPost Achievement Test e Posttest.
MSLQPost Motivated Strategies for Learning Questionnaire e Posttest.
ATPost2 Achievement Test e Posttest 2.
MSLQPost2 Motivated Strategies for Learning Questionnaire e Posttest 2.

Table 2
Participant demographic characteristics.

Demographic Group (n ¼ 52) Test Group (n ¼ 26) Control Group (n ¼ 26)

Gender Female f 14 11
% 53.84 42.31
Male f 12 15
% 46.16 57.69
Graduated High School Type Vocational/Technical High School f 18 20
% 69.23 76.92
General High School f 8 6
% 30.77 23.08

is scored using values ranging between “1 e Absolutely not true for
me” to “7 e Absolutely true for me”. In the Turkish adaptation,
internal consistency coefficients ranged between 0.52 and 0.86.
The other data collection tool was an achievement test devel-
oped by the authors to measure the academic achievements of the
students on programming. This achievement test included two
sections with 20 multiple-choice questions on basic programming
structures (variable, decision making and control, loop, string) and
four open-ended questions related to programming problems. In
order to design the part of the achievement test that included
multiple-choice questions, a pilot test containing 25 items was
formed with expert opinion. These experts were instructors who
teach programming at Mehmet Akif Ersoy University. As a reli-
ability study, the pilot test was applied to 114 students of Pro-
gramming Languages I course in the Computer Education and
Instructional Technologies Department at Mehmet Akif University
Faculty of Education. Participants in the pilot study were educated
at the third and fourth grades in the same department and had
already taken programming courses before. These students were
not involved in the current study.
Item analysis was conducted after the pilot application and
discrimination (r) and difficulty (p) indices for each item were
calculated. Due to the fact that discrimination index scores of five
items were below 0.20, these items were excluded from the main
test. Six items with scores ranging between 0.20 and 0.30 in the
item discrimination index were then reorganized through expert
opinion. After the items with low discrimination index values were
excluded from the test, the mean discrimination index was calcu-
lated as r ¼ 0.44. Difficulty index values for test items demonstrated
that they varied between 0.23 and 0.79, resulting in a mean diffi-
culty index calculation of p ¼ 0.51. Thus, it could be argued that the
test was of intermediate difficulty. Furthermore, KR-20 internal
consistency coefficient was calculated to measure the reliability of
the test as 0.712. Test-retest method was used to determine the
consistency of the achievement test. The developed achievement
test was applied twice to 61 junior students attending the Com-
puter Education and Instructional Technologies Department of
Mehmet Akif University, with a two-week duration between ap-
plications. The correlation between the two measurements was
calculated with Pearson Product-Moment Correlation Coefficient as
0.723. The multiple-choice test included 11 algorithm problems
and nine C# problems. Algorithm problems had been used to test
programming logic. There were question simple programming
problems that did not involve coding and could be solved step-by-
step on paper. Many similar level algorithm problems had been
used both during Scratch training and flowchart training. Learning
outcomes related to the questions in the multiple-choice test are
presented in Table 3.
To design the second section of the achievement test that con-
tained the open-ended programming problems, an item pool was
created which included analysis and synthesis level questions. Four
problems were selected based on the feedback obtained from ex-
perts and were reorganized to fit the requirement. Interrater reli-
ability was tested using Kendall W fitness test to establish scoring
reliability of open-ended programming problems (W ¼ 0.901,
p < .01).
2.3. Procedure
The study conducted within a Programming Languages I class
lasted for a total of 14 weeks, including the collection process of the
tests. The objective was to teach basic programming structures
during the initial seven weeks of the study. In the control group,
basic programming structures instruction was conducted as plan-
ned in the current curriculum using flowcharts. Thus, the basic
structures were explained through narration and presentation
methods and programming problems were solved using flow-
charts. In the test group, this process was utilized by conducting
with Scratch software. Participants used the characters and
graphics in Scratch to solve these problems in a game scenario. For
example, a problem on collecting three numbers entered from the
keyboard, control group solved the problem by using flow dia-
grams, whereas the test group solved by designing it as a Scratch
game (Fig. 1). Similar problems and examples were used for both
14 O. Erol, A.A. Kurt / Computers in Human Behavior 77 (2017) 11e18

Table 3
Learning outcomes related to problems.

Learning Outcomes Algorithm Problems C# Problems

Variable (Assignment, Basic operations) Q3-Q6-Q14-Q19 Q1-Q11

Loop (For, while etc.) Q2-Q3-Q5-Q7-Q10-Q16 Q8-Q18-Q20
Decision making (If else etc.) Q12-Q6-Q17-Q19 Q9-Q11-Q15
Arrays Q5-Q7 Q4-Q8

Control Group Test Group

Fig. 1. The solution of same problem in both groups.

groups and in both groups they wrote problem-solving algorithms
(solution steps) step by step.
In the second seven-week period of the study, C# programming
language instruction was conducted in both the control and test
groups. In both groups, instruction by presentation, demonstration,
exercise techniques, and similar problems and examples were
employed. In other words, the same methodology was utilized in
both groups during their C# programming language instruction.
The content and activities related to the course for both groups are
presented in Table 4.
2.4. Data analysis
In the analysis of data, in cases where more than one
comparison is needed, Bonferroni correction was implemented and
Type-I error probability was minimized. Furthermore, to determine
the magnitude of the difference, h2 effect size value was examined.
The Eta square correlation coefficient is interpreted as 0.01, 0.06
and 0.14, respectively, as small, medium and large effect sizes
(Cohen, 1988). In cases where there was no difference, statistical
power was reported.
To determine whether or not there was a significant difference,
between the group that learned with Scratch and the group where
instruction was conducted based on the existing curriculum, in
increasing the participant motivation and academic achievements
with learning programming languages, a 3
2 mixed design
MANOVA was employed. In this analysis, two different instruction
methods (learning with Scratch, and problem solving with

Table 4
Course content of test and control groups.

Week Control Group Test Group

Weeks 1 & 2  Flowcharts & Symbols  Introduction to Scratch

 Variables  Variables
 Solving related problems with flowcharts  Solving related problems with Scratch
Week 3  Control & Decision Making  Control & Decision Making
 Solving related problems with flowcharts  Solving related problems with Scratch
Weeks 4 & 5  Loops  Loops
 Solving related problems with flowcharts  Solving related problems with Scratch
Weeks 6 & 7  Arrays  Arrays
 Solving related problems with flowcharts  Solving related problems with Scratch
Weeks 8-14 C#
 Introduction to C#
 Variables
 IfeElse/Switch Case Codes
 For/While/DoeWhile Codes
 Arrays
 Solving related problems in C# Editor
 O. Erol, A.A. Kurt / Computers in Human Behavior 77 (2017) 11e18
flowcharts) reflect the in-group factor, while measurement times of
pretest, posttest and posttest2 applications of the measurement
tools (achievement and motivation) were the recurring measure-
ment factor. At the end of the analysis, simple main effect analysis
was conducted in order to determine whether or not the differ-
ences observed between achievement and motivation scores based
on the group and measurement times were statistically significant.
Thus, while both achievement and motivation scores were
compared within themselves separately based on the time of
measurement, dependent samples t-test was conducted for each
time of measurement and after Bonferroni adjustment, significance
level was divided by the number of tests. To compare the
achievement and motivation scores based on the groups, inde-
pendent samples t-test was conducted separately for each group
and following Bonferroni adjustment significance level was divided
by the number of tests.
3. Results
MANOVA test results were used to determine whether or not
there was a significant difference between the group that learned
with Scratch and the group where instruction was conducted based
on problem solving with flowcharts in increasing the motivation
and academic achievements of the participants on learning pro-
gramming languages. These results are presented in Table 5.
As can be seen in Table 5, there was a significant difference
between the participants' motivation and achievement scores
based on the group variable (Wilks' L ¼ 0.800, F (2,49) ¼ 6.106,
p < .025). Accordingly, it could be stated that the type of instruction
method applied in both the test and control groups had an effect on
motivation and achievement scores of the participants. Effect size
value was calculated as .200. Thus, it is possible to talk about a large
effect (Cohen, 1988). It was observed that there was a significant
difference between the participants' motivation and achievement
scores based on the time of measurement variable (Wilks'
L ¼ 0.172, F (4,47) ¼ 56.434, p < .001). Accordingly, it could be
stated that motivation and achievement scores of the participants
differed based on pretest, posttest and posttest2 measurements.
Effect size value was calculated as .828. Thus, it is possible to talk
about a large effect (Cohen, 1988). Another finding was that moti-
vation and achievement scores of the participants showed a sig-
nificant difference based on the interaction effect of time of
measurement and group variables (time of measurement * group)
(Wilks' L ¼ 0.712, F (4,47) ¼ 4.755, p < .025). Accordingly, it could
be stated that the motivation and achievement scores of the par-
ticipants received in pretest, posttest and posttest2 differed based
on the group variable; in other words, the instruction method used.
The finding that time of measurement and group (time of mea-
surement * group) interaction created a difference in the motiva-
tion and achievement scores of the participants was important.
Thus, motivation and programming achievement scores were
examined within the context of the interaction of time of mea-
surement and group variables, instead of running a single analysis
for each variable. This was important for a healthier interpretation
of the findings (Huck, 2000). Effect size value was calculated as
.288. Thus, it is possible to talk about a large effect (Cohen, 1988). To
Table 5
Multivariate analysis of variance (MANOVA) results for motivation and programming achievement mean scores based on time of measurement and group variables.
Source of Variance Wilks' L F Hypothesis df
Group .800 6.106
Time of Measurement .172 56.434
Time of Measurement* Group .712 4.755
15
examine the interaction of time of measurement and group vari-
ables, motivation and programming achievement variables were
individually addressed in a 3
2 mixed design ANOVA. Analysis of
variance results for the mean motivation score of the participants
based on time of measurement and group variables are presented
in Table 6.
As can be observed in Table 6, there was a significant difference
between the participants' motivation scores based on the group
variable (F (1,50) ¼ 6.320, p < .025, h2 ¼ 0.112). It could be argued
that the motivation scores of the participants differed based on the
group variable; in other words, the type of instruction method
applied. Furthermore, the effect size value (h2 ¼ 0.112) demon-
strated a large effect (Cohen, 1988). However, there was no signif-
icant difference found between the motivation scores based on the
time of measurement variable (F (1.750, 87.494) ¼ 160.920,
p ¼ 0.437, power ¼ 0.174). There was a significant difference based
on the interaction effect of time of measurement and group vari-
ables between the motivation scores (F (1.750, 87.494) ¼ 1584.052,
p < .01, h2 ¼ 0.136). Thus, it could be stated that the control and test
group motivation scores differed based on the time of measure-
ment. Analysis of variance results for programming achievements
of the participants based on time of measurement and group var-
iables are presented in Table 7.
According to Table 7, there was a significant difference between
the participants' achievement scores based on the group variable (F
(1,50) ¼ 11.021, p < .025, h2 ¼ 0.181). Thus, it could be argued that
the achievement scores of the participants differed based on the
group variable; in other words, the type of instruction method
applied. Furthermore, the effect size value (h2 ¼ 0.181) demon-
strated a large effect (Cohen, 1988). Based on the time of mea-
surement, it was observed that there was a significant difference
between the achievement scores (F (1.232, 61.624) ¼ 181.521,
p < .025, h2 ¼ 0.784). Accordingly, it could be argued that the
achievement scores of the participants differed based on pretest,
posttest and posttest2 measurements. A large effect could be
determined from the effect size value (h2 ¼ 0.784) (Cohen, 1988).
There was a significant difference based on the interaction effect of
time of measurement and group variables between the achieve-
ment scores (F (1.232, 61.624) ¼ 8.339, p < .025, h2 ¼ 0.143). Thus, it
could be stated that control and test group achievement scores
differed based on the time of measurement. To analyze the inter-
action effect in detail, test and control group achievement and
motivation scores based on the time of measurement are presented
in Fig. 2.
Conducted analyses of variance demonstrated that the time of
measurement and group variable interaction had a significant ef-
fect on motivation and achievement. However, as can be observed
from Fig. 2, this interaction had different effects on motivation and
achievement scores of the participants. While the motivation score
continuously increased in the test group, it initially decreased in the
control group, and then became stable. On the other hand,
achievement score reflected a continuous increase in both the test
and control groups. Simple main effect analysis was conducted in
order to determine whether or not these observed cases were
statistically significant.
When achievement scores within the groups were compared
Error df p h2
2.000 49.000 .004 .200
4.000 47.000 <.001 .828
4.000 47.000 .003 .288
16 O. Erol, A.A. Kurt / Computers in Human Behavior 77 (2017) 11e18

Table 6
Analysis of variance (ANOVA) results for the mean motivation score of participants based on time of measurement and group variables.

Source of the variance Sum of Squares df Mean Square F p h2

Between-groups
Group 6346.314 1 6346.314 6.320 .015 .112
Error 50,204.526 50 1004.091
Within groups
Time of measurement 281.590 1.750 160.920 .802 .437 .016
Time of measurement *Group 2771.897 1.750 1584.052 7.897 .001 .136
Error 17,550.513 87.494 200.591
Total 77,154.84 141.994

Table 7
Analysis of variance (ANOVA) results for programming achievements of participants based on time of measurement and group variables.

Source of the variance Sum of Squares df Mean Square F p h2

Between-groups
Group 3689.130 1 3689.130 11.021 .002 .181
Error 16,737.162 50 334.743
Within groups
Time of measurement 46,290.716 1.232 37,558.903 181.521 .001 .784
Time of measurement *Group 2126.452 1.232 1725.340 8.339 .003 .143
Error 12,750.760 61.624 206.912
Total 81,594.22 115.088

difference between the groups continuously increased, favoring the

test group. As a result, it could be stated that the instruction
methods used in both the control and test groups increased
achievement scores. However, since the difference between the
groups favored the test group, the instructional method imple-
mented in the test group was more effective in increasing
achievement. Furthermore, the reason the increase in achievement
score increased within the measurement interval where a common
instructional method was utilized (posttest e posttest2 measure-
ment interval) could be due to the instructional methods used in
the control and test groups. Thus, based on the significant differ-
ence in scores obtained at the posttest2 measurement favoring the
test group, it could be argued that the method used in the test
Fig. 2. Group achievement & motivation scores based on time of measurement.
group (programming instruction with Scratch) affected the
achievement more in the measurement interval where a common
instructional method was used for programming (posttest e
posttest2 measurement interval).
based on time of measurement (Appendix A), in the control group,
When motivation scores within the groups were compared
there was a significant difference between pretest and posttest
based on time of measurement (Appendix C), in the control group,
favoring posttest (t (25) ¼ 7.427, p < .001); posttest and posttest2
there was no significant difference between pretest and posttest (t
favoring posttest2 (t (25) ¼ 7.079, p < .001), and pretest and
(25) ¼ 2.563, p ¼ 0.017); posttest and posttest2 (t (25) ¼ .331,
posttest2 favoring posttest2 (t (25) ¼ 8.634, p < .001). In the test
p ¼ 0.743), and pretest and posttest2 (t (25) ¼ 2.034, p ¼ 0.053). In
group, there was a significant difference between pretest and
the test group, there was also no significant difference between
posttest favoring posttest (t (25) ¼ 12.054, p < .0001), posttest
pretest and posttest (t (25) ¼ 1.380, p ¼ 0.018), posttest and
and posttest2 favoring posttest2 (t (25) ¼ 8.695, p < .0001), and
posttest2 (t (25) ¼ 2.074, p ¼ 0.049), and pretest and posttest2 (t
pretest and posttest2 favoring posttest2 (t (25) ¼ 11.756,
(25) ¼ 2.943, p ¼ 0.007). However, this could be due to type-II
p < .0001). Accordingly, it could be observed that the achievement
error that could occur as a result of Bonferroni adjustment. In
scores significantly increased from pretest towards posttest2
fact, it is possible that a significant difference was not observed due
continuously in both the test and the control groups. Achievement
to the narrowing of the level of significance, while the comparison
scores were compared based on the groups in order to determine in
could have revealed significant differences. Analysis of mean values
which group this increase was significantly higher (between-
demonstrate that there was a decrease of 6.08 points between the
groups comparison). Analysis of the differences between groups
pretest and posttest measurements, and there was an increase of
(Appendix B) showed that the groups had similar achievement
1.04 points between posttest and posttest2 measurements in the
scores in pretest (t (50) ¼ .589, p ¼ 0.558). In the posttest, the test
control group, while there was a decrease of 7.88 points between
group was 9.191 points significantly higher (t (50) ¼ 3.666,
pretest and posttest2 measurements. Thus, it could be stated that
p < .001) than the control group, and in posttest2, the test group
participants' motivation scores decreased at posttest2 compared to
was 19.025 points significantly higher (t (50) ¼ 3.153, p < .008)
the pretest scores in the control group and there was a decrease
than the control group. Thus, it could be deduced that the
from the first measurement to the third. On the other hand, in the
 O. Erol, A.A. Kurt / Computers in Human Behavior 77 (2017) 11e18 17

test group, it was observed that participants' motivation scores comprehensible and entertaining. Scratch helps to visualize the
increased from the first measurement (pretest) to the second algorithm process. The visual structure of Scratch could make
measurement (posttest), and from the second measurement programming structure more concrete and the interesting which is
(posttest) to the third measurement (posttest2). Furthermore, it generally considered to be difficult and boring, thereby increasing
was also observed that motivation scores increased 11.81 points learner motivation and enabling participants to learn programming
between the first (pretest) and third (posttest2) measurements. better (Gee, 2005; Moreno, 2012). Learners' managing all the
Thus, it could be stated that participants' motivation scores design processes (determining problem, design, test etc.) helps
increased at the end of the process in the test group. To see in which identifying this process a more constructive approach. On the
group these differences were significant, scores obtained in the contrary, it could be stated that problem-solving activities with
measurements were compared based on groups (between-group flowcharts found in the current curriculum decreased the motiva-
comparison). Analysis of the differences between groups (Appendix tion of learners for programming, and that this effect continued
D) showed that the groups had similar motivation scores in the during their C# programming language instruction. Thus, it is
pretest (t (50) ¼ .205, p ¼ 0.838), in the posttest, the test group possible that low motivation made it difficult for the participants to
was 16.65 points significantly higher (t (50) ¼ 2.651, p < .008) and acquire programming logic, which in turn affected their program-
in posttest2, the test group was 20.81 points significantly higher (t ming achievements to a lesser degree. Learners who are bored with
(50) ¼ 3.434, p < .001) than the control group. Thus, it could be the flowcharting course and have lowered motivation, thus expe-
deduced that the difference between the groups continuously and rience problems in comprehending programming logic. Accord-
significantly increased in favor of the test group. Furthermore, it ingly, it is seen that activities with Scratch increases the motivation
could be argued that the method used in the test group (pro- of students, whilst flow diagrams decrease motivation.
gramming instruction with Scratch) continuously increased moti- Based on study results, it could be argued that learners were
vation of the participants; whilst the method used in the control able to learn programming logic as a result of the experiences ob-
group (programming instruction with flowcharts) decreased tained during Scratch process, and transfer this experience to the
motivation of the participants, and the effect on motivation learning of a new programming language. On the contrary, it was
continued in the test group. However, motivation stagnated in the observed that the limitations of problem-solving activities using
control group in the interval where the same method was used in flowcharts, as found in the current curriculum, such as their level of
both groups (posttest e posttest2 measurement interval). difficulty and vapidity, insufficiency in materializing the process
and learners remaining passive in the process decreased the
4. Discussion & conclusion motivation of learners and this factor negatively affected the pro-
cess of learning programming logic. According to the results of the
Study findings revealed that participants' programming research, the following could be suggested.
achievements showed an increase in both the test and control
groups; however, this increase significantly differentiated in favor Activities could be conducted in the introduction to program-
of the test group. Furthermore, it was observed that the difference ming courses using visual tools such as Scratch or Alice instead
continued in the third measurement, the posttest2 results, also of flowcharts to ensure the achievement of students under-
favoring the test group. The fact that the difference favoring the test standing programming logic.
group continued after the posttest2 measurement, could be inter- Learner-oriented learning environments could be designed in
preted as the participants having transferred the skills they ob- both programming logic and programming language instruction
tained during the process of Scratch to the process of programming to increase the leaner achievements and motivation.
with C#. This finding is supported by the results of similar studies Further studies could be conducted that would utilize different
found in the literature (e.g., Leiva & Salas, 2013; Rizvi et al., 2011; visual tools other than Scratch activities and compare those to
Westcott, 2008; Çag ıltay, 2007). Another finding of the current flowcharts.
study was that motivation of the participants showed differences in Similar empirical studies could be conducted to research the
both posttest and posttest2 measurements, favoring the test group. effect of activities using Scratch on programming achievements
In other words, it was demonstrated that activities conducted with and motivation at different educational levels (high school,
Scratch implemented in the test group was effective in increasing middle school, primary school).
the motivation of the participants. Furthermore, it was observed
that the increase in motivation in the test group continued during
Appendix A. Achievement within-group comparison
the programming with C# instruction process. Another significant
finding was the fact that at the end of the process, motivation of the
participants in the test group increased, and decreased in the
control group. This finding was parallel to related studies found in
the literature (Nikou & Economides, 2014; Osman, Loke, Zakaria &
Downe., 2012). However, in this current study, contrary to the Groups Tests n Sd df t pa
X
findings of other studies, it was found that problem-solving activ-
ities using flowcharting, which is preferred by several educators Control Group Pretest 26 6.62 5.507 25 7.427 .000
Posttest 26 20.32 8.709
especially in programming logic instruction, decreased the moti- Posttest 26 20.32 8.709 25 7.079 .000
vation of the participants (Leiva & Salas, 2013; Rizvi et al., 2011). Posttest2 26 39.62 20.413
As a result of the analysis of all study data, it could be stated Pretest 26 6.62 5.507 25 8.634 .000
that; it might be possible that high motivation during the process Posttest2 26 39.62 20.413
Test Group Pretest 26 7.58 6.236 25 12.054 .000
where programming logic was taught could have helped the par-
Posttest 26 29.51 9.357
ticipants to comprehend the programming logic better and could Posttest 26 29.51 9.357 25 8.695 .000
have also increased the students' motivation and programming Posttest2 26 58.64 23.027
achievements during their programming language instruction Pretest 26 7.58 6.236 25 11.756 .000
process. The reason for this could be Scratch making the algorithm Posttest2 26 58.64 23.027
a
process where programming logic was instructed more Bonferroni adjustment.025/6 ¼ 0.004.
18 O. Erol, A.A. Kurt / Computers in Human Behavior 77 (2017) 11e18

Appendix B. Achievement between-groups comparison Büyüko €ztürk, Ş., Akgün, O. € E., Kahveci, O.,€ & Demirel, F. (2004). Güdülenme ve
€g
o renme stratejileri o €lçeginin Türkçe formunun geçerlik ve güvenirlik çalışması.
Kuram ve Uygulamada Eg itim Bilimleri, 4(2), 207e239.
ıltay, N. E. (2007). Teaching software engineering by means of computer-game
Çag
development: Challenges and opportunities. British Journal of Educational
Technology, 38(3), 405e415.
Tests Groups n X Sd df t Pa Cohen, J. W. (1988). Statistical power analysis for the behavioral sciences (2nd ed.).
Hillsdale, NJ: Lawrence Erlbaum Associates.
Pretest Control 26 6.62 5.507 50 .589 .558 Cooper, S., Dann, W., & Pausch, R. (2003). Using animated 3D graphics to prepare
Test 26 7.58 6.236 novices for CSI. Computer Science Education, 13, 3e30.
Posttest Control 26 20.32 8.709 50 3.666 .001 Cooper, S., Powers, K., McNally, M., Goldman, K. J., Proulx, V., & Carlisle, M. (2006).
Test 26 29.51 9.357 Tools for teaching introductory programming: What works? ACM SIGCSE
Posttest2 Control 26 39.62 20.413 50 3.153 .003 Bulletin, 38, 560e561.
Test 26 58.64 23.027 Gee, J. P. (2005). Learning by design: Good video games as learning machines.
ELearning, 2, 5e16.
a
Bonferroni adjustment .025/3 ¼ 0.008. Goode, J., Estrella, R., & Margolis, J. (2006). Lost in translation: Gender and high
school computer science. In J. M. Cohoon, & W. Aspray (Eds.), Women in IT:
Reasons on the underrepresentation (pp. 89e114). Cambridge, MA: The MIT
Press.
Harel, I., & Papert, S. (1991). Software design as a learning environment. Interactive
Learning Environments, 1(1), 1e30.
Appendix C. Motivation within-group comparison
Howland, K., & Good, J. (2015). Learning to communicate computationally with flip:
A bi-modal programming language for game creation. Computers & Education,
80, 224e240.
Huck, S. W. (2000). Reading statistics and research (3rd ed.). New York: Addison
Wesley Longman.
Kafai, Y. B. (2006). Playing and making games for learning instructionist and
Groups Tests n X Sd df t pa constructionist perspectives for game studies. Games and Culture, 1(1), 36e40.
Kelleher, C., Pausch, R., & Kiesler, S. (2007). Storytelling Alice motivates middle
Control Group Pretest 26 151.92 22.509 25 2.563 .017 school girls to learn computer programming. In CHI 2007 proceedings of the
Posttest 26 143.00 22.293 SIGCHI conference on human factors in computing systems (pp. 1455e1464).
Posttest 26 143.00 22.293 25 .331 .743 Kinnunen, P., & Malmi, L. (2008). CS minors in a CS1 course. In Proceedings of the
Posttest2 26 144.04 23.787 fourth international workshop on computing education research, ICER (pp. 79e90)
Pretest 26 151.92 22.509 25 2.034 .053 (New York, USA).
Kurland, D. M., Pea, R. D., Clement, C., & Mawby, R. (1986). A study of the devel-
Posttest2 26 144.04 23.787
opment of programming ability and thinking skills in high school students. In
Test Group Pretest 26 153.04 16.153 25 1.380 .018
E. Soloway, & J. C. Spohrer (Eds.), Studying the novice programmer (pp. 83e112).
Posttest 26 159.35 22.165
Hillsdale. NJ: Lawrence Erlbaum.
Posttest 26 159.35 22.165 25 2.074 .049 Lahtinen, E., Ala-Mutka, K., & Ja €rvinen, H. (2005). A study of the difficulties of novice
Posttest2 26 164.85 19.713 programmers. ACM SIGCSE Bulletin, 37(3), 14e18.
Pretest 26 153.04 16.153 25 2.943 .007 Leiva, F. A. J., & Salas, C. A. C. (2013). Practices of advanced programming: Tradition
Posttest2 26 164.85 19.713 versus innovation. Computer Applications in Engineering Education, 21(2),
a 237e244.
Bonferroni correction .025/6 ¼ 0.004.
Malan, D. J., & Leitner, H. H. (2007). Scratch for budding computer scientists. SIGCSE
Bulletin, 39(1), 223e227.
Maloney, J., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010). The scratch
Appendix D. Motivation between-groups comparison programming language and environment. ACM Transactions on Computing Ed-
ucation (TOCE), 10(4), 16.
Mishra, P., & Girod, M. (2006). Designing learning through learning to design. The
High School Journal, 90(1), 44e51.
Moreno, J. (2012). Digital competition game to improve programming skills.
Educational Technology & Society, 15(3), 288e297.
Tests Groups n X Sd df t Pa Nikou, S. A., & Economides, A. A. (2014). Transition in student motivation during a
scratch and an app inventor course. In Proceedings of global engineering edu-
Pretest Control 26 151.92 22.509 50 .205 .838 cation conference (EDUCON) 2014 (pp. 1042e1045). Istanbul, Turkey: IEEE.
Test 26 153.04 16.153 Osman, M. A., Loke, S. P., Zakaria, M. N., & Downe, A. G. (2012). Secondary students'
Posttest Control 26 143.00 22.293 50 2.651 .007 perfectionism and their response to different programming learning tools. In
Test 26 159.35 22.165 Proceedings of humanities, science and engineering (CHUSER) (pp. 584e588). IEEE
Posttest2 Control 26 144.04 23.787 50 3.434 .001 (Sabah, Malaysia).
Test 26 164.85 19.713 Papert, S. (1980). Mindstorms: Children, computers and powerful ideas. New York:
Basic Books.
a
Bonferroni adjustment .025/3 ¼ 0.008. Papert, S. (1991). Situating constructionism. In S. Papert, & I. Harel (Eds.), Con-
structionism (pp. 1e11). Norwood, NJ: Ablex Publishing Corporation.
Pintrich, P. R., Smith, D., Garcia, T., & McKeachie, W. J. (1993). Reliability and pre-
dictive validity of the motivated strategies for learning questionnaire (MSLQ).
Washington, DC: Office of Educational Research and Improvement.
References Resnick, M., Maloney, J., Monroy-Herna ndez, A., Rusk, N., Eastmond, E., Brennan, K.,
et al. (2009). Scratch: Programming for all. Communications of the ACM, 52(11),
Anastasiadou, S. D., & Karakos, A. S. (2011). The beliefs of electrical and computer 60e67.
engineering students' regarding computer programming. The International Rizvi, M., Humphries, T., Major, D., Jones, M., & Lauzun, H. (2011). A CS0 course using
Journal of Technology, Knowledge and Society, 7(1), 37e51. scratch. Journal of Computing Sciences in Colleges, 26(3), 19e27.
Baldwin, L. P., & Kuljis, J. (2000). Visualization techniques for learning and teaching Westcott, S. (2008). Effectiveness of using digital game playing in a first-level pro-
programming. Journal of Computing and Information Technology, 8(4), 285e291. gramming course (Unpublished Doctoral Dissertation). USA: Pace University.
Bishop-Clark, C., Courte, J., & Howard, E. V. (2007). A quantitative and qualitative Wolz, U., Leitner, H. H., Malan, D. J., & Maloney, J. (2009). Starting with scratch in
investigation of using Alice programming to improve confidence, enjoyment CS1. SIGCSE Bulletin, 41, 2e3.
and achievement among non-majors. Journal of Educational Computing
Research, 37(2), 193e207.