Second Language Research Methods by Seliger and Shohamy

1
What is experimental research?
An attempt by the researcher to maintain control over all factors that may
affect the result of an experiment.

What is the role of the researcher?

In doing this, the researcher attempts to determine or predict what may occur.

2
 Experimental Research and
Research Parameters
1- Synthetic/ Analytic/
Holistic Constituent

2- Heuristic A Priori

3- Level of control and

manipulation: Low High

4- Level of explicitness of
data collection: Low High
EXPERIMENTAL RESEARCH DESIGN

According to the research parameters, experimental

research is analytic and deductive.
3
 The major components of
experimental research
All experimental approaches involve the control
or manipulation of three basic components:

the population
the treatment
the measurement of the treatment

4
 The Type and Number of
Groups

Experimental research is concerned with studying

the effects of specified and controlled treatments

given to subjects usually formed into groups.

Groups can be formed according to the purpose of

the experiment or they can be natural as they

appear prior to the research. (e.g. Spanish
speakers, males, students in 3-A)

5
It is up to researcher to use pre-existing groups or

to set up a new group for the experiment.

To use already existing group or the one which is

formed for the experiment is one of the

distinguishing points between experimental
designs and quasi-experimental designs.

6
 The Treatment

This refers to anything done to groups in order to

measure its effect.

Treatments are the independent variable in research.

The treatment is a controlled and intentioned

experience. (e.g. a language teaching method
constructed for the experiment)

7
 Measurement or Observation
 It is the evaluation or observation of the effects of the treatment.

In second language research, the effects of the

treatment(s) can be evaluated by

a language test

a judgment or oral test

a communicative task and etc.

8
components of the
experimental research
 X stands for an experimental treatment.

e.g. a teaching method

materials used in a language lab

 O stands for observation or measurement of the

effects of the treatment.

 R stands for randomization,

randomization or the random
assignment of subjects to groups in order to control
for extraneous variables.
9
Experimental Research Designs
One-Shot Design
Single Group Design One Group pre-test+ post-test
Time-Sampling Designs

Static Group or pre-experimental

design
Control Group Design


Pre-Post test with randomization of groups

Factorial Design

Quasi-Experimental Design

Separate Sample Design

10
11
 Single Group Designs

1. One-Shot Design: X O (Pilot Study Design)

 A single treatment is given to a single group or

individual, then they are observed or measured.

 It is more appropriate to pilot studies in which the

researcher wishes to try out instruments or

treatments before running a full-scale experiment.

12
 Single Group Designs
Example of One-Shot Design:

A second language teacher uses a new method for

teaching the communicative use of the target
language. After three weeks of using this method,
a test is administreted to the class. The class
performs well on the test.

X : the new method for communicative use of TL

O : the test applied to the class

13
 Single Group Designs
Advantages and Disadvantages of One-Shot
Designs

 The disadvantage is that there is no way of knowing the

characteristics of the group or the individual before the

treatment.

 The advantage is that it correctly identifies what to avoid

in experimental research. Another one is that it is used for

first testing of instruments of experimental procedures.

14
 Single Group Designs

2- One Group pre-test + post-test: O1 X O2

 Subjects are observed or measured twice on the dependent

variable. This design eliminates the need for a control group
design as it uses the subjects as their own controls.

15
 Single Group Designs
Example of One Group pre-test + post-test Design

In a study on the effects of second language instruction,

the researcher wishes to establish that the group has no
knowledge of a grammatical structure which will be
taught in the lesson. In order to establish this, two tests
are constructed which specifically test for the target
structure in a discrete point test. One test will be used as
the pre-test and the other will become the post-test.

16
 Single Group Designs
Advantages and Disadvantages of
One Group Pre-test & Post-test Designs

 The disadvantage is that there is no certainty that the

possible differences that appear in O2 are the result of

treatment. The result may be affected by attrition or
maturation.

 The other disadvantage is that the pre-test may sensitize

the subjects to specific aspects of the treatment and thus

confound what is measured by the post-test.

17
 Single Group Designs
Advantages and Disadvantages of One Group Pre-test
&Post-test Designs

 This design is useful as it controls a number of extraneous variables

which can affect the homogeneity of subjects when more than one
group is involved.

 As the same group is used for both pre and post tests, there is no

need to match to another group.

18
 Single Group Designs
3- Time Sampling Designs: O1, O2, O3, On…X,
On+1
 In this design, a number of samples or observations
are taken over a period of time. That’s why, in
other words, this design refers to “time-series”.
 The treatment is controlled, added after a number
of observations or measurements.
 This design can be used to avoid the problems of
One Group pre-test+post test.

19
Example of Time-Sampling Design

A study wishes to investigate the effectiveness of

teaching students the use of relative clauses in English.
For the purpose of the research, a treatment is
conducted consisted of a lesson which gives the formal
features of relative clause formation in English along
with practice in using relative clauses in speech and
writing. The treatment is preceded by several in-class
writing assignments spaced over several weeks. For
each writing assignment the number of relative clauses
in learners’ writing is tallied and categorized. After the
treatment, several similar writing assignments are
given and the number of relative clauses are counted
and categorized.

20
 Single Group Designs
Advantages and Disadvantages of Time-Sampling Design
This design let us ascribe any changes in the subjects’ performance to
the treatment with greater assurance.

It discounts the effect of history as in this design the researcher

develops a norm for the population.

In this design, there is no control group, therefore, the results are
obtained from one group.

21
 Single Group Designs

To sum up, collecting data over an extended

period before and after the treatment allows us to

establish a normal pattern of performance for the
language or acquisition behavior under
investigation. This allows the researcher to
exclude the possible interaction of incidental
exposure to language material outside the
classroom, or any natural development change
which may take place regardless of instruction.
22
23
 Control Group Designs
In single group designs, the experimental group acts as it
own control; the comparison is between its performance with
treatment and without treatment.

In designs using control group, there are two or more groups
and the comparison is between them.

The multi-group designs are concerned with measures to

standardize the groups being compared so that maintain
external and internal validity.

24
 Control Group Designs
1- Static Group or “pre-experimental” Design X O1

O1

 In this design, a treatment is developed for a group and its

performance is compared with equivalent group, which
does not receive any treatment.

 But, there is no clue if the groups are really equal to each

other before the treatment.

25
 Control Group Designs

The difference in performance on the dependent

variable may be due to intrinsic group differences
such as:
 the first language background

 sex

 exposure to second language

 time of the day which instruction takes place

 the level of motivation of groups

 the effects of different teachers on groups

26
 Control Group Designs

There is no need to form groups which is different

from the previous one, to reassign the teachers

again or to rearrange the timetables.

These features can be considered as advantage,

but, if the variables are not controlled, they can

cause invalidity.

27
 Control Group Designs
When using a static design, to avoid the problems related
to invalidity, the researcher can match subjects into two
groups for various characteristics such as placement test
scores, sex, first language and teacher rankings to make
the groups more comparable.

 Matching is a way of increasing the comparability of the

groups when subjects cannot be randomly assigned.

Group A: X O1
Group B: -- O1

28
 Control Group Designs
A Variation on the Control Group Matching
Designs

ESL cloze (o1) acceptability (02)

Film (X)
Comparison
The film is the treatment. ESL refers to experimental
subjects whileNS cloze
NS refers (01) subjects.
to control acceptability (02)the
Following
treatment, the groups take a modified cloze test in which
target concepts were left blank. It was followed by an
acceptability task in which the same test was presented
with the blanks filled in, some with acceptable cognates
and others with unacceptable ones. The acceptability
performance of ESL learners was compared with that of the
native speakers. The cloze test is the pre-test to establish
that the ESL learners did have the concepts with which to 29
 Control Group Designs
2- Pre-test/Post-test with Randomization of Groups

Group A’ : O1 X O2
Group A R
Group A’’ : O1 – O2

Randomization is a procedure with which to reduce the

amount of systematic error that might result from biases
in the assignment of subjects to groups.

30
 Control Group Designs
It also provides better control of variables that could
affect internal validity.

By randomizing, we are claiming that any effects of

extraneous variables occur by chance and that
chance is equally distributed between both groups.

An advantage of randomization is that it provides the

option of not using pre-test as a method of making
groups more comparable. When the subjects are
assigned to groups randomly, the effect of subject
variables is spread throughout the two groups.

31
 Control Group Designs
The larger the number of subjects in each group, the
more likely it is that the subject variables will negate
each other through the randomization process.

Internal Validity and Randomization

Volunteers are solicited for a study on the effects of

computer-assisted instruction on SL achievement. From
the population of volunteers, subjects are assigned to
computer-assisted group first and to the control group
receiving no computer-assisted instruction second.

32
 Control Group Designs
Although assigning of subjects to the treatment or

the control group would seem to be random, the

first volunteers may be different from those who
may have been cajoled into participating in the
study. The order of assignment may be a
reflection of motivation at some level and it may
affect the results of the study.

33
34
 Factorial Designs

Factorial designs are similar to the true experimental designs

and include all of the elements found in those designs, such
as grouping by randomization, pre and post-testing and
treatments.

The difference is that the effects of several independent

variables may be tested at the same time.

35
 Factorial Designs

In simple experimental designs, the additional variables are

referred to as extraneous variables. These variables must
be controlled if an effect is to be attributed to a specific
independent variable.

However, in factorial designs, more than one variable may

be treated as an independent variable.

36
 Factorial Designs
Example:

In a study to test for the effect of language

laboratory training on pronunciation, it is also
decided to measure for the effect of such training
on learners of different proficiency levels, as well
as the effect of different kinds of language
laboratory practice.

37
 Factorial Designs

The effect of practice vs. no practice in language lab. can be a

function of both the proficiency level of the learner and the
type of practice administered in the laboratory.

38
 Factorial Designs
Proficiency Level Factor: Advanced learners will
focus on their pronunciation performance as they are
aware of their problems. Beginners are not aware of
their pronunciation problems as they are dealing with
more basic issues in language learning.
Language Laboratory Factor: The type of practice
in the laboratory is another factor. The practice can
be contextualized -practice in communicative
settings- or it can be decontextualized –practice in
little or no communicative settings.

39
 Factorial Designs
Gb1 O1 Xd O2
R Gb2 O1 Xc O2
Gb3 O1 - O2 No laboratory practice
Ga1 O1 Xd O2
R Ga2 O1 Xc O2
Ga3 O1 - O2 No laboratory practice

pre-test post-test

Ga: Advanced Proficiency Xd: Decontextualized Practice

Gb: Beginning Proficiency Xc: Contextualized Practice

40
 Factorial Designs
Variations on Factorial Designs in Second
Language Research

A variation of the factorial design can be found in

Flynn (1984) in a study of the ability of Japanese and
Spanish speakers to correctly imitate sentences with
right or left branching adverbial phrases. Her
hypothesis was that the primary branching direction
of the first language would affect the ability of the
learner to imitate such sentences in English.

41
 Factorial Designs
In this study, subjects were not randomized as would
be the case in true experimental designs. Learners
were given a pre-test to establish proficiency level and
were then divided into three levels: beginning,
intermediate and advanced. They were further
grouped according to first language. There was no
experimental treatment, but an experimental task
was administered, in which learners were asked to
imitate a set of stimulus sentences.

42
 Factorial Designs
In this kind of studies, the treatment took place

before the experiment and consisted of the

subjects’ exposure to the second language. The
purpose of the pre-test in these studies is to
establish levels of ability before the task is
administered. In addition, there is no
randomization in assigning subjects to groups,
because the experimental groups consist of
natural first language groups.
43
44
 Quasi-experimental Designs
Quasi-experimental designs (Campbell and Stanley 1963)
are constructed from situations which already exist in the
real world, and are probably more representative of the
conditions found in educational contexts.

Quasi-experimental research is likely to have external

validity as it is conducted under conditions closer to those
normally found in educational contexts.

As this designs are less disruptive than others, it is easier

to gain access to subject populations and thus easier to
conduct such research.

45
 Quasi-experimental Designs

For these reasons, quasi-experimental designs are

ideal for teacher-conducted research and for pilot

studies, in which the exploration of a research
idea is the primary goal. But, the greatest problem
is controlling for sources of internal validity.

46
47
 Separate Sample Designs
Separate Sample pre-test/post-test Design
This design is useful for those situations in which
the researcher has access to only one group of
subjects at a time.

Example:
As in the previous example, the aim of the
study is to measure the effects of language
laboratory training on the pronunciation of
advanced level class entering every three weeks.

48
 Separate Sample Designs
In this design, the same experiment is replicated

each time with a different class, the population

characteristics are assumed to be the same.

Group 1: (Week 1) O1 X O2

Group 2: (Week 3) O3 X O4

Group 3: (Week 6) O5 X O6

49
 Separate Sample Designs
The Advantages of Using This Design
It allows for a larger population to be treated, and
overcomes the problem of lack of access to large
enough groups of subjects at any one time.
It allows us to control for changes due to history. The
other way to control for history using this design, is to
conduct inter-group comparisons between post and pre
tests.

50