Part one

Chapter one

Materials Evaluation
by BrianThomlinson

Presented by
Rustiyani
Materials evaluation is a procedure
that involves measuring value (or
potential value) of a set of learning
materials.
 Types of material evaluation:

There are different types of evaluations. They differ in purpose, in

personnel, in formality, and in timing:

• Pre-use evaluation
• Whilst-use evaluation
• Post-use evaluation:
 Pre-use evaluation
It involves making prediction about potential value of materials for their users.

 Context-free as in review of materials injournals

 Context-influenced  as in a review of draft material for publisher with target
users in mind
 Context-dependent  when a teacher selects course book for use with her
particular class.
Making use of checklist criteria has become popular in material evaluations
and certain checklists from the literature have been frequentlymade use of in
evaluations.
Tomlinson & Masuhara (2004: 7) proposed the following
questions for evaluating criteria:

a) Is each question an evaluation question?

b) Does each question only ask one question?
c) Is each question answerable?
d) Is each question free of dogma?
e)Is each question reliable in the sense that other evaluators
would interpret it in the same way?
 Whilst-use evaluation

In which the value of materials are measured while using them or

while observing them being used, is more objective and reliable than
pre-use evaluation, though it has its own limitations. It is purely
based on observation and so it measures solely what isobservable

It can observe short term memory through observing learners’

performance on exercises but it can not measure durable and effective
learning because of the delayed effect of instruction. It is therefore very
useful but dangerous too, as teachers and observers can be misled by
whether the activities seem to work or not.
o Clarity of instruction
o Clarity of layout
o Comprehensibility of text
o Credibility of task
o Achievability of tasks
o Achievement of performance objectives
o Potential for localization
o Practicality of the material
o Teachability of the material
o Flexibility of the material
o Appeal of the material
o Motivating power of the material
o Impact of the material
o Effectiveness in facilitating short term learning
 Post-use evaluation
Post-use evaluation measures the actual effects of the materials on the
users.

It provides data with which reliable decisions regarding the use, adaptation
or replacement of the materials can be made. So I feel that post-use
evaluation should be administered by the teachers to find out whether the
textbooks they use meet the learner and learning needs, and to propose
suggestions for additions and deletions to enhance the relevance and utility
of the textbooks for the targeted group of learners; that is, post-use
evaluation will help the teachers to use the English textbooks selectively and
will suggest to them the kind of supplementary materials that should be used
in an English class
 The main limitation of post-use evaluation is, it is time consuming and
requires expertise to measure post-use effects reliably. Still, it should be
executed as this painstaking effort on the part of the teachers will be
prove to be helpful to the users of the textbooks.
The different ways of measuring post-use effects of materials mentioned in
Tomlinson (2009) are:
 Tests of what has been ‘taught’ by the materials
 Tests of what the students can do
 Examinations
 Interviews
 Questionnaires
 Criterion-referenced evaluation by the users
 Post-course diaries
 Post-course ‘shadowing’ of the learners
Post-course reports on the learners by employers, subject tutors,etc.
 Developing criteria for materials evaluation

To develop a set for formal criteria for use on a particular evaluation and
then to use that set as a basis for developing subsequent context-specific
sets. Initially this is demanding and time consuming , but it not only helps
the evaluators to clarify their principles of language learning and teaching
but it also ensures the future evaluations (both formal and informal) are
systematic , rigorous and above all principled .one way of developing a set of
criteria is a follows.
 1. Brainstorm a list of universal criteria

• Universal criteria are those which would apply to any language learning
materials anywhere for any learners.
• Brainstorm is a very useful way of beginning an evaluation and the most
useful way that has been found of doing it is phrase the criteria as specific
questions rather than to list them as general headings.
 2. Subdivide some of the criteria

• If the evaluation is going to be used as a basis for revision or

adaptation of the materials, or if it is going to be a formal
evaluation and is going to inform important decisions, it is useful
to subdivide some of the criteria into more specific questions.
 3. Monitor and revise the list of universal criteria

• Is each question an evaluation question?

• Does each question only ask one question?
• Is each question answerable?
• Is each question free of dogma?
• Is each question reliable in the sense that other evaluators would interpret
in the same way?
 4. Categorize the list
• It is very useful to rearrange the random list of universal criteria into
categories which facilitate focus and enable generalizations to be. An extra
advantage of doing this is that you often think of other criteria related to the
category as you are doing the categorization exercise.

5. Develop media-specific criteria

• These are criteria which ask questions of particular relevance to the medium
used by the materials being evaluated.
 6. Develop content-specific criteria

• It related to the topic and the points of materials being evaluate.

• There would be a set of criteria relevant to the reading skills book which would
not be relevant to the evaluation of a grammar practicebook.

7. Develop age-specific criteria

• It is related to the age of the target learners. It is the only criteria which is
suitable for 5 years old for 10 years old, teenage, young adults, and for mature
adults
• Related to  cognitive and affective development, to previous experience, to
interest, wants and needs
 8. Develop local criteria
• This is related to the actual or potential environment of use.
• These are questions which concerned with measuring the value of
materials for particular learners in particular circumstance.
Features if the environment which would determine this set of materials
are:

• Syllabus
• Class size
• Background , needs and wants oflearners
• Type of institution
• Background , needs and wants ofteachers
• Resource of institution
• Target examinations
• …..
 9. Develop other criteria
Other criteria which might appropriate to develop include:
• Teacher specific
• Administrators-specific
• Gender-specific
• Culture-specific
• L1-specific criteria

10.Trial the criteria

This is important or trial the criteria to ensure that the criteria are
sufficient, answerable, reliable and useful.
 11.Conducting theevaluation

• Keep in mind that evaluations should be based on operational and theoretical

frameworks and that they should be incorporated in a programme’s planning
stages. Baseline and situation analyses are critical to monitoring and evaluation
efforts, but rarely conducted.
Thank you….