Instrument Development

Lely Lusmilasari

OUTLINE
Scale construction
Scale/Instrument development
procedures

Pengukuran ?
Pengukuran merupakan fundamental activity of
science
Kegiatan membandingkan suatu hal dengan satuan
ukuran tertentu sehingga sifatnya menjadi kuantitatif
Proses pengumpulan data melalui pengamatan
empiris yang digunakan untuk mengumpulkan
informasi yang relevan dengan tujuan yang telah
ditentukan.
Proses untuk mengumpulkan informasi mengenai
jenis dan / atau jumlah atribut atau karakteristik yang
dimiliki oleh benda-benda sesuai dalam
kelompoknya( Waltz , et.al , 2010)

Instrumen/alat ukur ?
Alat yang digunakan untuk mengukur
fenomena alam maupun sosial yang diamati
Alat yang digunakan untuk merekampada
umumnya secara kuantitatifkeadaan dan
aktivitas atribut-atribut psikologis

CONTOH?

PILIHAN KETIKA MEMBUTUHKAN

INSTRUMEN

Merancang dan mengembangkan

instrumen
Menggunakan instrumen yang
telah ada

KEUNTUNGAN & KELEMAHAN

MENGGUNAKAN INSTRUMEN YANG ADA

Mudah
(Sudah banyak instrumen yang dihasilkan baik)
Baik dalam pengembangan ilmu
- Memiliki uji validitas lebih
- Bisa membandingkan
- Efisien

Kelemahan

Tidak sesuai dengan tujuan

Membutuhkan tes psikometri kembali
Harus membayar atau mendapatkan ijin

KEUNTUNGAN & KELEMAHAN

MENGEMBANGKAN INSTRUMEN BARU

Sesuai dengan tujuan

Novelty ---- belum ada/sedikit instrumen
sebelumnya

Kelemahan

Waktu lama
Belum dapat dibandingkan

PERTIMBANGAN SAAT EVALUASI

INSTRUMEN
Match to objective
Psychometric evidence
Feasibility / logistics

SCALE/INSTRUMENT DEVELOPMENT

to construct a reliable and valid measurement instrument

to have any confidence in drawing conclusions about the
construct(s) being

INSTRUMENT DEVELOPMENT AND VALIDATION

process of accumulating evidence

SCALE CONSTRUCTION PROCESS

(Croker & Algina, 1986)
First phase: Scale design
1)identify the primary
purpose(s)
2)identify behaviors that
represent the construct or
define the domain
3)3) prepare a set of test
specifications
4)construct an initial pool
items
5)have items reviewed

Second phase: Scale

development
1)hold preliminary items tryouts
2)field-test the items on large
sample representative
3)determine statistical
properties
4)design and conduct reliability
and validity studies for the final
form of the test
5)develop guidelines for
administration, scoring, and
interpretation of the test scores

SCALE DEVELOPMENT PROCEDURES

(DeVellis, 2003; Benson & Clark, 1982)
PLANNING PHASE

CONSTRUCTION
PHASE

QUANTITATIVE
VALIDATION
PHASE

- Identifying the - Selecting a

- Establishing
construct to
response
reliability and
be measured
format
validity
- Determining
- Generating an - Selecting
the target
item pool
items for the
group for
final
- Obtaining
which the
content
instrument
measurement
validation
is intended
- Establishing
operational
definitions of
the construct

CONTOH
PLANNING PHASE
Clarification of the structural domain of the parental feeding behavior
Formulation of the statement

Review literature

Formulation of the operational definition

CONSTRUCTION PHASE
Generating item pool of the PFBQ for Indonesian parents with
toddlers
(Initial item pool)
Conducting content validity by 6 experts
The first draft of the PFBQ
Pretest study on the first draft of the PFBQ
The second draft scale
VALIDATION PHASE
Main study for testing of the second draft scale
Establishing psychometric properties
Construct validity using
confirmatory factor analysis

Construct validity using contrasting

group technique

Construct Reliability and

stability

Contoh: The Development of the Parental Feeding Behaviors

Questionnaire (PFBQ)
PLANNING PHASES:
Concept clarification/determine clearly what is to be
measured
The first steps, this
most important
phase begin with the
formulation of the
statement of the
purpose of the
intended instrument

In this study, the instrument constructed

to assess the feeding behavior of
Indonesian parents to maintain adequate
food intake for their toddlers, where
feeding behavior of parents is the
construct to be measured and the target
group is Indonesian parents with toddlers.

The second steps.

Requires more time
than the first steps
since it involves a
review of the related
literature

First, this step ensures that an

appropriate, reliable, and valid instrument
does not already exist
Second, the review of the literature aids
in formulating an operational definition for
the construct to be measured ----- Orems
Theory, Feeding Behavior of Parents in
Indonesian context, Toddlers nutrition

A review of the literature reveals that the measurement of

parental feeding beahiors of parents with toddlers was only
focused on parents activities during feeding as follow:
Instrument
, author
(year)

Construct
definition

Items
&scales

Population

Psychometric
measure

Feasibility/a
pplicability

Strength and weakness

Comprens
ive
feeding
practice
questionn
aire
(CFPQ);
MusherEizenman.
, & Holub,
(2007)

The CFPQ
is develop
using
previous
measures
(child
feeding
questionna
ire &
preschool
feeding
questionna
ire) as an
initial
framework

All item
are
scored
on 5points
likerttype
scale.)

Parents
of
children
between
the ages
of 18
months
and 8
years old

Content validity
Construct validity
The Flesch-Kincaid
reading level
Internal consistency

Self-report
questionn
aire

Strenght:
Through the three
studies, the author
attempt to capture a
broad range of
behaviors that
parents might be
engage in when
feeding their children
that might be related
to healthier or
unhealthier feeding in
their children
Weakness:
Less is currently know
about the reliability of
the measure. The
index of reliability is
lower (average
range : 1.69 -4.86)

Only one instrument, the Parents Nutrition Dependent-Care

Questionnaire (PNDCQ) (Moore, et.al., 2005), was created to
measure parents nutrition behavior for their adolescents that includes
three kinds of operations as follow:
Instrument,
author (year)

Target
population

Scope

Psychometric
measure

The Parent
Nutrition
Dependent-Care
Questionnaire
(PNDCQ);
Moore, et.al.,
(2005)

Parents with
adolescent

Parents nutrition dependentcare for their children both

cognitive and psychomotor
activities include estimative,
transitional, and production self
care operations

Content validity
Items analysis
Internal consistency

This instrument is not appropriate for parents with toddlers, original item
wording is not relevant to the current population
More items should be added to this instrument reflecting estimative,
transitional and productive operations in providing appropriate dietary
intake pattern, providing safety and healthy food and providing a pleasant
environment during feeding activities based on toddlers characteristics

The Development of the PFBQ cont..

CONSTRUCTION PHASES:
Selecting a response format, generating an item pool,
and obtaining content validation

Selecting a
response
format

In this study, parents feeding behavior will

be assessed with a 5-choice Likert type
scale format: 1 (never), 2 (rarely), 3
(sometimes), 4 (most of the time), and 5
(always). The rating scores are summated
by total score. The higher the total score
is, the better the rating of the parental
feeding behaviors or indicating better
feeding behaviors of parents

Generating an
item pool

Next slide..

Obtaining
content
validation

Next slide..

THE EXAMPLE OF INITIAL ITEMS

1. Providing healthy food

Acquiring
and
gathering
informatio
n to know
about
toddlers
diet,
healthy
and
safety
food

I read about balanced, healthy and safety foods for toddlers from
various resources (e.g. magazine/books/pamphlets/leaflet or others)
I learn on how to prepare healthy menu by asking healthcare provider, cadre
or others
I ask other parents and family about their experience to enhance good eating
behavior of toddlers
I will make home-prepared foods as often as possible to prevent foodborne
illness
I will identify the sources of harmful substances in food that consumed by my
child
I wash hands during preparing and cooking food
I cook food thoroughly, especially beef, fish, poultry, and eggs
I give snacks (mung bean porridge or biscuit or nagasari or others) at least
twice a day between meal

CONTENT VALIDATION
Concerns the degree to which an instrument has an
appropriate sample of items for the construct being
measured and adequately covers the construct
domain
the degree to which the items, questions or
elements of and instrument are representative of
the universe of content or the domain of content
(Nunnaly & Berstein, 1994)
the purpose of content validation study is to assess
whether the items adequately represent a
performance domain or construct of specific
interest (Croker & Algina, 1986)

CONTENT VALIDATION cont.

described content validation as a rigorous
assessment consisting of two-stage process,
development and judgment-quantification (Lynn,
1986)
DEVELOPMENTAL STAGE composed of three steps:
domain identification, item generation and
instrument formation
JUDGMENT-QUANTIFICATION STEPS, determining
the number of experts and application of the index
of content validity (CVI)

Expert Review is advisable to

undertake two round of review (Polit &
Beck)

Contoh :The development of the PFBQ cont

EXPERTS REVIEW PROCEDURES :

The total experts in the first and second phase are six
experts which include pediatric nurses (3), dietitian (2), and
pediatrician (1)
Each of the six experts individually evaluate the PFBQ
They will be given the objectives and items and used a form
independently rate the representativeness, clarity and
simplicity as 1 (not relevant), 2 (somewhat relevant), 3
(relevant), or 4 (very relevant)
Additionally, interpretability will be assessed if there was
ambiguous wording in question
the result from content validity are to identify the items that
should be refined, changed, or deleted following comments of
suggestion of a panel expert

INTRODUCTORY GUIDE FOR EXPERTS REVIEW

PROCEDURES

Background and the purpose of questionnaire

The reason why they were choosen
The characteristic of the instrument
What they should do

Parental feeding behavior items

Representativeness

Burden
dimension
1 = subjective

The feeding behavior of parents with 1 = the item is not

toddlers is the activities of Indonesian
representative of
2 = objective
parents related to the maintenance of a
parents nutrition
sufficient intake of food for their toddlers
dependent-care
3 = unable to
involving not only to providing food that are
classify
2 = the item needs major
nutritionally adequate, healthy and safety
revisions to be
but also to helping the toddlers to establish
representative of
good eating behavior; and establising eating
nutrition dependent-care
environment
that
are
pleasant
and
interactive. The activities of the parents will 3 = the item needs minor
be described as follow:
revisions to be
representative of
1)Provide food
that are adequate,
parents nutrition
healthy and safety is defined as activities of
dependent-care
parents for their toddlers that involve finding
resources about
adequate, healthy and 4 = the item is
safety food; identifying the kind of foods and
representative of
beverages a toddler eats and drinks, the kind
parents nutrition
of hazard such as the presence or noxious
dependent-care
substances in food; making decisions about

toddlers daily intake; planning

healthy
meals/menu and safety food; and taking
action (perform activities that related to
providing adequate, healthy and safety food


1
3 4
1 2 3
1. I ask health provider or cadre
2
(health volunteer at health center in Comments: Comments:
sub-village) about healthy food to

eat

1. I will make menu planning every

day for toddlers

1 2 3 4
1 2 3
Comments: Comments:

1. I give snacks (mung bean porridge

or biscuit or nagasari or others)
twice a day between meals

1 2 3 4

1 2 3

Comments: Comments:

Clarity Are the parental feeding behavior items well written,

and at an appropriate reading level for parents with toddlers in
urban and rural area?
____ Yes, the following items are clear ( in the space below,
indicate which items are clear)
____ No, some of items are unclear (in the space below,
indicate which items are unclear)
Suggestion for making the items clear:

The result of content validity by expert review (N=6)

experts)
Aspect
Criteria
Result
Conclusion
Representativeness
I-CVI

>.78

.83-1.00

Meet Lynns
criteria

S-CVI

>.90

.98

Meet Lynns
criteria

80%

83%100%

Meet criteria

Clarity
Percentage clarity
agreement

Comprehensiveness:
All dimensions of the content domain are included in the
instruments, but 16 item with minor revision and 10 items
added
80 items for pre-test study

The development of the PFBQ cont

The result of pre-test study: initial item analysis (N=30)

Item
analysis

Criteria

Result

Conclusion

Inter-item >.3 but

correlation not to
large (< .
8)

Two pairs have

score > .08

Item 72 deleted
because too similar
with item 66

Item to
> .3
total
correlation

First analysis:
23 items were
deleted because
< .3
Second analysis:
Only item 16
have low
correlation

56 items have
correlation > .3

Cronbach
alpha

.94

Meet criteria

> .7

56 items used in the main study

VALIDATION PHASE
PSYCHOMETRIC PROPERTIES TEST

VALIDITY
(how well the test accurately
measures the construct of
interest)

Face, content, construct,

and criterion validity

RELIABILITY
(the test's ability to measure
the construct of interest
consistently)

Internal consistency, stability

and equivalence

CONTOH VALIDITAS
(instrumen tersebut dapat digunakan
untuk mengukur apa yang seharusnya diukur
Validitas Isi (Content Validity) : instrumen yang berbe)ntuk test yang
sering digunakan untuk mengukur prestasi belajar (achievement)
dan mengukur efektivitas pelaksanaan program dan tujuan.
Validitas Konstruk (Construct Validity) :Jika instrumen dapat
digunakan untuk mengukur gejala sesuai dengan yang didefinisikan
sesuai dengan teori-teori yang relevan
Validitas yang berkaitan dengan kriteria (Criterion-related Validity) :
Terjadi ketika sebuah instrumen membedakan individual pada kriteria
yang akan diperkirakan

CONTOH RELIABILITAS

(menunjukkan konsistensi dan stabilitas dari

suatu skor/skala pengukuran)

Reliabilitas Instrumen dapat diuji dengan :

1)Test-retest Reliability;
2)Equivalent / Paralel-form Reliability
3) Internal Consistency Reliability

The development of the PFBQ cont

Confirmatory factor analysis
Goodness of Fit

Values

Statistic

Hair and
colleagues
(2010)
< .05

DF

Hypothesized

Modified

model

Model

5481.68

2383.31

(p=.01)

(p=.01)

1481

1355

2/df

< .03

3.70

1.76

CFI

< .90

.66

.91

TLI

< .90

.65

.90

RMSEA

< .07

.07

.03

SRMR

< .08

.06

.04

Figure: The Standardized of

modified measurement model
of the PFBQ

The development of the PFBQ cont

Factor loading, construct reliability and Average variance Extracted in each
dimension (N=548)

Dimensio

Constr

uct

Unstand

Standar-

ar-dized

dized

factor

factor

loading

loading

(b)
Provide

Parenta

food

Enhance

feeding
Parenta

eating

Provide

feeding
Parenta

environm

ent

feeding

S.E.

R2

AVE

CR

(B)
.851

.001

.725

.731

.92

.931

.911

.14

6.647

.001

.82

.820

.96

1.406

.915

.147

9.585

.001

.837

.839

.93

Based on the result of validation phase :

The PFBQ composes of three dimensions with 56 items that valid and
reliable

THANK YOU