Principles of Effective

Evaluation
Prof Keithia Wilson
Academic Leader Student Success
& Retention
Importance of Evaluation
Results in evidence-based practice
Helps us to identify the most effective &
strategic interventions (make a difference
or add value)
Leads to sustainable & enjoyable
practice
Provides accountability to School & Group
leadership
Contributes to scholarship of the FYE


Kirkpatrick (1994) Model
of Evaluation - Overview
4 levels of outcome evaluation

1. Level 1 Reaction
2. Level 2 Learning
3. Level 3 Behaviour
4. Level 4 Results

Level 1 - Reaction
Measures participants reactions to the training
program, including:
reactions to the overall program (outcomes) e.g.,
* To what extent did you find the Orientation
program useful?
reactions to specific components of the program
(processes) e.g.,
* What aspects of Orientation did you MOST
APPRECIATE and found USEFUL?

* What aspects of Orientation did you LEAST
APPRECIATE and feel is MOST IN NEED OF
IMPROVEMENT?



Level 1 - Reaction
Important to include closed-ended items
including rating scales e.g.,
How effective was the Orientation program overall in helping you to
make a transition to university?
not at all effective very effective
1 2 3 4 5 6 7
How useful was the initial information session on the predictors of
success in first year?
not at all useful very useful
1 2 3 4 5 6 7
As well as open-ended items
What else would you have liked included in the Orientation
program today?
Level 1 - Reaction
Some FYE examples
Starting@Griffith Survey questions on
Orientation (retrospective student survey)
School specific Orientation Program
evaluation at the end of O-Day & the
Common Time (ongoing Orientation
Program) student surveys, focus groups
Peer Mentoring Program final evaluation by
mentors & mentees




Level 2 - Learning
Measures what participants have learned from
involvement in the program
What is measured needs to relate to what was
covered in the program viz. learning objectives
Typically covers knowledge, skills, or attitudes
Needs to include both rating scales & open-ended
questions
Can include self-report & tests of actual knowledge

Level 2 - Learning
Some examples
1. Student self-report
* I feel confident that I know what to do to be successful academically
this semester
not at all confident very confident
1 2 3 4 5 6 7

* What have you learned about the transitions you will need to make
as
a commencing student in order to be successful?

2. Student knowledge
* I know the name of my First Year Advisor yes no (circle one)
* I know the difference between a lecture and a tutorial (circle one)
Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree


Level 2 - Learning
Some FYE examples
Starting@Griffith Survey data evaluates learning
outcomes in terms of knowledge & attitudes (e.g.,
knowing where to go to get help; perceptions of
good teaching)
Outcome evaluation for Peer Mentees (What are
your 3 most important learnings from your
participation in the Peer Mentoring Program?)
First-Assessment-First-Feedback amber alert
intervention (What have you learned as a result of
your participation?)




Level 3 - Behaviour
Measures the transfer of knowledge, skills & attitudes
from the training context to in vivo or real-life
contexts (the classroom & study).
Evaluate both before & after the program if
practical
Use survey, focus groups, interviews with students,
mentors, staff (convenors, sessionals)
Starting@Griffith Survey (Did you attend
Orientation? How much time are you spending on
study?)


Level 3 - Behaviour
Some FYE examples
Starting@Griffith survey data
Attendance at E-Day, O-Day &
Common Time/ASP Program
Attendance at lectures & tutes
Assessment submission rates in
courses


Level 4 - Results
Measures return-on-investment, or the extent to
which the program has produced results.
Some FYE examples include
Hard student outcomes such as -
* pass rates in courses for assessment items
* GPA in semester 1, 2 & FY overall
* student retention into year 2
Soft student outcomes such as
* student satisfaction
Evaluation
Key question What difference do our FYO&E
interventions (e.g., Orientation, Peer Mentoring,
Attendance, First Assessment Intervention etc.)
make overall?

Program/Intervention1 Reactions2 Learning 3
Behaviour 4 Overall Productivity/Result

Program/InterventionFeelKnowDowhat is the
result or difference overall?