Barbee 1

Stephanie A. Barbee
DETC 620-9020
July 13, 2014
A-2: Multimedia Evaluation
Word Count: 1,078
Multimedia Evaluation: Khan Academy & NeoK12
Introduction
The Multimedia Educational Resource for Learning Online Teaching (MERLOT) is a
teaching learning database that provides peer-reviewed materials relating to the arts, business,
humanities, mathematics, science and technology for students and faculty in the California State
University system (MERLOT, 2014). Each year MERLOT presents an award to the multimedia
program that best fits their overall vision according to predetermined standards and criteria. For
the upcoming multimedia award, the Barbee Multimedia Evaluation Jury (BMEJ) was tasked
with assessing and evaluating two multimedia product finalists based on the usability and
pedagogical design of each product. The two finalists selected by MERLOT, under the category
of K-12 Educational Resources, are Khan Academy and NeoK12. This evaluation includes a
description of each finalist, outlines the framework and methodology of the evaluation process,
assesses each multimedia program using the Learning Object Review Instrument (LORI)
approach, and provides a summary explaining BMEJs chosen award winner.
Multimedia Program Descriptions
The first finalist, Khan Academy, is completely free of charge and offers education-based
materials and resources to teachers, parents, and students world-wide (Khan Academy, 2014).
Barbee 2

Khan Academy includes an interactive library, challenges, assessments, videos, practice
materials, a library of content, a private profile to track progress, achievements, and access to
learning coaches. This multimedia resource offers learning materials in the disciplines of
mathematics, science, economics, humanities, computing and test prep. An interesting feature of
Khan Academy is that it is partners with the J. Paul Getty Museum, NASA, MIT and many other
reputable companies and institutions to provide students with exciting and up-to-date
information that facilitates learning as they progress through their educational journeys.
The second finalist, in the field of K-12 Educational Resources is NeoK12. NeoK12s
believes that kids learn best when they interact with the content they are learning, and this
multimedia resource provides innovative tools to enhance their overall learning experience
(NeoK12, 2014). This program offers quiz games, interactive diagrams, flow chart games,
vocabulary games, puzzles, presentation tools, and an interactive whiteboard. In addition, there
are supplementary resources that allow teachers to create and share materials and instructions
with other teachers, students and parents. This program addresses subjects like geography,
science, mathematics, the human body, history, and many others. NeoK12 (2014) is part of the
American Library Association, and was awarded a Viewer Choice Award and Homeschool Top
100 Award.
Evaluation Overview
The BMEJ will evaluate Khan Academy and NeoK12 using the LORI tool, which
considers quality criteria specific to resources like those found in the MERLOT database. LORI
is an evaluation method that allows reviewers to rate learning objects, and results are presented
as a set of averaged ratings, one per item, that are summarized as a single average (Nesbit,
Barbee 3

Belfer, & Leacock, 2007, p. 2). Learning objects are information resources utilized in online
learning environments, and are rated based on the following criteria: content quality, learning
goal alignment, feedback and adaptation, motivation, presentation design, interaction usability,
accessibility reusability and standards compliance (2007, p. 2). According to Leacock and
Nesbit (2007), there are three specific reasons that digital resources need to be evaluated:
the design of multimedia learning materials is not informed by relevant research in
psychology and education
the efficacy of using quality metrics for search results is directly dependent on the
validity of the evaluative tool that generates quality ratings
the quality criteria for summative evaluations have the potential to drive improvements in
overall design practice (p. 44)
The Numerical Weight and Sum (NWS) was not a preferred evaluation method because it
assumes a linear scale of utility for criteria. In addition, the NWS method addresses learning
objects on an ordinal scale that allows for ranking, but not for consistent or incredibly accurate
calculation (Baumgartner & Payr, 1997). Another common evaluation tool, known as the
Qualitative Weight and Sum (QWS), addresses some of the methodological difficulties found in
the NWS. However, this evaluative tool was not a preferred method because it does not offer a
clear decision algorithm and often has to be applied several times before drawing conclusion
based on the results it presents (Baumgartner & Payr, 1997). Reeves and Harmon (1994)
emphasize the significance of clearer evaluations and further advancements within the usability
and pedagogical dimensions, which are paramount in order to effectively evaluate multimedia
learning resources. As a result of the inadequacies of the NWS and QWS evaluative tools,
BMEJ chose to utilize the LORI approach to ensure criteria was evaluated based on a summative
Barbee 4

discourse that identifies each learning objects strengths and weaknesses (Leacock & Nesbit,
2007).
Evaluation
In the LORI evaluation worksheet (figure 1), Khan Academy is labeled as multimedia
product A (MM-A), and NeoK12 is labeled as multimedia product B (MM-B). The LORI
evaluation worksheet includes the score each product received for each of the criteria and a
rationale for each score given. Learning objects are assessed using a five point rating scale
where one is a low rating and five is considered a high rating. BMEJs overall goal is to produce
a winner for the multimedia award based on an engaging, factually accurate, pedagogically
sound, effective and reusable learning object (Krauss & Ally, 2005, p. 7).





MERLOT Multimedia Award Evaluation



Criteria Rationale MM-A MM-B
Content for MM-A & MM-B was without errors, bias
Content Quality or discrimination. Key points and ideas were supported 5 5
and balanced with appropriate detail.
Learning goals were declared and age appropriate
Learning Goal Alignment for both products. Some content with MM-B did not 5 4
align with goals.
Both products maintain a model of the learner and
Feedback and Adaptation include tailored messages for responses to activities. 5 5
Learner responses trigger subsequent presentations.
Both products include highly motivating content that
Motivation aligns with learner's goals & interests. Activities 5 5
are realistic and challenging; fosters improvement.
Both products enable efficient learning, with legible
Presentation Design text & clutter-free content. Writing is clear, concise 5 5
Barbee 5

& free of errors; aesthetically pleasing to the eye.
Both products had clear instructions and navigation
Interaction Usability was easy and intuitive. However, MM-B had inconsistent 5 4
behavior and some delays and MM-A did not.
Both products meet the guidelines at level AAA, but
Accessibility MM-B provided a higher degree of accommodation for 4 5
learners with disabilities than MM-A.
Both products avoid references to other course
Reusability components, but often uses a higher level of terminology 4 4
that doesn't fit a broad range of learners.
Both products adhere to standards and specifications
Standards Compliance outlined by IMS, IEEE, SCORM, & W3C. Metadata 5 5
for both is in tagged code & available in page form.

TOTAL SCORE: 43 42

Figure 1. MERLOT Multimedia Award: LORI Evaluation Worksheet for Finalists
Analysis/Conclusion
As a result of the LORI evaluation, Khan Academy outscored NeoK12 by a very close
margin. Both learning resource products scored well, but there were a couple areas where one
was clearly superior over the other. For example, under the criteria Learning Goal Alignment,
NeoK12s content did not consistently align with learning goals. While learning goals were
clearly identified for both products, NeoK12 often included questions on module quizzes that did
not align with the content presented for that module. Under the criteria Interaction Usability,
Khan Academy presented a user interface that was easy to navigate, with clear instructions to
guide the overall use of the product. While NeoK12 also presented a user-friendly interface,
oftentimes navigation was inconsistent with excessive delays. The one area where NeoK12
outscored Khan Academy was Accessibility. While both products met the IMS Guidelines for
Accessible Learning Applications and conform to W3C Web Content Accessibility Guidelines,
NeoK12 clearly presented a higher-degree of accommodation for learners with disabilities.
Barbee 6

It is clear that both multimedia learning resources have a high quality of content
development that is facilitated by sound theories of learning and cognition. Khan Academy and
NeoK12 are exceptional representations of multimedia learning resources that will benefit
MERLOT and the online learning community. With that said, the Barbee Multimedia
Evaluation Jury has determined that this years MERLOT Multimedia Award winner, under the
category of K-12 Educational Resources, is Khan Academy.













Barbee 7

References
Khan Academy (2014). A free world-class education for anyone anywhere. Retrieved from
http://www.khanacademy.org/about
Krauss, F., & Ally, M. (2005). A study of the design and evaluation of a learning object and
implications for content development. Interdisciplinary Journal of Knowledge and
Learning Objects, 1(1), 1-22. Retrieved from http://www.ijelllo.org/Volume1/v1p001-
022Krauss.pdf
Leacock, T. L., & Nesbit, J. C. (2007). A Framework for Evaluating the Quality of Multimedia
Learning Resources. Educational Technology & Society, 10(2), 44-59. Retrieved from
http://www.ifets.info/journals/10_2/5.pdf
MERLOT (2014). About MERLOT: Who we are. Retrieved from
http://info.merlot.org/merlothelp/index.htm#who_we_are.htm
NeoK12 (2014). Educational Videos, Lessons and Games for K-12 School Kids. Retrieved from
http://www.neok12.com/
Nesbit, J., Belfer, K., & Leacock, T. (2007). Learning object review instrument (LORI) user
manual. Retrieved from
http://www.transplantedgoose.net/gradstudies/educ892/LORI1.5.pdf
Reeves, T. C., & Harmon, S. W. (1994). Systematic evaluation procedures for interactive
multimedia for education and training. In S. Reisman (Ed.), Multimedia computing:
Preparing for the 21
st
century. Hershey, PA: Idea Group Publishing.