Questionmark White Paper

Delivering Assessments Safely and Securely

This paper discusses the issues to consider in delivering assessments using computer or web-based technologies. It is designed to help readers select delivery methods and technologies that are most in keeping with a particular assessments purpose and nature. The paper explores the various means and technologies available for deploying a wide range of assessment types safely and securely.

Authors:

Eric Shepherd John Kleeman Joan Phaup Greg Pope

1.

Introduction

The use of computer- and web-based applications to assess knowledge, skills, and attitudes is now widespread. Today, distinguishing between the various delivery and security requirements for each style of assessment is becoming more important. Over-engineering low-stakes assessments can result in unnecessary costs and wasted time. Under-engineering high-stakes assessments can undermine peoples confidence, organizations processes, and undermine the face validity of the assessment. This paper focuses primarily on assessments that utilize computer- and web-based applications rather than paper-based assessments. It describes various types of assessments and explains numerous delivery options so that you can select appropriate methods for deploying each type of assessment safely, securely, and cost effectively. The paper also explains how Questionmark products can be used to ensure the safe and secure delivery of assessments. One important consideration when selecting a delivery method is whether assessments are used to: a. Measure knowledge, skills, attitudes, and personality traits; or b. Promote learning and reduce forgetting. Another way of choosing a delivery method relates to an assessments consequences. After some assessments people might be hired, fired, promoted, demoted, graduated, not graduated, released from custody, and/or authorized or certified to perform a particular job or role. The potential outcomes of an assessment could affect a participants propensity to cheat and consequently how much security must be applied. Specific assessments are often referred to as quizzes, tests, exams, or surveys depending on whether the intent of the assessment is to measure or promote learning. We have created some distinctions for some terms in the table below that we will use throughout this paper: Term Assessment Definition Any systematic method of obtaining evidence from posing questions to draw inferences about the knowledge, skills, attitudes and other characteristics of people for a specific purpose. A summative assessment used to measure a students knowledge or skills for the purpose of documenting their current level of knowledge or skill. A formative assessment used to measure a students knowledge or skills for the purpose of providing feedback to inform the student of their current level of knowledge or skill. A diagnostic assessment to measure the knowledge, skills, and/or attitudes of a group for the purpose of determining needs required to fulfill a defined purpose. Measure High Measure or Learn Consequences or stakes

Exam

Quiz

Promote learning

Low

Survey

Measure

Low

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 2 of 24

Test

A diagnostic assessment to measure a students knowledge or skills for the purpose of informing the student or their tutor on their current level of knowledge or skill.

Measure to promote learning

Medium

(See the Appendix for detailed examples of assessments types.)

In addition to this terminology we can classify assessments by using the following terms: Name of assessment Formative Summative Diagnostic Uses These assessments are designed to help students turn information into knowledge by asking questions that tend to increase remembering and reduce forgetting. These assessments are designed to measure knowledge, skills and attitudes by asking questions and measuring the responses. These assessments are designed to provide a diagnosis and prescription to help people reach their objectives.

The method of delivering an assessment depends largely upon the use and consequences of the results, but also on the motivation of the participant to complete the assessment. Type of assessment Exams Consequences Motivation to complete assessment High

Consequences for passing or failing an exam might extend to being hired, fired, promoted, demoted, released from custody, authorized or certified, graduating or not graduating. There are few if any consequences for passing or failing a quiz. Quizzes are normally formative assessments used to promote learning and reduce forgetting rather than forming a summative judgment. There should never be direct consequences to a respondent i.e. the person answering a survey. However, as results are tabulated for the group, training courses, job aids, and other interventions may be planned. Sometimes the opinion might have consequences for others such as an instructor in the context of a course evaluation or managers in the context of an employee attitude survey. The consequences for passing or failing an individual test are not high. However, consistent failures might extend to being hired, fired, promoted, demoted, graduating or not graduating.

Quiz

Medium

Survey

The challenge is to get people to respond.

Low

Test

Medium

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 3 of 24

This paper does not attempt to describe all of the consequences, legal liabilities, validity, reliability and planning issues that are involved with writing an assessment that is suitable for its purpose. Many other publications detail the processes and procedures used to produce good and fair assessments. However, as you can see from the chart below all of these issues are related to the stakes of an assessment.

What kind of assessments are we talking about?

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 4 of 24

This paper details issues related to safely and securely delivering assessments without over- or underengineering the delivery and deployment process. The following table details some of the issues that you must consider when using computer or web-based applications to deploy assessments: Type of assessment Exams Delivery Issues to consider o Face validity (trust of the results) o Motivation to cheat is high o Over exposure of content o Content is expensive or time consuming to produce o Content protection o Authentication of candidate o Correct authorization (pre-requisites completed) o Time window to limit access o Disallow repeated access to exam o Consistency of delivery o Secure player required o Answers saved regularly in case of a technical failure o Content protection o Environment similar to performance situation o Motivation to cheat is medium o Ballot rigging o Anonymity of respondent o Face validity (trust of the results) o Motivation to cheat is high o Over exposure of content o Content protection o Environment similar to performance situation o Time window to limit access o Limit the number of times a person can have repeated access to test o Consistency of delivery o Secure players are recommended o Answers saved regularly in case of a technical failure

Quiz Survey Test

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 5 of 24

2.

Environments for delivering assessments

The delivery environment for an assessment is driven by the stakes of the assessment. The higher the value of the assessment, then the more care will be taken to control the environment and the supervision of the assessment. As assessments come attached with a wide variety of consequences, there are a wide variety of ways and places that can be used to deliver an assessment. Delivery method
High-Stakes Assessments Professional monitor (proctor/invigilator) Dedicated workstation Dedicated workstation Secured workstation Professional center Franchised center Training center

Any computer controlled environment Any computer in an uncontrolled environment Low-Stakes Assessments No monitoring Low volume High volume

2.1 Testing Centers Testing centers are controlled environments set up to provide groups of users with a venue where the conditions laid out by the organization that is running the assessment can be met. This means that candidates can rely on fair conditions under which to take the assessment. High-stakes assessments must offer a consistent environment that is free from distractions together with some form of monitoring by an invigilator or proctor. Low-stakes assessments do not require such rigor. The greater the rigor that is expected, the more expensive it will be to deliver an assessment. The physical environment, technology, monitoring standards and authentication methods for delivering an assessment are stipulated by the sponsoring organization that owns the assessment.

Physical Environment: Testing centers should be enclosed rooms without through traffic. They

should be quiet and free from distractions, well lit and with a comfortable temperature throughout the year. Workstations should be separated from one another by acoustic partitions and the equipment should offer similar operation, accessibility options and performance. Candidates should not be able to print or capture the assessment content and should be prevented from accessing resources, such as web pages, that might unfairly assist them while they take the assessment. Also: See tips in Sections 4 and 5 of this white paper.

Technology Environment: Assessments should be delivered on consistent and comparable

technology. Screen sizes, processor speeds, and network connections do not have to be the latest and greatest but they do need to be comparable. It is vital to anticipate and plan for

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 6 of 24

technical requirements for delivering bandwidth-intensive content such as video, which puts increased demands on computer networks.

Monitoring: Testing centers can be monitored (or proctored/invigilated) in a number of ways. Sections 4 and 5 of this white paper.

Candidates should be in an environment that permits constant visual monitoring. This can be achieved directly by a person in the room or via video surveillance equipment. Also: See tips in

Authentication: It is important to confirm that the person taking the test or exam is truly the
person authorized to do so. This can be done by confirming a candidates identity via a photographic ID provided by the government, academic institution, or, in some cases their employeror simply a signature. Also: See tips in Section 4 about screening candidates.

To aid the following discussion we have described the issues to consider and established the following grades to help you quickly understand the suitability of the delivery method for a particular style of assessment.

Quick Reference Guide for Suitability

Suitability for style of assessment Very suitable Suitable Suitability will vary by circumstance Unsuitable Definitely not suitable 2.2 Types of dedicated testing centers Testing centers come in various guises. Some of the ways in which they can be defined are as follows: Grade A B C D F

Professionally controlled centers: Professionally controlled centers provide a very

consistent and highly controlled environment where a candidate could expect to receive exactly the same treatment and receive the same experience from one testing center to another. This consistency would start with the reception and continue through the assessment process to the time that they leave the premises. These centers are used for very high-stakes tests such as nursing and medical exams and are expensive to use and maintain. These centers provide the consistency and rigor required for very high stakes exams such as government regulated certifications exams. Non-regulatory exams are rarely delivered in these centers due to their high running costs.

Professionally controlled centers

Form of assessment Very high-stakes exams High-stakes exams Quiz Survey Test Grade A+ A F F D

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 7 of 24

Franchised centers: Franchised centers are similar to professionally controlled centers but

they cannot enforce the rigor required for very-high-stakes assessments. However, these centers are less expensive and provide a great place to offer high-stakes tests in a reasonably consistent environment. These centers are far more prolific than professionally controlled centers and deliver government regulated exams as well as non-government regulated exams. There are thousands of franchised centers around the world.

Franchised centers
Form of assessment Very high-stakes exams High-stakes exams Quiz Survey Test Grade B A F F C

Higher education centers: Many higher education institutions offer their testing facilities to
local companies and are sometimes part of a franchised assessment network. These centers are similar to professionally controlled centers and they enforce the rigor required for their institutions. These centers are less expensive and provide a great place to offer high-stakes tests in a reasonably consistent environment. These centers are ideal, not only for the institutions use but also for local companies that need to administer high-stakes assessments in conjunction with a local institution.

Higher Education centers

Form of assessment Very high-stakes exams High-stakes exams Quiz Survey Test Grade B A D F C

Corporate centers: Increasingly large organizations are establishing their own testing

centers to provide a consistent, confidential, and cost-effective method to deliver high-stakes assessments. These centers might be testing potential employees, partners or existing employees on safety or government regulation related issues. These centers are equipped in a similar fashion to professionally controlled or franchised centers.

Corporate centers
Form of assessment Very high -exams High-stakes exams Quiz Survey Test Grade A A C C B

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 8 of 24

2.3 Non-dedicated testing centers Most assessments are taken within non-dedicated testing centers. Moreover, as the features, quality and ingenuity of assessment software grows and methods of authentication improve, more and more higher-stakes assessments are being administered within this type of environment. Non-dedicated assessments centers can be one of the following:

Training rooms doubling as testing centers: Training rooms may not offer such a tightly

controlled environment as you might find at a dedicated testing center, but they could be appropriate for certain situations. For example, a training room is unlikely to offer screening between computers, but these limitations can be easily overcome with a little thought and by deploying professional assessment software. Applications such as Questionmark Perception and Questionmark Secure can provide the technology to turn training rooms into testing centers. Professional software packages can prevent printing, task switching, capturing assessment content and exiting inappropriately. Assessment software can also enable you to randomize the distracters in multiple choice and response questions or even create an assessment from randomly drawn questions, thereby making it difficult to cheat.

Training rooms
Form of assessment Very high-stakes exams High-stakes exams Quiz Survey Test Grade BAB A A

Closely supervised in work place: Of all the issues that affect the delivery of

assessments, monitoring (invigilating or proctoring) remains critical for delivering high-stakes certification exams. Widely dispersed candidates can be monitored by their local supervisors or managers, making supervised assessments in the work place a practical, valid and cost effective way to provide high-stakes assessments. Beyond that, the advances in assessment software are now proving that even very valuable assessments can be scheduled at an assessment takers own desk and taken with limited supervision. Assessments can open at a specified time, preventing the candidate from getting a sneak preview.

Closely supervised in work place

Form of assessment Very high-stakes exams High-stakes exams Quiz Survey Test Grade C B+ C C A

Remotely supervised in work place: With advances in technology it is now possible to

remotely monitor candidates using video cameras. Although not suitable for very high-stakes assessments, this can provide a valid way to administer exams remotely.

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 9 of 24

Remotely supervised in work place

Form of assessment Very high-stakes exams High-stakes exams Quiz Survey Test Grade C C+ C C B

Unsupervised at work place: Low- and medium-stakes assessments taken unsupervised at

a participants desk or other work venue are becoming increasingly important as organizations strive to discover how effective their training is, determine who needs training, and provide new ways of promoting the learning process. Although this form of assessment doesnt require the same monitoring, it does require assessment software to prevent people printing or capturing confidential information.

Unsupervised in work place

Form of assessment Very high-stakes exams High-stakes exams Quiz Survey Test Grade F D A A A

Supervised at home: Some distance learning institutions and small-scale certification

authorities are now administering higher-stakes assessments at home. However, the candidate must provide the organization with a choice of professionals (e.g. a doctor, lawyer or accountant) to supervise the assessment. Once again, the success of this kind of delivery lies in the assessment software and the selection methods for the monitor.

Supervised at home
Form of assessment Very high-stakes exams High-stakes exams Quiz Survey Test Grade C A C C A

Unsupervised at home: Most commonly used for delivering low-stakes self-assessments

aimed at promoting learning and reducing forgetting. Form of assessment Very high-stakes exams High-stakes exams Quiz

Unsupervised at home
Grade F F A

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 10 of 24

Survey Test

A B

Unsupervised in a public place (library): Most commonly used for delivering low-stakes
self-assessments aimed at promoting learning and reducing forgetting.

Unsupervised at home
Form of assessment Very high-stakes exams High-stakes exams Quiz Survey Test Grade F F A A B

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 11 of 24

3. Creating the right environment Delivering assessments securely is becoming easier as the quality of networks and assessment software improve. Secure assessment players such as Questionmark Secure, as well as monitoring solutions, are pushing back the barriers of what types of assessments can be safely and securely delivered either remotely, at the desk or in a training room. When close monitoring and a consistent environment are essential for geographically dispersed candidates, there are many excellent testing centers to choose from. Many organizations are now being empowered by assessment software to create and run high-stakes assessments within their own environments. As the quality of software, monitoring and network solutions grow, this trend will certainly continue.

Case study
Not all organizations believe that it is essential to have exams proctored.
A U.S. software company perceived that it was more important to ensure that certification was available to everybody who wanted to get it than to insist that all certification exams be proctored. Setting proctored assessments in specific locations at specific times inevitably limits the opportunities for sitting the assessments. Instead, following the successful completion of the software training courses, candidates can register using a credit card for the exam to become a certified professional in the use of this companys products. Candidates then take the appropriate assessments at home or wherever they wish. The assessments are authored, delivered and reported on using Questionmark Perception. The company argues that, should anybody dispute the validity of a certified professional, they can arrange to serve as proctor and request that the person retake the test. As a result, the company has found it much easier to run its certification program and is confident that the system is not abused. Many organizations have varying approaches to delivering their assessments. Below are several case studies that exemplify how Questionmark is used in varying contexts.

Case study
Healthcare Financial Management Association (HFMA)
The Healthcare Financial Management Association (HFMA) is a not-for-profit organization for CFOs, comptrollers, accountants and other financial management professionals at healthcare organizations. HFMA members can earn the Certified Healthcare Financial Professional (CHFP) designation through professional experience, education, and successful completion of two certification exams: a general exam in healthcare finance practices and an exam in one of four -specialties: accounting and finance, patient and financial services, financial management of physician practices, or managed care. Exams are administered using Questionmark Secure, which locks down candidates desktops and prohibits

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 12 of 24

task switching. Each examinee must enter a password, (good for only 24 hours and allowing only one attempt). The proctora certified HFMA member who has signed an affidavit agreeing to abide by stated proctoring policiesmust also enter a password. There are three versions of each exam, and questions pools are updated every two years.

Case study
University of South Alabama, College of Medicine
At the University of South Alabama (USA) in Mobile, the College of Medicine offers courses in specialties such as neuroscience and microbiology to students in their first two years of medical school. For many years, tests were administered by way of paper and pencil. But the Board of Medical Examiners requires medical students to take its certification exams online, and faculty wanted to help students prepare for them using online tests. High-resolution images and graphics have replaced the low-quality photocopies and slides that students previously viewed together during tests. Now students can move at their own pace, viewing clear images of cells, abnormalities, and processes that they must identify. A portable computer lab utilizes rented laptops dedicated strictly to online testing with the protection of Questionmark Secure.

Case study
San Francisco Bay Area Rapid Transit District (BART)
The San Francisco Bay Area Rapid Transit District (BART) rail system has long required job candidates to take tests before they are hired. For many years, test questions were handwritten on 3 x 7 cards. BART now uses Questionmark Perception to screen potential hires, most of them applying for jobs in electronics. Pre-employment tests pose multiple-choice questions about such electrical concepts as Ohm's Law, circuits, currents, voltage, and other knowledge. Because the candidates are often very tech-savvy, Questionmark Secure prevents instant-messaging, Web surfing and other tasks. The software locks the desktop so that none of these applications can be used during the test.

Case study
Louisiana State Universitys Health Sciences Center
At Louisiana State Universitys Health Sciences Center, tests for pathology students incorporate images of specimen slides, tissues, and other similar visual material. In the days of paper-based tests, Individual test booklets had to be duplicated and then shredded after each exam, a costly process. When the U.S. Medical Licensing Exam or USMLE began using computerized testing, LSU followed suit. LSUs medical school had already set the groundwork for using computerized testing by requiring all students to own laptops that run the same operating system and use the same software. In addition to the static images that have always been an important part of pathology testing, online questions delivered from a centralized server now also incorporate video clips and flash

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 13 of 24

animations. Every student gets the same exam, but questions are randomized to produce individualized tests. Even the choices are randomized, making the tests much more secure and reducing the temptation to cheat.

Case study
Security for assessments tied to employee promotions
A California-based financial services firm tests loan trains its underwriters, account executives, account managers and quality control auditors in keeping with Federal Deposit Insurance Corporation (FDIC) regulations and investors guidelines. The company uses Questionmark Perception to create and administer assessments for required procedures in underwriting, fraud detection, regulatory management, and funding, among others. Some of the companys assessments t are tied to promotions, and therefore require particularly tight security, so they are delivered using Questionmark Secure which locks down computers and makes it impossible for test takers to access the Internet, print the test or use other software programs during test-taking. Creating the physical right environment: The ideal assessment environment would include the following characteristics: Comfortable, clean and smoke-free Adequate lighting, ventilation, comfortable seating and work surfaces for the candidate during the assessment session Free from external distractions No opportunity to use equipment such as printers, fax machines, copiers or phones while taking an assessment Provide a viewing window and/or video surveillance system If a monitor is present, they should have an unobstructed view of each candidate within the assessment room. IT equipment should be consistent and robust and comply with the local accessibility rules and regulations. Deploying software to deliver secure assessments: Although many issues can be addressed within a physical environment, new software features can also be used to ensure that assessments are delivered in a secure way. The following features are to be found in Questionmark Perception:

Authoring secure assessments: There are many safeguards that can be taken during the

assessment authoring process to help protect the security of the assessment. One of the most popular is to shuffle the order of the choices. The questions can also be delivered in a random order themselves. Both of these features help prevent cribbing when users are sitting in nonscreened testing centers.

Encrypted communications: With so many assessments being delivered via the Internet or an
intranet, it is important that communications between the browser and server are encrypted. This means that if someone sniffs the network they would not be able to see the information sent from server to browser and vice versa. Perception deploys Secure Sockets Layer (SSL), a protocol

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 14 of 24

that allows the browser and web server to encrypt their communication. This means that anyone intercepting the communication would not be able to read it.

Scheduling assessments: Using Perception enables users to set start times of assessments,
the length of assessments and the number of times that assessments may be taken. It also enables user names to be specified and password protects the assessments.

Monitoring assessments: Perception also includes a monitor option. A monitored assessment

cannot be started by the participant until a proctor or invigilator has logged on after the candidate. The monitor can also be limited to a range of IP addresses to ensure that a certain physical location is used to administer the assessment.

Secure browsers: One of the key recommendations is to lock down computers that are being

used in assessments to keep users from accessing other applications and websites while engaged within a medium- or high-stakes assessment. This lockdown is usually performed via a secure browser, such as Questionmark Secure, and prevents candidates from printing, capturing screens, accidentally exiting the assessment viewing source, task switching, etc. The secure browser must be effectively authenticated by the server to prevent spoofing by technically competent cheats. Having considered the alternatives for delivering assessments, it is now worth considering the assessment software, tools and environments that you might need to create, deploy and report on your own secure assessments.

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 15 of 24

4. Tips to help prevent cheating and ensure assessment security Ensuring the security of assessment results is crucial to ensuring the validity of scores. This section provides some tips regarding how to prevent cheating and help ensure that assessment security is maintained. These can be used in conjunction with the technologies outlined in section 6.

Screening tests
A small pre-screening can be administered to prevent people from taking an assessment for which they are not yet prepared. This could contain a few questions about the course they have taken. If the participant cant answer a certain number of these questions correctly they will not be allowed to see the remainder of the assessment. This can be achieved using jump blocks within Perception.

Candidate agreements
Candidate agreements or examination honor codes are codes of conduct that a participant must agree to before they start an assessment. Candidate agreements generally are phrased in a personal manner; the participant agrees by clicking on an OK or Yes button to the code of conduct for the exam. An example of a simple candidate agreement may be: I agree to answer the questions on this assessment without obtaining assistance from another person or via electronic means. I agree to not share my answers with anyone during or after the exam. I further agree to not memorize or otherwise steal the intellectual property contained in this exam. I accept that if any of these conditions are violated my exam results will be set to a zero, I will not be able to retake the exam for a period of 10 years, and I may be charged with a crime under regional laws. Topics that may be covered in a candidate agreement: The test vendor will have the option to terminate the assessment if suspicious behavior is detected The candidate must abide by the rules of the test center, organization, or program The candidate will not provide false ID or false papers The candidate cannot take the test on behalf of someone else The candidate will not engage in cheating in any form The candidate will not assisting others to cheat The candidate will not assist others to cheat by disclosing information about the assessment The candidate will not using aid that are not allowed The candidate will not solicit someone else to take the test The candidate will not cause a disturbance in the testing center The candidate will not tampering with the test center in any way The candidate will not share information about the assessment content they saw (nondisclosure agreement)

Limiting content exposure/leakage

In order to limit the amount of question content being shown to a participant at any given time, consider using question-by-question templates. These present questions one at a time to participants so that exam content is not completely exposed on screen. Participants who may intend to take

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 16 of 24

pictures of the exam content or otherwise steal intellectual property will not be able to do so all at once.

Screening participants who achieve perfect scores

Many organizations will automatically investigate participants who achieve perfect scores on an assessment, since obtaining perfects scores is generally a rare event. They may have a perfect score because answer keys have been compromised or because they were aided in answering the questions. The Questionmark Score List Report provides a fast and easy way to identify participants who obtain 100% on their assessments. An organization can then conduct an investigation of these participants to ensure that no suspicious behavior had occurred (e.g., interviewing the exam proctor).

Verifying expected IP addresses

If assessments are to be taken from a specific location, often the IP address of the computer in that location will be known. Verifying that all participants who took an assessment did indeed take the assessment from the expected location(s) is a useful way to screen whether participants somehow took an assessment from an unauthorized location. The Questionmark Score List Report provides the IP address for all participants taking an assessment and is a useful tool for screening expected IP addresses.

Reviewing time to finish information

The overall time it takes for a participant to complete an assessment can be a useful way to screen for suspicious behavior. If a participant takes a very short period of time to complete their assessment yet achieve a high score this could be an indication that they cheated in some way,

Using Trojan horse or stealth items

Trojan horse or stealth items can be used to help detect whether a participant has memorized the answer key. Stealth items are inserted into an assessment and look just like the other questions, but they are purposely keyed incorrectly. The actual correct answer is marked incorrectly and one of the distracters is marked as the correct answer. These items are generally included as non-scored items on the assessment. If a participant is simply memorizing content and keyed correct answers they will likely choose alternatives that they have memorized. Participants with overall reasonable assessment scores who got the stealth items correct might have memorized the answer key

Post information that cheater prevention tactics are used

An effective tool in preventing cheating is to inform participants that reviews are regularly conducted to identify cheaters. Participants should not know details about what sort of reviews are conducted (for example the tips listed in this section) but should be told that cheater-detection tactics are regularly employed. This can help to deter the low-motivation cheaters.

Proper seating arrangements for participants

Implementing a seating plan where participants are adequately spaced with limited ability to see another participants screen/paper is an import strategy for enhancing test. The proctor should be aware of cheating techniques such as the flying V seating arrangement where the giver at the

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 17 of 24

point of the V feeds information to a number of receivers behind them. The givers and receivers can communicate in a number of ways, using sign language, dropping notes on the floor, etc. Example of the flying V answer copying formation (Cizek, 1999):

This and other techniques for cheating are eloquently documented in Dr. Gregory Cizeks book Cheating on Tests: How to Do it, Detect it, and Prevent it, which is an excellent resource on this topic.

Using unique make-up exams

Many organizations offer make-up exams for participants who were sick or had legitimate excuses for not being able to take an assessment at the scheduled date and time. If an organization uses the same exam that was administered at the scheduled date and time for their make-up exam, they open themselves to risks of the exam form being compromised. Sometimes the make-up exams are not administered in the same strict proctored environment as the scheduled exam, allowing participants the opportunity to cheat or steal content. Having another test form available specifically for make-up exams can lessen the risks of cheating and exposure for the actual large-scale exam.

Using more constructed response questions

Constructed response questions, like essay or short answer questions, provide less opportunity for participants to cheat because they require them to produce unique answers to questions. There is no answer key to steal, and participants who copied other peoples constructed response answers are easily identified via a side-by-side comparison of answers.

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 18 of 24

5. Tips to help ensure the security of intellectual property

Create and administer multiple test forms

Rather than having only one form of the assessment being administered, delivering multiple forms of the same exam can help limit item exposure. If one exam form is breached, the other exam forms can stay in circulation. This method also allows for the possibility of interspersing large-scale integrated beta test questions within the forms to collect psychometric information on newly developed questions. For low-stakes exams, Perception allows for the random selection of questions from topics for each participant, lessening the chance that participants will see the same questions and thereby limiting item exposure (depending on the size of the repository).

Restrict and control administration of beta test items

Beta testing questions is an important part of high-stakes assessment, ensuring the psychometric quality of questions before they appear on actual assessments. However, it is vital that a well conceptualized beta test model is in effect which limits the exposure of newly developed questions to participants. Also, the beta test questions must be administered in secure environments, in similar conditions to the actual exam. This prevents the exposure of new questions before they appear on an actual assessment. Some rejected beta test questions could be considered to be part of exam prep materials, discussed later in this section.

Update exam forms periodically

Letting exam forms become stale (not cycling out old question and cycling in new questions) can over-expose questions to participants, increasing the likelihood of IP theft. Periodically updating exam forms (e.g., annually) can help limit the exposure of questions. An organization could consider retiring old exam forms and turning them into exam prep materials that can be sold to participants. In this way, participants could periodically expect new practice questions. The sales of the exam prep material or practice tests can help offset increased question development costs required to provide evergreen exam forms.

Produce exam prep materials

Organizations should consider making exam prep materials available to participants before an assessment. This will help reduce the demand for participants to try and obtain exam questions via illegal means as they will have access to the type of questions that will be asked on the actual assessment.

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 19 of 24

6. Questionmark Technologies for Deploying Assessments Safely and Securely Questionmark provides technologies to help you author, deliver and report on assessments safely and securely. Questionmark also provides services to help you deploy its technologies successfully. However, Questionmark does not provide testing centers (bricks and mortar) or monitoring services (invigilators or proctors). The following chart details certain features that Questionmark technologies provide to help you deploy your assessments safely and securely. Type of assessment Exams Questionmark Perception Features to use o Questionmark Secure o Keeps assessment content secure o Prevents printing, capturing content, task switching, etc. o Secure Sockets o Prevents network pirates intercepting content o Monitors o To authenticate candidate o To monitor the candidate o Monitor from an specific range of IP addresses o Ensures the correct room/computer is used o Assessment only available during a limited time window o Reduces opportunities for people to relay information about the test o Shuffle choices o Prevents candidates seeing which choice another candidate selected o Randomized questions o Candidates get different questions, which reduces the possibility that they will see the answers from another candidate o Jumps can stop tests for poor candidates o Prevents exposing content to candidates who obviously wont pass o Restrict delivery to specific test centers o Prevents participants from taking exams in unauthorized locations o Define assessment access periods and time periods (e.g., number of days) before a re-attempt o Prevents participants from taking an assessment more often than they are supposed to o Questionmark Secure o Keep assessment content secure o Prevents printing, capturing content, task switching, etc. o Secure Sockets o Prevents network pirates from intercepting content o Assessment only available during a limited time window o Reduces opportunities for people to relay information about the
Copyright Questionmark 2003, 2009 Page 20 of 24

Quiz

3.29.200910:18:24

test and short-circuit the cognitive process o Randomized questions o Candidates get different questions to practice with o Provide repeated search and retrieval practice Survey o Respondents details (name, IP address) can be erased to: o Maintain anonymity o Eliminate ballot rigging o Questionmark Secure o Keeps assessment content secure o Prevents printing, capturing content, task switching, etc. o Secure Sockets o Prevent network pirates intercepting content o Monitors o To authenticate candidate o To monitor the candidate o Monitor from an specific range of IP addresses o Ensures the correct room/computer is used o Assessment only available during a limited time window o Reduces opportunities for people to relay information about the test o Shuffle choices o Prevents candidates seeing which choice another candidate selected o Randomized questions o Candidates get different questions, which reduces the possibility that they will see the answers from another candidate o Jumps can direct students to helpful content o Prevents exposure of content that they arent ready for

Test

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 21 of 24

Appendix:

Detailed examples of assessment types

Surveys Level 1 Surveys, Course evaluations Needs Analysis Surveys Job Task Analysis Surveys Employee Attitude/ Opinion Surveys Customer/Partner Satisfaction Surveys Information and Opinion Surveys 360 (Level 3) Surveys 360 degree survey (employee appraisals) 360 learner peer review assessments Main purpose of assessment is to gather information or opinion These are specific evaluations, given after a course, to get the students feedback on the learning activity. Level 1 Surveys are also called course evaluations or smile sheets. Level 1 Surveys typically ask for the learners evaluation of the course content, course materials, the instructor, the information delivery, etc. These surveys are given to a group to explicitly determine what areas they want to learn more about, or where their knowledge is weak, with the purpose of providing learning offerings to meet those needs. These surveys are given to a group of people doing a job to determine which tasks they perform, how regularly they perform them and the significance of the task to their role. These surveys are used within an organization to collect input from employees about their feelings on one or more topics. These surveys are used by an organization with other groups than their employees; it might include customers, prospects, partners, and others outside the organization. Classic market research is an example of this category. These surveys are used to gather information and opinions that do not fall into the categories above. For example surveying conformance with maturity models or six sigma. These are surveys designed to measure whether behavior has changed on the job, usually delivered by a 360/180 survey to colleagues. Often data is aggregated to identify the effectiveness of learning activities Information gathered from someone's self-assessment and assessment by peers (and where relevant superiors and subordinates) to obtain input on how the person can improve. Organized by HR as part of a performance appraisal Information or judgments gathered from peers in a learning context to aid learning and provide feedback.

Formative assessments Quizzes during Learning Practice tests

Used to strengthen memory recall by practice and to correct misconceptions and to promote confidence in ones knowledge Quizzes are typically used at the end or in intermediate points of learning activities. Usually do not store results (or if do so just to see if questions should be promoted or for other very low stakes use). Practice versions of higher stakes exams, made available or sold for practice purposes

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 22 of 24

Diagnostic assessments Pre Tests

Placement tests Self-diagnostic tools Personality assessments

Assesses knowledge, skills, behaviors, understandings and/or attitudes to determine gaps and potentially provide a diagnosis and prescription of learning and/or other activities A diagnostic assessment before a specific learning activity. Used to create intrigue, to set a benchmark for comparison with a post course test, as a pre-requisite or route to an appropriate learning activity, and to provide instructors and mentors information on the student's abilities. A diagnostic assessment used to measure place someone in the right course or learning activity. Has a summative element but main purpose is to diagnose and prescribe. Used to route to the correct learning activity. Assessment which asks non-judgmental questions and gives feedback to the participant to provide recommendations for products, services and/or learning activities. An example might be a financial survey which asks for information and recommends financial products. Assessment which analyzes personality traits in order to predict behaviors includes MyersBriggs. Sometimes gives information directly to participant and sometimes requires an expert to review and interpret. Main purpose of assessment is to measure or certify knowledge, skills and aptitudes (KSAs) Post Course Tests (aka Post Tests) are typically given to identify the learner's KSAs and may contribute to passing the course. Sometimes data from these tests are used to indicate how much a student or group of students have advanced their knowledge, skills and attitudes by comparison to the pre-test. Assessments given during or at the end of a prolonged course of study which are used to define the grade or passing of an extended learning activity, for instance university courses. Exams are normally available only to students who have registered with an organization and studying with organization, often in a cohort. Most commonly academic, e.g. university exams Exams open to a closed community, for instance employees or partners set by an organization for its own purposes. Examples include regulatory compliance exams and partner verification exams. Exams open to anyone who qualifies that are used to certify or score KSAs. Common examples are IT certifications (e.g. Microsoft, Cisco) or general academic entry exams (e.g. TOEFL, SAT) and IQ tests. Authority comes from an organization not a legal source, but passing can be a passport to greater pay or rewards and so the stakes are often high. Exams which must be passed to meet a legal requirement or a quasi-legal requirement, for example driving tests, medical licensing exams. These are very high stakes and need to be legally defensible. Asks questions of applicants about KSAs and experience to gate applicants. Used to determine whether applicants meet basic requirements and can be considered for the next stage of a recruitment process. In some cases close to a survey. Clinical psychological assessments which ask questions and then determine a psychological profile. These are typically validated against a norm population. Tests given in pre-employment to measure KSAs to see if the applicant is suitable for recruitment.

Summative tests Post Course Tests Exams during study

Internal exams Certification exams Licensing exams Preemployment screening Psychological assessments Preemployment tests

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 23 of 24

About Questionmark: Questionmark provides technologies and services that enable organizations to measure knowledge, skills and attitudes securely for certification, regulatory compliance and improved learning outcomes. Questionmark solutions enable reliable, valid and defensible assessments by empowering subject matter experts through collaborative authoring, accommodating participant needs with blended and multilingual delivery and informing stakeholders through timely reporting and meaningful analytics. The Questionmark Perception assessment management system, available as a licensed, hosted or subscription-based solution, enables organizations to create, administer and report on surveys, quizzes, tests and exams. Complete details are available at http://www.questionmark.com

Questionmark 535 Connecticut Avenue Suite 100 Norwalk, CT 06854 Tel: (800) 863-3950 (203) 425-2400 Fax: (800) 339-3944 info@questionmark.com

Questionmark 4th Floor, Hill House Highgate Hill London N19 5NA United Kingdom Tel: +44 (0)20 7263 7575 Fax: +44 (0)20 7263 7555 info@questionmark.co.uk

Questionmark Parc dActivits Economiques Avenue Lon Champagne, 2 B 1480 Tubize (Saintes) Belgium Tel: +32 (0)2 398 02 01 Fax: +32 (0)2 398 02 00 info@questionmark.be

Legal note This document is copyright Questionmark Corporation (Questionmark) 2003 and 2009. Although Questionmark has used all reasonable care in writing this document, Questionmark makes no representations about the suitability of the information contained in this and related documents for any purpose. The document may include technical inaccuracies or typographical errors, and changes may be periodically made to the document or to the software referenced. This document is provided as is without warranty of any kind. See your Perception support contract for further information. Company and product names are trademarks of their respective owners. Mention of these companies in this document does not imply any warranty by these companies or approval by them of this guide or its recommendations.

3.29.200910:18:24

Copyright Questionmark 2003, 2009

Page 24 of 24