See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/299388692

Program Effectiveness Survey

Working Paper · March 2016

DOI: 10.13140/RG.2.1.1153.9602

CITATIONS READS
0 4,194

1 author:

Robert Laukaitis
Headquarters Air Force Reserve Command
11 PUBLICATIONS   0 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Program effectiveness View project

All content following this page was uploaded by Robert Laukaitis on 23 March 2016.

The user has requested enhancement of the downloaded file.

Running head: PROGRAM EFFECTIVENESS SURVEY 1

Program Effectiveness Survey

Robert J. Laukaitis, Jr.

Doctor of Philosophy Program – Industrial/Organizational Psychology

PSY7660 – Survey Construction and Administration

Winter Quarter, 2016

11 January 2016 – 18 March 2016

Telephone: 478-394-2405

Email: rlaukaitis@capellauniversity.edu

Instructor: Dr. Daisuke Akiba

PROGRAM EFFECTIVENESS SURVEY 2

Abstract

Common definition of program effectiveness is essential for good stewardship of federal dollars.

This research attempted to aid the organization in creating a foundation from which a shared

understanding of program effectiveness could be built. An instrument was designed to act as

indicator of the construct of decisions and actions that contributed to program effectiveness. The

survey was administered to 50 participants of an estimated 175-person population. 28.5% of

those surveyed responded. Six questions were posed to participants. A four-option scale was

used to begin understanding the construct. Reliability assessment returned a negative Cronbach’s

alpha (-0.397). This indicated that a negative value was returned in the covariance of the scoring

scale. The results of the instrument validity and reliability treatment concluded that one survey

item contributed significantly to negative alpha. Further development of the instrument was

indicated.

Keywords: program effectiveness, survey reliability,survey validity, decision-making

construct
PROGRAM EFFECTIVENESS SURVEY 3

Table of Contents
Abstract ........................................................................................................................................... 2

Introduction ..................................................................................................................................... 5

Needs Analysis................................................................................................................................ 5

Psychological Construct............................................................................................................ 5

Purpose...................................................................................................................................... 6

Population ................................................................................................................................. 7

Sampling Frame ........................................................................................................................ 7

Sampling Procedure .................................................................................................................. 8

Ethical Principles and Standards ............................................................................................... 8

Determining the Type of Survey..................................................................................................... 9

Scoping the Survey ................................................................................................................... 9

Designing Survey Questions ......................................................................................................... 10

Survey Question Rationale ..................................................................................................... 11

Advantages and Disadvantages of Proposed Questions ......................................................... 12

Strategies that Minimized Disadvantages of Survey Questions ............................................. 12

Field Testing ................................................................................................................................. 13

Survey Test Group Feedback .................................................................................................. 13

Construct Validity of the Survey ............................................................................................ 13

Strengths and Concerns of Validity ........................................................................................ 14

Analyzing Reliability .................................................................................................................... 14

PROGRAM EFFECTIVENESS SURVEY 4

Survey Construct and Scaling ................................................................................................. 15

Reliability Summary Statistics................................................................................................ 15

Cronbach’s Alpha ................................................................................................................... 16

Survey Item Analysis .............................................................................................................. 16

Ethical Standards and Guidelines Related to the Survey .............................................................. 19

Higher Headquarters and Agency Guidance........................................................................... 19

APA Guidance ........................................................................................................................ 20

Boundaries and Protections..................................................................................................... 21

Conclusions ................................................................................................................................... 21

Needs Analysis and Psychological Construct ......................................................................... 21

Survey Approach .................................................................................................................... 22

Construct Validity Concerns ................................................................................................... 22

Reliability Conclusions ........................................................................................................... 23

Ethical Concerns ..................................................................................................................... 23

Summary ................................................................................................................................. 23

Appendix A: Program Effectiveness Survey Questions ............................................................... 29

Appendix B: Program Effectiveness Survey Dataset ................................................................... 30

PROGRAM EFFECTIVENESS SURVEY 5

Introduction

In order to determine the most effective use of federal resources, government

organizations must determine and defend program effectiveness in order to advocate for funding.

As a result, efforts to determine program effectiveness have grown over the past decade. Some

federal agencies have seen their organizations mature and learn, while others have struggled with

this issue. In support of one government agency’s efforts to determine the effectiveness of

assigned programs, a detailed analysis must be performed. This effort requires the creation of a

baseline definition of those factors the organization believes to be representative of program

effectiveness.

Needs Analysis

Organization program managers are attempting to strive for evidence-based efficiency

and effectiveness (Franceschini & Turina, 2013). To fulfill a commitment to help the

organization assess and the effectiveness of assigned programs, the organization endeavored to

create a reliable, valid and accurate assessment of program decisions and outcomes (Asan &

Ayhan, 2013). Given the lack of a standard definition of program effectiveness within the

organization, a survey instrument was developed as a method by which program managers could

build and share an organization definition of program effectiveness. Lacking this definition could

result in resource reallocation without a clear understanding of future mission impacts or of

second- or third-order effects in other associated programs.

Psychological Construct

An initial assessment was conducted via telephone and personal interviews to determine

the current state of a shared definition of program effectiveness. A randomized convenience

sample of current program managers in the organization was conducted in late 2015. Of the 15

people contacted, it was determined that a common definition of program effectiveness did not
PROGRAM EFFECTIVENESS SURVEY 6

exist. The underlying assumptions of effective programs served as a filter through which the

decision-making process construct could be clarified.

Each federal organization is assigned programs in support of mission requirements or

desired outcomes. Funds are allocated as part of the yearly planning, programming and

budgeting process. Programs are rarely funded to 100% of the actual requirement; program

managers must balance the lowest level of acceptable risk while still executing the most effective

programs possible. The series of decisions that accomplish an optimal reduced risk/optimal

effectiveness balance create a dynamic construct that is not clearly understood by the

organization. Experiences are typically shared as lessons learned between program managers.

Although this approach has supported organization success thus far, it does not support a

repeatable, comparative method by which program effectiveness can be examined.

Purpose

Programs assigned to the organization are a collection of people, facilities and equipment

needed to perform given missions. The time and resources it can take to produce, distribute,

collect and analyze an instrument must be justifiable. Currently, the organization has four

program managers that oversee a variety of programs totaling over $4 billion in federal funds.

The complexity of relationship dynamics makes personal interviews difficult to accomplish

given travel budget constraints yet increased the scope of data collection beyond the

organization.

Driving factors of the survey began with a varied and nebulous perception of program

effectiveness provided to auditors during a 2015 review of organization functions. Although a

definition of ‘effectiveness’ exists (Merriam-Webster, 2016), organization leadership admitted

that a common approach to program effectiveness would improve executive decision-making.

Thus, the organization’s leader and his leadership team had a need for a successful research
PROGRAM EFFECTIVENESS SURVEY 7

effort. Lastly, there was no existing indicator for the construct. An instrument had to be created

to meet this requirement. Past programmatic management behaviors might be indicated by those

factors discovered and developed because of survey data. Program management behaviors could

be categorized as contributing factors, detracting factors (Houger, 2015), or external factors (e.g.,

political, economic, etc.) having the most significant impact on effective use of resources.

Population

The target population for this instrument included program managers and program

element liaisons. The target population was approximately 25 individuals within the organization

and 150 people spanning the enterprise of a 175-person population. The demographics for this

population were primarily male, over age 50, having served at least 17 years as commissioned

officers. All members of the population had at least a post-secondary education, with most

having advanced degrees and senior-level military professional developmental education. The

demographics are important as they describe the culture within which the survey instrument was

developed, deployed, collected and analyzed. In addition, the demographics represent potential

bias that could have been introduced. By identifying the bias, skewed results could be addressed

in the interpretation process.

Sampling Frame

As the population was relatively small, a survey of the population was considered ideal,

practical and convenient. A higher-than-normal response rate was also anticipated due to

population size, the opportunity to contact participants personally and the ability to follow up

with participants after the instrument was distributed (Fowler, 2014). An advantage for selecting

a 100% sample size was that the effects of error (e) were theoretically reduced. Challenges to

this sampling approach included a risk that Reservists tend to be in a higher age and rank bracket

than their active duty counterparts (Preiss, 2013), resulting in skewed or biased data collected.
PROGRAM EFFECTIVENESS SURVEY 8

Sampling Procedure

In order to obtain results reliable at the α=.05, with a margin of error set at 5%, a sample

of approximately 121 individuals would need to be sampled (Warner, 2013). Increasing the

margin of error to 10% would require a sample of 63 individuals. In order to understand the

construct, it was determined that the largest sample size possible could provide the most

information about the construct under examination.

Sampling electronically proved the most effective at reaching the target population.

According to McPeake, Bateson and O'Neill (2013), response rates for electronic surveys tended

to be slightly lower than response rates of surveys delivered via postal mail are. The authors

identified concerns that the population being surveyed might not have reliable access to the

internet, a potential lack of familiarity or trust with participation in electronic surveys or a

concern of confidentiality of data collected via electronic methods. These concerns were

mitigated via personal contact and a culture excepting of electronic surveys. Organization

members have participated in electronic surveys for more than a decade. Areas for concern in

this context included Freedom of Information Act (FOIA) and Privacy Act (1974) issues, labor

relation issues, legal and security reviews and Internal Review Board (IRB) evaluation from an

accredited organization.

These risks were also mitigated by adhering to the guidance in organization operating

instructions, DoDI 1100.13 - Surveys of DoD Personnel (U.S. Department of Defense, 2015) and

Office of Personnel (OPM) employee survey guidance (2016). In addition, standards outlined by

the American Psychological Association (APA) in the Ethical Principles of Psychologists and

Code of Conduct (2010) were also considered.

Ethical Principles and Standards

PROGRAM EFFECTIVENESS SURVEY 9

Within the organization, ethical standards are generally based on legal and mission

requirements, planning, programming guidance, and organization directives. Standards

maintained by the APA (2010) carried significant weight on the use of this survey and

participant protections. To maintain protection of participants, response data was not stored in an

open environment. Data could not be accessible by other than the survey research personnel.

Once validated, the defining factors of effectiveness would be used to evaluate program

managers and resource allocation proposals. Bias could have been introduced due to differing

attitudes towards program effectiveness (Walsh, et al., 2012). An ethical concern was that

individuals in the target population were not been made aware of the reasons for defining

program effectiveness. Data collected was not intended to serve as the sole source for personnel

decisions. The intent was that program managers with the most effective programs would be

asked to mentor others to improve decision-making skills in support of increased program

effectiveness.

Determining the Type of Survey

Program evaluation includes defining expected outcomes, activities, performance

indicators and actual program results (Gajda & Jewiss, 2004). Gajda and Jewiss (2004) indicated

that in order to evaluate program performance, survey questions needed to indicate qualities of a

given program’s outcome, the activities needed to accomplish the outcome and any indicators of

progress towards a given program’s intended purpose. The survey developed served as an initial

review of the existence of expected program outcomes, activities, performance indicators and

evaluation methods in support of the creation of a definition of ‘program effectiveness’.

Scoping the Survey

A survey instrument must be scoped such that it captures the intended information

relative to the context of its use (Asan & Ayhan, 2013). This translated into a survey being able
PROGRAM EFFECTIVENESS SURVEY 10

to act as an indicator of program outcomes in terms of quantity and quality. The questions

needed to provide indicators applicable to any program managed by the organization. However,

as the definition in this construct was not established, determining program effectiveness was not

supportable. Thus, this survey instrument focused on the decision-making process and associated

indicators of program effectiveness. This survey consisted of six items that were aimed at

determining factors of program effectiveness present in the organization. Fowler (2014)

suggested that survey questions be clear, concise and direct statements of the constructs being

measured. In AFRC, program guidelines or expectations were evident in directives, policies or

other forms of senior leader guidance.

Designing Survey Questions

Grajda and Jewiss (2004) suggested that in order to determine program effectiveness,

questions must be asked to establish baseline program expectations and performance. The

authors suggested that by understanding the desired outcomes of a given program, its goals and

expected accomplishments over time and program performance could be evaluated. Process

indicators could also be created in order to determine how one might arrive at the expected

accomplishments and the associated activities be defined in order create discrete observations of

program performance. Lastly, they suggested that a known indicator exist for the progress

towards desired program outcomes. Thus, each concepts proposed by Grajda and Jewiss (2004)

were converted into affirmative statements measureable via a Likert-type scale.

PROGRAM EFFECTIVENESS SURVEY 11

Open-ended questions* Open-ended questions converted to forced-

choice statements
What are the desired outcomes of a program? Each program I manages has defined desired
outcomes.
What are the goals? Each program I manage has associated goals.
What is the organization trying to accomplish All of my program goals are time-bound.
within the next month/quarter/year(s)?
How will the organization get there? I have action plans developed to help each
program meet desired outcomes.
What activities will enable the organization to Each action plan contains discrete activities
reach desired outcomes? that will enable my programs to reach desired
outcomes.
What will indicate that progress toward the I have indicators established for each program
desired outcomes is occurring? to monitor progress towards desired outcomes.
*Questions adapted as suggested by Gajda and Jewiss (2004, p. 3).
Figure 1: Open- to closed-ended question conversion

An even number of options was offered as the Likert-type scale. This removed a ‘no-

opinion’ option and forced respondents to report agreement or disagreement with each statement

(Fowler, 2014). Final survey conversion and construction are provided in Appendix A: Program

Effectiveness Survey Questions.

Survey Question Rationale

Gajda and Jewiss (2004) proposed that in some cases, the components of what is

considered critical to program evaluation might already exist in an organization. In the case of

the organization’s programs, evidence supporting defined mission objectives of each program

existed. As each program is unique, a common, enterprise-wide descriptor cannot exist. Instead,

each program manager is expected to allocate program resources with expected mission

performance percentages, levels of individual and equipment readiness and other factors driven

by higher headquarters and national security requirements. A program effectiveness survey

tested the degree to which each program manager understood the components of effectiveness

relative to his or her programs. Organization program managers are expected to communicate

with other like organizations maintaining like programs. Cross-functional coordination creates a
PROGRAM EFFECTIVENESS SURVEY 12

community of understanding that could be used to build definitions of program performance and

effectiveness.

Advantages and Disadvantages of Proposed Questions

Future research could build on the existence of factors determined critical to effective

program performance. Supporting documentation for each statement should have been readily

available if the statements were affirmed by participants. An advantage of the proposed questions

was that question phrases were positive assertions that each program manager should have been

able to support. Program resource accountability is maintained through the yearly planning and

programming cycle (i.e., fiscal year to fiscal year). Organization processes also require each

program manager to defend program changes to his or her peers before submitting through the

corporate structure.

A disadvantage of asking questions in this format was that if the respondent did not

understand the question, or did not know the answer to a particular question, he or she was

forced to provide a false response (Bilodeau, 2006; Fowler, 2014). This practice could have led

to skewed survey results, increasing the rate of error and decreasing the reliability and validity of

the instrument (Leedy & Ormrod, 2013; Warner, 2013).

Strategies that Minimized Disadvantages of Survey Questions

In any survey instrument, the possibility exists that questions or terms used will not be

understood by the respondent. Interpretation bias can increase the probability of erroneous data

reported (Gobo & Mauceri, 2014). One strategy used to minimize ambiguity with the distribution

and collection of a forced-choice survey instrument was to provide clear directions at the

beginning of the survey (Fowler, 2014). Gobo and Mauceri (2014) also suggested using words

that have common cultural meanings. This required that definitions be provided as part of the

survey instructions such that respondents developed a standard understanding of question

PROGRAM EFFECTIVENESS SURVEY 13

phrasing and meaning. In addition, the length of the survey did not lend itself to biased, patterned

responses (Morren, Gelissen, & Vermunt, 2012). The six-item survey length helped minimize

disadvantages of forced-choice responses.

Field Testing

Validity is considered evidence that an instrument has a purpose: the accuracy of the

survey instrument (Sullivan, 2011). Serving the designed purpose is based on a relationship

between the how well the answers provided by a participant measure the intended focal

characteristic that would be considered unobservable by other methods (Kaplan & Saccuzzo,

2012; Leedy & Ormrod, 2013). Construct validity was determined as the most appropriate

approach for initial testing of this instrument.

Survey Test Group Feedback

Five test group members provided initial feedback. Overall response was positive. Test

group members reported that the survey length was appreciated and appropriate. Responses to

questions were favorable. Questions were considered understandable by test group members.

Personal interviews conducted with the test group indicated a relatively common understanding

of question terms and phrasing.

Construct Validity of the Survey

Construct validity provides the researcher with a measure of how well a series of

questions in an instrument measure the construct of interest not otherwise observable (Leedy &

Ormrod, 2013). As the construct under review was the decision making process relative to

program effectiveness, the opinions, beliefs, motivation and other behaviors displayed by

program managers was a primary consideration. Understanding foundational elements of the

construct required interviews conducted with participants in the test group. Interviews consisted

of survey item review and clarifying statements as requested by participants. In this case, since
PROGRAM EFFECTIVENESS SURVEY 14

the construct of interest was not directly observable, the evidence supporting the level of validity

was based on what was assumed reported as fact by survey participants (Leedy & Ormrod,

2013). Construct validity appeared relatively high based on statements made by test group

members and their personal experiences when reallocating resources between programs.

Strengths and Concerns of Validity

The method by which instrument validity is determined is limited to the domain for

which the survey was designed. This limits the instruments recommended application generally

to the domain of intended use. Therefore, the validation approach should be supported by

evidence. Even so, validity remains an estimated indicator of a given construct (Sullivan, 2011;

Warner, 2013). During the validation process, a determination must be made as to those survey

items to retain and those to exclude based on item performance (Forrest, Eatough, & Shevlin,

2005). In this field test, items that failed to measure the construct adequately would have been

discarded. However, test group participants felt that the questions reflected their personal

experiences and expertise with regard to program effectiveness.

Analyzing Reliability

In order to determine the reliability of a survey instrument, an assessment of the

instruments ability to provide evidence of a given phenomenon consistently must be made.

Cronbach’s alpha (α) represented the degree to which survey items and score scale behave

consistently (i.e., reliably) (Warner, 2013) with regard to the measurement of a given construct.

A four-point Likert-type (ordinal) scale was used to determine the degree to which program

managers had specific elements of program oversight in place. Thus, the presence of these

indicators was considered the first step in determining program effectiveness.

PROGRAM EFFECTIVENESS SURVEY 15

Survey Construct and Scaling

The instrument consisted of six questions (items) based on the components of program

effectiveness (e.g., defining expected outcomes, activities, performance indicators and actual

program results (Gajda & Jewiss, 2004)). 175 surveys were distributed with 28.6% total

responses (50 participants) returned within the time limit specified in the instructions. All

participants responded to all six items. Responses were coded into an IBM Statistical Package

for the Social Sciences (SPSS) (IBM, 2016) dataset for storage and testing. Respondent personal

identifying information was omitted from electronic communication methods. Coded responses

did not identify participants, demographics or specific programs.

Survey analysis was completed using the reliability function resident in IBM SPSS (IBM,

2016). Accepted reliability testing statistics (i.e., summary analysis, Cronbach’s alpha (α)

statistics and item-level statistics) are provided in Tables 1 through 7 as produced by reliability

analysis. The data represented by the item variables is ordinal data and consideration should be

given to the use of other, nonparametric statistics (Warner, 2013).

Reliability Summary Statistics

Table 1 provides a summary of the number of surveys (i.e., cases) returned for inclusion

in this research effort. In this round of survey distribution and collection, no items or responses

were deleted for non-response.

Table 1
Case Processing Summary
N %
Cases Valid 50 100.0
a
Excluded 0 0
Total 50 100.0
a. Listwise deletion based on all variables in the procedure.
Table 1
PROGRAM EFFECTIVENESS SURVEY 16

Cronbach’s Alpha

A primary concern with regard to this survey was the negative value of α. Given the

number (N) of questions in the survey was six and there were four response options, the formula

implied that the larger the number of survey items and the greater the number of scoring

possibilities, the more likely the instrument would produce a greater level of α (Nichols, 1999).

A review of raw data did not reveal a coding error. However, Dunn, Baguley and Brunsden

(2014) proposed that α was more a representation of the variability between the score scale and

the construct being measured. Moreover, an underestimation in error (Tiffina & Rolling, 2012)

due to the low number of items and very low variability in score responses could produce -α.

Mathematically, -α is possible if the sum of individual item variances is greater than the variance

of the scale (Kaplan & Saccuzzo, 2012). Another cause could be that the items in the survey did

not accurately measure the intended construct. Understanding of the construct is in development;

another instrument measuring the same construct in the same environment did not exist.

Table 2
Reliability Statistics
Cronbach's Alpha a Cronbach's Alpha Based on Standardized Items a N of Items
-.397 -.480 6
a. Negative value due to a negative average covariance among items.
Table 2

Survey Item Analysis

Tables 3 through 7 represent the performance of survey items and the survey scale itself.

Assuming a normal distribution, items within the survey performed relatively consistently. Mean

scores of 50 responses displayed in Table 3 ranged from 2.5 to 3.06. Standard deviations of

scores ranged from 0.24 to 0.646. Here, reliability can be inferred by a parametric approach

(Dunn, Baguley, & Brunsden, 2014).

PROGRAM EFFECTIVENESS SURVEY 17

Table 3
Item Statistics
Mean Std. Deviation N
Each program I manage has defined desired outcomes. 2.70 .505 50
Each program I manage has associated goals. 3.06 .240 50
All of my program goals are time-bound. 2.70 .580 50
I have action plans developed to help each program meet
2.50 .614 50
desired outcomes.
Each action plan contains discrete activities that will enable my
2.84 .468 50
programs to reach desired outcomes.
I have indicators established for each program to monitor
2.54 .646 50
progress towards desired outcomes.
Table 3

Correlations displayed in Table 4 indicate that questions 1, 3, and 6 shared positive

correlations. The correlation between questions 1 and 3 was significantly high (.801). Of the

remaining item correlations, item 4 represented the most significant negative correlation. The

relative weight of this correlation is indicated in Table 6 as having the largest positive impact on

Cronbach’s α if deleted. This suggested that the survey item performed outside expectations

(Norman, 2010). A strategy to address the performance of the instrument would be to increase

the number of items addressing the same subscale within the construct.

Table 4
Inter-Item Correlation Matrix
Questions Question 1 Question 2 Question 3 Question 4 Question 5 Question 6
Question 1 -
Question 2 -.185 -
Question 3 .801 -.161 -
Question 4 -.559 .069 -.544 -
Question 5 -.207 .087 -.180 -.142 -
Question 6 .382 -.082 .496 -.180 -.451 -
Table 4
PROGRAM EFFECTIVENESS SURVEY 18

Table 5 indicates summary statistics for the survey items used in the instrument. Item means

indicated central tendency with low variance (.042) of survey items.

Table 5
Summary Item Statistics
Maximum/
Mean Minimum Maximum Range Minimum Variance N
Item Means 2.723 2.500 3.060 .560 1.224 .042 6
Inter-Item Correlations -.057 -.559 .801 1.360 -1.433 .138 6
Table 5

Table 6 provides scenarios for consideration of elimination of survey items. Those items with the

highest negative corrected item-total correlations accounted for the negative value of α. As

understanding the construct is being developed, removing an item from the survey at this stage of

research was not considered, however Umbach (2005) and Yan, Kreuter and Tourangeau (2012)

suggested rewording the poorly performing item and readdressing the pretest stage of survey

development.

Table 6
Item-Total Statistics
Scale Mean Scale Corrected Squared
if Item Variance if Item-Total Multiple Cronbach's Alpha if
Deleted Item Deleted Correlation Correlation Item Deleted
Question 1 13.56 .741 .394 .676 -.994a
Question 2 13.24 1.288 .097 .036 -.269a
Question 3 13.56 .578 .481 .719 -1.451a
Question 4 13.76 1.982 -.590 .413 .488
Question 5 13.42 1.677 -.460 .352 .231
Question 6 13.76 .676 .283 .528 -.984a
a. Negative due to negative average covariance among items. Reliability assumptions violated.
Table 6

Table 7 summarizes the mean and variance of the survey scale used. The presence of central

tendency was reinforced. This indicated an appropriate scale proposed for the development of

understanding of the construct (Harmon, Morgan, & Gliner, 1999).

PROGRAM EFFECTIVENESS SURVEY 19

Table 7
Scale Statistics
Mean Variance Std. Deviation N of Items
16.34 1.249 1.118 6
Table 7

Ethical Standards and Guidelines Related to the Survey

Ethical, legal and organizationally accepted methods of survey use are critical to the

integrity of organizational research. Department of Defense Instruction 1100.13 - DoD Surveys

(DoDI1100.13) (2015) prescribed expectations regarding surveys and DoD personnel. The

American Psychological Association (APA) provided the Ethical Principles of Psychologists and

Code of Conduct (2010) as a set of standards that include the ethical administration, collection,

development and use of survey instruments. Lastly, the Office of Personnel Management (OPM)

provided guidance regarding the use of surveys on civilians in federal service (2016). Therefore,

surveys used in the organization must comply with a number of protections of participants and

organization information. Several layers of protection and compliance greatly influence surveys

and organization development efforts in the organization.

Higher Headquarters and Agency Guidance

Guidance provided by the DoD (2015) was built on the same framework as guidance

provided by the APA. Protection of DoD survey participant personal and mission-sensitive

information inform survey review bodies resident within each of the military services. The Air

Force created the Air Force Survey Office to act as the responsible body for review, distribution,

data gathering and interpretation of organizational surveys. The OPM also has an interest in the

use of surveys and tests on the civilian workforce, regardless of grade or labor organization

affiliation (2016). OPM guidance supported similar ethical and protective issues as those issued

by the DoD and Air Force. In addition to guidance, OPM developed and reviews surveys for use
PROGRAM EFFECTIVENESS SURVEY 20

by all federal organizations. This multifaceted survey construct could help support the

development of a successful Industrial/Organizational (I/O) psychology practice within the

AFRC headquarters. Successful I/O practices would include submitting all proposed surveys

through the Air Force Survey Office for review and approval prior to administration.

APA Guidance

Based on the Belmont Report (The National Commission for the Protection of Human

Subjects, 1979) and other reports regarding how prisoners of war, under-education members of a

given population, those not capable of defending themselves, etc., the APA recognized the need

for standards of ethical treatment of human subjects (2010). Several sections in the APA’s

Ethical Principles of Psychologists and Code of Conduct were dedicated to qualifications and

ethical practice of psychological practitioners when using survey instruments.

Standard 2.0 suggested that practitioners be competent in the area in which they practice.

This includes maintaining the competence and capability to administer instruments used for

psychological evaluation and research. Standard 4.0 described expectations regarding the privacy

and confidentiality of human subjects. Standard 7.0 addressed the education and training

component of the ethical practice of psychology. The APA suggested that those using a given

instrument should be trained in the administration and interpretation of the instrument by the

issuing body. Standard 8.0 provided the ethical conduct of research and information published

because of survey administration must ensure beneficence and non-malfeasance to the greatest

degree possible. Section 9.0 addressed assessments directly: the resulting impact could

substantively change the quality of an individual’s life. APA expectations were clear with respect

to the use of survey data, the need for informed consent of participants, transparency in scoring

and interpretation of instrument results, and the security and maintenance of survey instrument

data (2010).
PROGRAM EFFECTIVENESS SURVEY 21

Boundaries and Protections

Individual personal and physical data protection must be considered a part of ethical

survey research. Since the program effectiveness survey was conducted in a secure environment,

physical protections of participants and survey data were accounted for. Future continued

validation of similar efforts would not require the personal identification of participants.

Therefore, there was no reason to link participant information to survey responses. Recognition

of demographic changes is not required to conduct longitudinal studies in programmatic changes

(Fowler, 2014). Documentation of protective efforts was maintained in order to support the

ethical and scientific character of the research effort (Trochim, 2006) and to provide evidence of

appropriate survey development and administration to the Air Force Survey Office. Digital

security controls confirmed that data was only accessible by those in the research function.

Physical security of survey data consisted of a pass card-enabled secure work area, locked

drawers and password protected database access (Hall, Howerton, & Bolin, 2005).

Conclusions

The process of designing, validating and refining the survey instrument contributed to the

understanding of the construct. The decision-making process by which program managers

determined their assigned program risks and resources has been clarified. Clarity has not yet

been reached to support making comparative determinations within and between programs

themselves. However, the point at which the understanding the construct contributed to creating

a working definition of program effectiveness can now be examined.

Needs Analysis and Psychological Construct

It is clear that a need persists for the continued understanding of the construct. The

decision-making process as it related to program effectiveness was scoped as the optimal

construct, but might not have represented the full spectrum of construct variables. Factors
PROGRAM EFFECTIVENESS SURVEY 22

beyond the control of the program manager can significantly affect the outcome of programmatic

choices and the redistribution of resources. For example, changes in legislation can alter the

course of a program with very little notice and at times contrary to organization desires. This can

result in having to rebalance program dollars outside of the systematic approach accepted by the

organization for future fiscal years. Second- and third-order effects emerge, as short-notice

changes require increasing risk in programs considered highly effective. The development of a

program effectiveness approach could support the organization’s ability to react to external

influences while minimizing risks to current programs. Enterprise program (and organization)

effectiveness can be made more stable in current and future years.

Survey Approach

The sample process was relatively successful. As the small population allowed for greater

access to survey participants, the convenience sample used provided adequate data for construct

analysis. The scope of the survey instrument contributed to the effort of understanding the

construct. Survey items could have been expanded, and some items could have been reworded

prior to final distribution of the instrument. However, this was not evident until after the results

of reliability testing. Future efforts should consider changing the survey scale to allow for a

greater span of response options. Survey items will also be expanded based on the feedback

provided by reliability examination. The addition of personal experiences in the form of

qualitative responses could also be captured to provide basis for case study options of the same

construct (Fowler, 2014).

Construct Validity Concerns

Survey questions were designed to gain understanding of the decision making process

with regard to program effectiveness. More items could be added that capture the process used

by the organization to prioritize yearly programming decisions made by the corporate structure.
PROGRAM EFFECTIVENESS SURVEY 23

As this is formalized in organization documentation and guidance, greater understanding of the

construct could be gained by including any elements of standardized organization processes and

their relationship to the construct under investigation. These could then be compared to the

existing program manager subscale in search of any relationships between the corporate process

and the individual decisions and behaviors regarding resource allocation.

Reliability Conclusions

Reliability analysis revealed an issue with one of the survey items. According to the

analysis, had the item been deleted Cronbach’s alpha would have changed to a positive number

and could have reflected a significant level of alpha relative to new research in a construct

without previous comparative values. This was not feasible as the number of items was relatively

low (six survey items). However, should the number of items be expanded to measure the

construct, the option to eliminate poorly performing items might produce more reliable results.

Ethical Concerns

Ethical issues were minimized with regard to the use and interpretation of survey data.

Physical and electronic controls were used to protect participant responses and survey data

integrity. Continued validation and reliability information would also need to be provided in

support of the strength of the instrument. Adhering to APA (2010), OPM (2016) and DoD (2015)

guidance will ensure this research continues to contribute to understanding the decision-making

construct as it relates to organization effectiveness.

Summary

The instrument on its first field test did not perform as expected. The results of

Cronbach’s α being negative could indicate a failure to represent the construct being measured

or that more items should have been present to test the construct. Further development of the

instrument should address this issue. Continued development could yield a more reliable
PROGRAM EFFECTIVENESS SURVEY 24

instrument by which the decision-making process influences AFRC program effectiveness. This

includes adding items to measure each component of the construct and expanding the instrument

for use outside of the organization.

PROGRAM EFFECTIVENESS SURVEY 25

References

American Psychological Association. (2010, 06 01). Ethical principles of psychologists and code

of conduct. Retrieved Mar 04, 2015, from American Psychological Association:

http://www.apa.org/ethics/code/principles.pdf

Asan, Z., & Ayhan, H. Ö. (2013). Sampling frame coverage and domain adjustment procedures

for internet surveys. Quality and Quantity, 47(6), 3031-3042. doi:10.1007/s11135-012-

9701-8

Bilodeau, A. (2006). Non-response error versus measurement error: A dilemma when using mail

questionnaires for election studies. Australian Journal of Political Science, 41(1), 107-

117. doi:10.1080/10361140500507310

Dunn, T., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the

pervasive problem of internal consistency estimation. British Journal Of Psychology,

105(3), 399-412. doi:10.1111/bjop.12046

Forrest, S., Eatough, V., & Shevlin, M. (2005). Measuring adult indirect aggression: The

development and psychometric assessment of the indirect aggression scales. Aggressive

Behavior, 31(1), 84-97. doi:10.1002/ab.20074

Fowler, F. J. (2014). Survey research methods (5th ed.). [VitalSource Bookshelf version]. SAGE.

Retrieved from https://bookshelf.vitalsource.com/books/9781483323596

Franceschini, F., & Turina, E. (2013). Quality improvement and redesign of performance

measurement systems: an application to the academic field. Quality & Quantity, 47(3),

465-483. doi:10.1007/s11135-011-9530-1

Gajda, R., & Jewiss, J. (2004). Thinking about how to evaluate your program? These strategies

will get you started. Practical Assessment, Research & Evaluation, 9(8). Retrieved from

http://PAREonline.net/getvn.asp?v=9&n=8
PROGRAM EFFECTIVENESS SURVEY 26

Gobo, G., & Mauceri, S. (2014). Asking and questioning. In Constructing survey data: An

interactional approach (pp. 73-102). London, GBR: SAGE Publications Ltd.

doi:10.4135/9781446288481.n4

Hall, J. D., Howerton, D. L., & Bolin, A. U. (2005). The use of testing technicians: Critical

issues for professional psychology. International Journal of Testing, 5(4), 357-375.

Harmon, R. J., Morgan, A. G., & Gliner, J. A. (1999). Evaluating the validity of a research study.

Journal of the American Academy of Child & Adolescent Psychiatry, 38(4), 480-485.

doi:10.1097/00004583-199904000-00023

Houger, V. P. (2015). How program design affects program performance and participant

motivation. Performance Improvement, 54(7), 10-18. doi:10.1002/pfi.21492

IBM. (2016). Statistical Package for the Social Sciences (SPSS) software. Retrieved Feb 28,

2016, from www.IBM.com: http://www-01.ibm.com/software/analytics/spss/

Kaplan, R., & Saccuzzo, D. P. (2012). Psychological testing: Principles, applications, and issues

(8th ed.). [VitalSource Bookshelf version]. Cengage Learning. Retrieved from

http://online.vitalsource.com/books/9781285529141

Leedy, P. D., & Ormrod, J. E. (2013). Practical research: planning and design (10th ed.).

Boston, MA: Pearson. Retrieved from

http://online.vitalsource.com/books/9781256815457

McPeake, J., Bateson, M., & O'Neill, A. (2013). Electronic surveys: how to maximise success.

Nurse Researcher, 21(3), 24-26.

Merriam-Webster. (2016). Effectiveness. Retrieved Jan 16, 2016, from Merriam-Webster online

dictionary: http://www.merriam-webster.com/dictionary/effectiveness
PROGRAM EFFECTIVENESS SURVEY 27

Morren, M., Gelissen, J., & Vermunt, J. K. (2012). Response strategies and response styles in

cross-cultural surveys. Cross-Cultural Research, 46(3), 255-279.

doi:10.1177/1069397112440939

Nichols, D. P. (1999). My coefficient a is negative! Retrieved from www.ats.ucla.edu/stats/spss:

http://www.ats.ucla.edu/stat/spss/library/negalpha.htm

Norman, G. (2010). Likert scales, levels of measurement and the ‘‘laws’’ of statistics. Advances

in Health Science Education, 15(5), 625-632. doi:10.1007/s10459-010-9222-y

Office of Personnel Management. (2016). Data, Analysis & Documentation: Employee surveys.

Retrieved Mar 14, 2016, from OPM.gov: https://www.opm.gov/policy-data-

oversight/data-analysis-documentation/employee-surveys/

Preiss, R. (2013). Briefing to National Guard Association of the United States Legislative

Workshop. Reserve Forces Policy Board, Washington, D.C.

The American Psychological Association. (2010, 06 01). Ethical Principles of Psychologists and

Code of Conduct. Retrieved from The American Psychological Association:

http://www.apa.org/ethics/code/index.aspx

The National Commission for the Protection of Human Subjects. (1979, Apr 18). The Belmont

Report. Retrieved Sept 03, 2013, from U.S. Department of Health & Human Services:

http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.html

The Privacy Act of 1974. (1974). 5 U.S.C. § 552a. Retrieved from

http://www.justice.gov/opcl/privstat.htm

Tiffina, P. A., & Rolling, K. (2012). Structure of the health of the nation outcome scales for

children and adolescents: An ordinal factor analysis of clinician ratings of a sample of

young people referred to community mental health services. 197(1-2), 154-162.

doi:10.1016/j.psychres.2012.02.007
PROGRAM EFFECTIVENESS SURVEY 28

Trochim, W. M. (2006). Research methods knowledge base. Retrieved from

socialresearchmethods.net: http://www.socialresearchmethods.net/kb/index.php

U.S. Department of Defense. (2015). Department of Defense Instruction 1100.13 - DoD Surveys.

Washington, DC.

Umbach, P. D. (2005). Getting back to the basics of survey research. New Directions For

Institutional Research, 2005(127), 91-100.

Walsh, M. C., Trentham-Dietz, A., Gangnon, R. E., Nieto, F. J., Newcomb, P. A., & Palta, M.

(2012). Selection bias in population-based cancer case-control studies due to incomplete

sampling frame coverage. Cancer Epidemiology, Biomarkers & Prevention, 21(6), 881-

886. doi:10.1158/1055-9965.EPI-11-1066

Warner, R. M. (2013). Applied statistics: From bivariate through multivariate techniques (2nd

ed.). Thousand Oaks, CA: Sage. Retrieved from

http://online.vitalsource.com/books/9781452268705

Yan, T., Kreuter, F., & Tourangeau, R. (2012). Evaluating survey questions: A comparison of

methods. Journal of Official Statistics, 28(4), 503-529.

PROGRAM EFFECTIVENESS SURVEY 29

Appendix A: Program Effectiveness Survey Questions

Program Effectiveness Survey Questions Scale

Strongly Disagree Agree Strongly
Disagree Agree
Each program I manages has defined desired 1 2 3 4
outcomes.
Each program I manage has associated goals. 1 2 3 4
All of my program goals are time-bound. 1 2 3 4
I have action plans developed to help each 1 2 3 4
program meet desired outcomes.
Each action plan contains discrete activities 1 2 3 4
that will enable my programs to reach desired
outcomes.
I have indicators established for each program 1 2 3 4
to monitor progress towards desired outcomes.
*Questions adapted from those suggested by Gajda and Jewiss (2004, p. 3)
PROGRAM EFFECTIVENESS SURVEY 30

Appendix B: Program Effectiveness Survey Dataset

Cases Question 1 Question 2 Question 3 Question 4 Question 5 Question 6

Case 1 3 3 2 3 3 2
Case 2 3 3 2 3 3 3
Case 3 3 3 3 2 2 3
Case 4 3 3 2 2 3 2
Case 5 2 3 2 3 3 2
Case 6 3 3 2 3 3 3
Case 7 3 3 2 3 3 2
Case 8 3 3 2 3 3 3
Case 9 3 3 3 2 2 3
Case 10 3 3 2 2 3 2
Case 11 2 3 2 3 3 2
Case 12 3 3 2 3 3 3
Case 13 3 3 2 3 3 2
Case 14 3 3 2 3 3 3
Case 15 3 3 3 2 2 3
Case 16 3 3 2 2 3 2
Case 17 2 3 2 3 3 2
Case 18 3 3 2 3 3 3
Case 19 3 3 2 3 3 2
Case 20 3 3 2 3 3 3
Case 21 3 3 3 2 2 3
Case 22 3 3 2 2 3 2
Case 23 2 3 2 3 3 2
Case 24 3 3 2 3 3 3
Case 25 3 3 2 3 3 2
Case 26 3 3 2 3 3 3
Case 27 3 3 3 2 2 3
Case 28 3 3 2 2 3 2
Case 29 2 3 2 3 3 2
Case 30 3 3 2 3 3 3
Case 31 3 3 2 3 3 2
Case 32 3 3 2 3 3 3
Case 33 3 3 3 2 2 3
Case 34 3 3 2 2 3 2
Case 35 2 3 2 3 3 2
Case 36 3 3 2 3 3 3
Case 37 3 3 2 3 3 2
 PROGRAM EFFECTIVENESS SURVEY 31

Case 38 3 3 2 3 3 3
Case 39 3 3 3 2 2 3
Case 40 3 3 2 2 3 2
Case 41 2 3 2 3 3 2
Case 42 3 3 2 3 3 3
Case 43 3 3 2 3 3 2
Case 44 3 3 2 3 3 3
Case 45 3 3 3 2 2 3
Case 46 3 3 2 2 3 2
Case 47 2 3 2 3 3 2
Case 48 3 3 2 3 3 3
Case 49 3 3 2 3 3 2
Case 50 3 3 2 3 3 2

View publication stats