Professional Documents
Culture Documents
Noreen Nouza
Office of Program and Professional Development, Oneida-Herkimer-Madison BOCES
Cami Berry
Safe Schools Unit, Division of Educational Services, Riverside County Office of Education (RCOE)
School-based character education and violence prevention programs focus on improving prosocial competen-
cies and reducing negative behaviors in students. The Capturing Kids Hearts Campus by Design model is a
school-level intervention that impacts student behavior by enhancing school climate through improved rela-
tional and conflict management skills. The purpose of this randomized controlled study is to evaluate the
effect of the intervention model on risk factors and protective factors that impact student behavior. The results
demonstrate that the level of principal support and quality of faculty implementation is related to and predic-
tive of student acquisition of specific prosocial behaviors (protective factors). Using school archival data for
discipline referrals, univariate ANOVA analyses demonstrated that schools implementing Capturing Kids
Hearts Campus by Design experienced on average a 22% decrease in discipline referrals whereas control
schools experienced on average an 11% increase in referrals. In addition, students in intervention schools
exhibited a 26% increase in prosocial behaviors associated with the training whereas students in control
schools exhibited a 15% decrease in these behaviors.
Carol K. Holtzapple, Director of Research, The Flippen Group, 1199 Haywood Dr., College Station, TX 77845. E-mail:
carol.holtzapple@flippengroup.com
Journal of Research in Character Education, 9(1), 2011, pp. 7190 ISSN 1543-1223
Copyright 2011 Information Age Publishing, Inc. All rights of reproduction in any form reserved.
72 Journal of Research in Character Education Vol. 9, No. 1, 2011
opportunity for participants to learn and prac- Building Healthy Relationships specifically
tice skills that exert a positive impact on class- address how to implement this intervention
room management. strategy. Implementation measures (surveys
The theoretical framework underlying the and observational tools) are then used to deter-
Capturing Kids Hearts Campus by Design mine whether or not the program components
learning approach is the Social Cognitive The- within these modules were delivered with
ory developed by Bandura (1986). A central fidelity (i.e., adherence).
premise of Social Cognitive Theory is that Previously reported experimental (Castro
behavior is dynamic, dependent upon both per- Johnson, & Smith, 2008; Danaher, 2006) and
sonal constructs and environmental factors that quasi-experimental (Cirillo-Teverbaugh, 1994;
influence each other simultaneously (Perry, Cirillo-Teverbaugh & Colwell, 1993; Sher-
Baranowski, & Parcel, 1991). Social Cognitive wood, 2003) studies, as well as numerous lon-
Theory asserts that the likelihood that a behav- gitudinal case studies (The Flippen Group,
ior will be repeated is a function of the individ- 2006), have highlighted the positive effects
uals expectancy that the behavior will lead to that components of this comprehensive process
a particular outcome and of the extent to which exert on behavioral and academic outcomes.
that outcome is valued (Norman, 1991). The magnitude of the effect is impacted by a
In addition, a person must have self-effi- number of variables, including administrative
cacy in order for the behavioral change to support for the process and level of program
occur: individuals must first have the convic- implementation by school faculty members.
tion that they can execute the behavior suc- Comprehensive interventions usually are
cessfully before the desired behavior is not implemented fully in the first year, and
acquired. This conviction is acquired through even in subsequent years, not all schools
observing others and implementing self-regu- implement all of the core components of inter-
latory mechanisms (Bandura, 1986). Self-reg- ventions as prescribed by the developer
ulation involves self-assessing behavioral (Berends, Bodilly, & Kirby, 2002; Glennan,
performance, keeping track of actions, com- 1998; St. Pierre & Kaltreider, 2004) due to
paring actions to a standard of behavior, and conflicts or issues that inevitably arise when
self-directing acquired behaviors. An out- attempting to disseminate a school-wide pro-
standing summary of the assumptions, state- gram. As a result of this variability in imple-
ments, and concepts of Social Cognitive mentation, Zhang, Shkolnik, and Fashola,
Theory that help explain how people develop (2005) report that student outcomes such as
and maintain behaviors was developed by academic achievement may lag years behind
Glanz, Rimer, and Lewis (2002). The 11 fac- initial program implementation. In addition,
tors that affect behavioral change, the inter- the relationship between faculty implementa-
vention strategies that promote positive tion and student outcomes is often weak, and
changes in behavior by addressing these fac- becomes weaker when implementation stag-
tors, and the program inputs in the Capturing nates or declines over time.
Kids Hearts process designed to implement Interventions that impact an entire school
the intervention strategies are provided in community must be implemented fully and
Table 1. For instance, observational learning with fidelity in order for schools to obtain
(factor 7) is a factor that impacts behavior by maximum effect. A number of studies (Domi-
helping the learner acquire new behavior by trovich & Greenberg, 2000; Durlack & Wells,
observing the behavior of someone else. The 1998; Gresham, Gansle, Noell, & Cohen,
intervention strategy includes providing credi- 1993) report that few programs provide data
ble role models (teachers and student leaders) on levels of implementation or consider
of the targeted behavior. The modules Lead- whether implementation quality is related to
ership Skills, EXCEL Model, and Values/ intervention outcomes. Hallfors and Godette
74 Journal of Research in Character Education Vol. 9, No. 1, 2011
TABLE 1
Relating Selected Capturing Kids Hearts Topics and
Activities to Behavioral Factors and Intervention Strategies Identified by Social Cognitive Theory
# Factors That Definitions Intervention Capturing Kids Hearts
Affect Behavior of Factors Strategies Topics & Activities
1 Environment Factors external to person Provide opportunities for social Name Game Warped Speed
such as social support and support SocialAnxiety/
family DecreasingSocial Anxiety
2 Situation Perception of the Correct misperceptions and First Impressions/What
environment promote healthy and safe People Notice
atmosphere Styles of Interaction
3 Behavioral Knowledge and skill to Promote mastery learning 1 Minute Speeches
Capability perform a given behavior through social skills and skills Leadership Skills
training Steps for Personal Growth
4 Expectations Anticipatory outcomes of Model positive outcomes of EXCEL Model
a behavior healthful behavior Listening Skills
5 Expectancies The values that the person Present outcomes of change that Social Contract
places on given outcomes; have functional meaning Discipline Model
incentives
6 Self-Regulation Personal goal-directed Provide opportunities for EXCEL Model
behavior or performance self-assessing, self-monitoring, Personal Mastery
goal setting, problem solving, Steps for Personal Growth
and self-reward
7 Observational Behavioral acquisition that Include credible role models of Leadership Skills
Learning occurs by watching the the targeted behavior EXCEL Model
actions and outcomes of Values/Building
others behavior Healthy Relationships
8 Reinforcements Responses to a persons Promote self-initiated rewards EXCEL Model
behavior that increase or and incentives Leadership Skills
decrease the likelihood of Steps for Personal Growth
re-occurrence
9 Self-Efficacy The persons confidence in Approach behavioral change in EXCEL Model
performing a particular small steps to ensure success Reframing
behavior
10 Emotional Coping Strategies or tactics that Provide training in problem EXCEL Model
Responses are used by a person to solving and stress management Reframing
deal with emotional Question Process/Discipline
stimuli Model
11 Reciprocal The dynamic interaction of Consider multiple avenues to Every process in the Captur-
Determinism the person, the behavior, behavioral change, including ing Kids Hearts training
and the environment in environment, skill acquisition, addresses this assumption
which the behavior is and personal change
performed
(2002) found that while most school districts school-based program does not produce con-
selected research-based programs, many pro- sistent student outcomes if it is not imple-
grams were implemented with low fidelity. mented with fidelity.
Other studies (Kam, Greenberg, & Walls, Greenberg, Domitrovich, Graczyk, and
2003, and references therein) demonstrate that Zins (2005) identify a number of reasons for
implementation quality impacts student out- including implementation assessment when
comes; thus, even an empirically-validated, evaluating interventions. For instance, lack of
Implementation of a School-Wide Adolescent Character Education and Prevention Program 75
implementation data may result in researchers the leadership team developed a BOCES-wide
incorrectly determining that a program is inef- vision specifically addressing student behav-
fective when, in fact, the program was imple- iors and skills that would prepare students to
mented so poorly that no analyses can provide be successful in the workplace beyond high
meaningful conclusions. Additionally, pro- school graduation. The findings from the
gram evaluators may be faced with consider- National Longitudinal Study on Adolescent
able unexplained variance in program effects. Health (Resnick et al., 1997) indicated that the
Obtaining implementation information can leading factor supporting positive outcomes
help researchers with both types of research and preventing youth participation in negative
difficulties by providing data that allows them behaviors is connectedness to an adult role
to draw valid conclusions and identify reasons model.
for variability in program outcomes. Oneida-Herkimer-Madison BOCES imple-
In this study, we report the first-year results mented a strategic plan to increase student
obtained after implementation of a connectedness. A Professional Development
school-based character education program, Council was convened and involved a commit-
Capturing Kids Hearts Campus by Design. tee of 20 superintendents, assistant superinten-
The primary questions of the study include: (1) dents, and other district leadership members
How well do principals in intervention schools tasked with examining program options. The
support the program as measured by personally team met for a number of months, reviewing
modeling the skills taught in the process and regional needs assessments, interviewing prin-
providing a systemic mechanism by which pro- cipals and staff, and engaging in extensive
gram progress can be discussed? (2) How well committee discussions.
do teachers in intervention schools implement All members of the Professional Develop-
Capturing Kids Hearts skills in the classroom? ment Council were asked to consider profes-
(3) What is the relationship between principal sional development programs that would help
support and teacher implementation? (4) How connect administrators, teachers, and students.
well do students acquire the skills that are mod- Eight research-based programs (as evidenced
eled by teachers? (5) What is the relationship by inclusion of the program on a model pro-
between teacher implementation of the process gram list or by publication of research articles
and student acquisition of skills? (6) What spe- in peer-reviewed journals) were identified that
cific changes occur in student behaviors (i.e., would address relational connectedness. The
changes in the number of discipline referrals or Capturing Kids Hearts process, a comprehen-
increases in specific prosocial behaviors) as a sive, school-based program designed to
result of program implementation? The rela- improve prosocial competencies and reduce
tionship between fidelity of implementation negative behaviors in students, was the pro-
and changes in student outcomes is discussed. gram that was chosen. The committee then
developed a test group of individuals to partic-
ipate in the training and provide feedback.
METHODS The test group, consisting of 52 teachers
and administrators, attended a Capturing Kids
Background Hearts 3-day workshop. A BOCES-wide meet-
ing was then convened to discuss outcomes of
School leadership teams from the the training, determine the districts options,
Oneida-Herkimer-Madison Board of Coopera- and develop next steps. A unanimous vote was
tive Educational Services (BOCES) in New provided by the test group indicating that the
York participated in a state-wide meeting that program would be of great benefit for the stu-
led to the directive for districts to improve stu- dents and would help the schools meet
dent outcomes. In response to this directive, state-directed, district-wide goals.
76 Journal of Research in Character Education Vol. 9, No. 1, 2011
Before committing a significant amount of mate the apparent effectiveness of the inter-
funding to implement the training dis- vention. Eighteen public high schools served
trict-wide, the BOCES leadership team by BOCES and RCOE provided academic and
decided to invest time and resources in a pilot behavioral data from the 3 years preceding the
study that would be capable of measuring the study (SY2005, 2006, and 2007). After deter-
effectiveness of the program in local high mining the level of variability in the informa-
schools. Principals within the region that were tion derived from the 3-year discipline referral
willing to participate in the study were asked to data (Holtzapple, 2011), eight schools exhibit-
submit school archival data from the preceding ing the lowest variability (< 25%) in the 3-year
three years describing demographic, behav- data were paired according to school type
ioral, and academic trends. Due to limited (grades 7-12 or grades 9-12) and then ran-
funding, BOCES was able to include only six domly assigned to intervention or control
schools in the research study. Statistical mod- groups. Schools with greater levels of variabil-
eling (Holtzapple, 2009; Spybrook Rauden- ity in the data were not eligible for participa-
bush, Liu, & Congdon, 2006) indicated that for tion in the study.
a six-school study using randomization at the
school level, researchers could encounter sig-
Participant Sample
nificant difficulties in measuring outcomes.
Therefore, a partner district was considered in The study sample included 8,350 students
order to increase the number of schools in the in Grades 7-12 and 469 teachers from 6 high
study, thus improving the likelihood that schools located in districts served by the
changes in measurable outcomes could be Oneida-Herkimer-Madison Board of Coopera-
detected if they occur. tive Educational Services (BOCES) in New
The Riverside County Office of Education York and 2 high schools served by the River-
(RCOE) in California also sought to document side County Office of Education (RCOE). In
the effectiveness of this program in local the intervention group, 4% of the students
schools. The leadership had previously devel- were African American, 33% were Hispanic,
oped a strategic initiative to increase school and 57% were Caucasian. In the control group,
connectedness within the county and had 5% of the students were African American,
implemented the Capturing Kids Hearts pro- 26% were Hispanic, and 65% were Caucasian.
cess for a number of years, but had not partici- Overall, 4% of the students were African
pated in a randomized, controlled research American, 28% were Hispanic, and 62% were
trial. The RCOE leadership team asked princi- Caucasian, with 30% of the students eligible
pals interested in participating in a study to for free or reduced lunch.
submit the same data as that submitted by the
BOCES schools.
Program Description
vey contained 19 items. When developing sur- Teacher and Principal Implementation
vey tools evaluating factual events Measures
(observable behaviors), key principles
involved in good survey design include (1) The proximal outcomes of the training
asking questions that respondents can answer, (teacher and principal implementation) were
(2) clearly defining key terms so that respon- assessed using the Capturing Kids Hearts
dents are all answering the same question, and Observational Form at the classroom level
(3) providing adequate context so that respon- through observation of 40 randomly selected
dents will see the value of answering the ques- classrooms (using a web-based number ran-
tions accurately (Fowler, 1995). In order to domizer; 5 teachers per school), and at the
follow these key principles, the final survey individual-level (all teachers) using the Cap-
and observational form only included items turing Kids Hearts Survey. Each wave of
measuring specific behavior. classroom-level observational data was aggre-
Table 2 demonstrates how key outcomes gated to represent a school-level construct
(Tseng & Seidman, 2007). A data triangulation
taught in the Capturing Kids Hearts process
approach was used in that each wave of indi-
correspond to items described in the taxon-
vidual-level (survey) data was also aggregated
omy of variable outcomes developed by
and used to determine whether or not teachers
Berkowitz and Bier (2005). The table further
perspectives of how well they model Captur-
links these outcomes to items within an
ing Kids Hearts behaviors correspond with
established psychometric survey (WestEd,
the implementation ratings obtained from
2008), and provides examples of how psy-
observing teachers in the classroom. School
chometric survey items were converted to
implementation ratings were generated by
measures of observable behaviors in the Cap-
trained members of the program developers
turing Kids Hearts Survey or Observational
research staff, each of whom had taught the
Form. For instance, to measure the leader-
process and had been part of the evaluative
ship skill of engagement (first item in Table process for more than 5 years. Implementation
2) the more general terms used in the Cali- analyses were based upon two, on-site obser-
fornia Healthy Kids Survey (supportive vational sessions for each school during the
and inviting) was converted to a specific school year. The first wave of observational
behavior that can be observed (the teacher is data (midyear data point) was obtained
at the door greeting students). Other behav- approximately 3 to 5 months after teachers
ioral measures of prosocial skills are pro- attended Capturing Kids Hearts training, and
vided in Table 2. the second wave was obtained at the end of the
Both the Capturing Kids Hearts survey school year. The evaluation measured proxi-
and observational form contain similar ques- mal outcomes in three areas: (1) principal sup-
tions, but are specific to the person filling out port for the process (i.e., (s)he personally
the tool (i.e. The teacher [is at the door] is modeled Capturing Kids Hearts skills and
an example of the beginning of a sentence established procedures for following up with
used in the observational form whereas I on-site mentors), (2) teacher implementation
[am at the door] is an example of the corre- of the Capturing Kids Hearts relational skills,
sponding item used in the self-report sur- and (3) teacher implementation of specific
veys). A review of these items by former classroom management techniques. All three
school administrators and teachers, current areas were rated based upon a 3-point Lik-
teachers, and program trainers with knowl- ert-type scale with 0 indicating no implemen-
edge of the subject matter established the tation, 1 indicating partial implementation, and
content validity (Litwin, 1995) of the survey 2 indicating full implementation. Fractional
and observational form. points were not used in the evaluation of indi-
Implementation of a School-Wide Adolescent Character Education and Prevention Program 79
TABLE 2
Relating Capturing Kids Hearts EXCEL Knowledge and Skills to a
Variable Outcome Taxonomy, Items From the California Healthy Kids Survey, and
Items From the Capturing Kids Hearts Observable Behaviors Survey and/or Implementation Form
Example Measures of
Observable Behaviors in the
Capturing Kids Hearts
EXCEL Knowledge Variable Outcome Taxonomy* Example Items From the Survey and Observation
& Skills (*Berkowitz & Bier, 2005) CA Healthy Kids Survey Form
E Learn Meet & Greet 2.2 Personal Morality This school is a supportive The teacher is at the door
skills (Leadership Skills 2.2.8) and inviting place for students greeting students as they
to learn. (6) enter the classroom (E1)
Model social skills2.4 Communicative
Competency Adults at this school treat The teacher uses a
Model professional (Communication Skills 2.4.1) every student with respect. welcoming tone of voice
skills (Attentive Listening 2.4.2) (39) (E2)
X Explore needs of 2.3 Proocial Behaviors This school gives all students For 3-5 minutes at the
others (Caring Concern 2.3.6) equal opportunity to beginning of class, the
(Sharing 2.3.12) participate in classroom teacher asks students to
Implement Good discussions or activities. (17) share (X6)
Things 2.4 Communicative
Competency Adults at this school
(Attentive Listening 2.4.2) acknowledge and pay The teacher asks
attention to students. (34) open-ended questions
(X7)
C Acquire effective 2.3 Prosocial Behaviors & The school encourages The teacher and students
listening skills Attitudes opportunities for students to created a Social Contract
(Teamwork and cooperation decide things like class that outlines (C9)
Build a Social 2.3.7) activities or rules. (16)
Contract
2.4 Communicative
Ask for feedback Competency
(Communication Skills 2.4.1)
2.7 Citizenship
(Democratic Values 2.7.1)
E Implement a Social 2.2 Personal Morality The school handles discipline The teacher reviews the
Contract (Respect 2.2.3) problems fairly. (27) Social Contract (Em15)
Develop 2.3 Prosocial Behaviors & The school effectively The teacher uses a question
self-managing skills Attitudes handles student discipline and approach to redirect
(Helping others 2.3.8) behavioral problems. (28) misbehavior (Em16)
Use nonverbal 4.4 Problem-Solving Skills Adults at this school treat all The teacher does not avoid
signals to manage (Conflict resolution strategies students fairly. (38) conflictaddresses ALL
behavior 4.4.5) misbehavior (Em17)
(Behavioral adjustment 4.4.3)
L Use meaningful 2.2 Personal Morality Adults at this school want The teacher ends class with
ending to class (Leadership Skills 2.2.8) every student to do their best. a powerful message such
(35) as (L19)
vidual teachers. An aggregated implementa- reduction of risk factors such as discipline
tion score (reported as a percentage of full referrals and acquisition of protective factors
implementation) was calculated for each such as prosocial relational skills).
school.
Stratification of Schools by Level of
Student Acquisition of Prosocial Skills Variance in Discipline Referral Data
Measures
Using discipline referral data as a measur-
Observational measures of students acqui- able outcome of program implementation can
sition of prosocial skills (distal outcomes) spe- be problematic because the application of
cific to the intervention were obtained during referral rules and policies can vary greatly
the two on-site days for each school in which between teachers within a school and between
teacher and principal implementation mea- schools within a district. Additionally, new
sures were acquired. Selected items from the policies may be implemented in one or more of
California Healthy Kids School Climate Sur- the intervention or control schools involved in
vey (2008) were matched to specific behav- a research study such that the number of refer-
iors/skills taught in the Campus by Design rals reported at the end of the study is not
model, and these behaviors/skills were indicative of what actually occurred in the
matched to the taxonomy of outcomes school. For instance, a principal in a control
described by Berkowitz and Bier (2005). school could implement stricter discipline
These outcomes were divided into subscales rules during the year such that it would seem
that measured modeling of specific skills dem- that referrals increased relative to the previous
onstrating personal morality (sense of justice/ years data, but in fact, the increase would only
fairness, respect), prosocial behaviors (caring reflect changes in policies, not true increases in
concern for others, teamwork, helping others, disruptive student behaviors. Thus, although
sharing), communicative competency (com- discipline referral records continue to be used
munication skills, attentive listening), citizen- to evaluate behavioral changes within schools
ship (democratic values), and problem-solving (Irvin, Tobin, Sprague, Sugai, & Vincent,
skills (consequential thinking, behavioral 2004), the integrity of the data may be compro-
adjustment, conflict resolution). An aggre- mised when consistency in the discipline refer-
gated prosocial skills outcome score was also ral system is not maintained (Wright & Dusek,
calculated, giving equal weight to each sub- 1998).
scale. Sprague, Sugai, Horner, and Walker,
(1999) attempted to overcome potential vari-
Data Collection ability in schools discipline referral systems
by selecting schools that had an established
Schools were provided a prestudy data form discipline referral system and that were in the
requesting demographic, academic, and initial stages of a plan to improve school-wide
behavioral data for the 3 school years preced- discipline procedures. These two strategies
ing initiation of the study. Principals from each were utilized in the present study. In addition,
of the schools provided the data sheet and sub- a stratification of schools strategy based
mitted it to their respective leadership teams. upon the variability in the discipline referral
For Capturing Kids Hearts Campus by Design data in the 3 years preceding the study was
(CKH CBD), a process that impacts school cli- implemented so that changes in referrals could
mate through strengthening teacher-student be detected if they occurred.
connectedness, the BOCES and RCOE leader- Stratification of schools to meet statistical
ship teams were primarily interested in mea- demands of research trials (Holtzapple, 2011)
suring differences in behavioral outcomes (i.e. is particularly critical to the success of studies
Implementation of a School-Wide Adolescent Character Education and Prevention Program 81
involving fewer than 10 schools. In the present because scheduling conflicts prevented com-
study, funding was available for only 8 pletion of classroom observations in the inter-
schools; therefore, given the statistical diffi- vention schools prior to the training. The pre-
culties involved in measuring outcomes when and postsurvey (end of year) means (23.25
using fewer than 10 schools, the variability in 5.68 and 25.25 5.44, respectively; p
the number of discipline referrals in the 3 years [two-tailed] = 0.6222) from the control schools
preceding the study was determined for each provides a test-retest reliability measure of the
school that applied to be a study participant. survey instrument. The presurvey and
The variability in the data ([standard deviation/ mid-year observation means (23.25 5.68 and
3-yr mean number of discipline referrals] 18.75 5.5, respectively; p [two-tailed] =
expressed as a percentage) ranged from 1% to 0.2803) in the control schools demonstrates
42% (see Appendix). Because previous studies that the results from prestudy self-report sur-
had demonstrated that the Capturing Kids veys are a reliable substitute for the prestudy
Hearts process can reduce discipline referrals observational instrument. A sampling of 5 ran-
in schools 25% to 75%, and because school domly selected teachers from each school pro-
leadership teams wanted to be able to detect vides good information about faculty
changes in this outcome, the 8 schools whose follow-through and the pervasiveness of
school-level variability in discipline referral implementation. Therefore, the presurvey
data was less than 25% were chosen for the results served as the initial baseline for all
study. schools, whereas midyear, and end-of-year 1
data sets were obtained using direct observa-
tion of behaviors and skills.
Statistical Analysis
Initially, intervention and control schools
Statistical power analyses (Holtzapple, demonstrated similar baseline levels of specific
2009) were conducted using the Optimal behaviors and skills associated with the Cap-
Design software (initially version 1.76 and turing Kids Hearts training. By the
then the updated version 2.0) for cluster ran- end-of-year 1 evaluation, control schools
domized trials (Spybrook et al., 2006; Spy- exhibited levels of CKH-CBD skills that were
brook, Raudenbush, Liu, & Congdon, 2009). similar to baseline levels; however, the inter-
The significance level () was held constant at vention schools exhibited increased, but vary-
0.05 for all calculations. Univariate ANOVA ing levels of CKH-CBD skills, with
analyses were conducted using SPSS version implementation ranging from 32% to 72% as
16.0 software. determined by observation of CKH-CBD skills
exhibited by teachers and principals. In order to
determine which factors affect program imple-
RESULTS & DISCUSSION mentation and, ultimately, student outcomes,
we investigated: (1) the level of principal sup-
Implementation Analyses port of the intervention, (2) the relationship
between principal support and teacher imple-
School-level prestudy, midyear, and end of mentation of the intervention in the classroom,
year implementation outcomes are highlighted and (3) the relationship between teacher imple-
in Figure 1. The prestudy survey data provides mentation and student acquisition of specific
the baseline level of Capturing Kids Hearts prosocial skills.
Campus by Design (CKH-CBD) prosocial Figure 2 demonstrates the end-of-year 1
behaviors self-reported by school principals level of principal support for the intervention
and faculty for each of the eight schools. The for all 8 schools as measured by direct obser-
presurveys were used as a surrogate for vation of principal-initiated skills related to
prestudy direct observation of behaviors Campus By Design. Note that even in the con-
82 Journal of Research in Character Education Vol. 9, No. 1, 2011
Intervention
Percent CKH-CBD Implementation
40
Intervention school
30
that did not
implement CKH-
CBD well
20
Control schools that
did not receive
10
CKH-CBD training
FIGURE 1
Assessing Implementation Progress. Teacher and principal implementation of CKH-CBD skills were mea-
sured prior to CKH training, midyear, and at the end of the school year.
trol schools (represented by the bar graphs Levels of these skills in principals from control
located in the small box in Figure 2), the base- schools ranged from 10% to 33%.
line level of principal support/principal leader- Initially, the principals of the intervention
ship is not zero. Posttest evaluation of schools demonstrated a similar level of pro-
principal support/principal leadership was gram-specific skills; however, by the end of
based upon evaluator-observed use of specific the school year, the principals from 3 of the
behaviors and skills by the principals. Thus, intervention schools had implemented and
the levels of Campus By Design skills were modeling for their teaching faculty
observed in principals from control schools between 72% and 87% of the skill set taught
would be skills that may be expected in indi- by the intervention. In contrast, the principal
viduals as a result of natural skills, college from the fourth intervention school (I4) imple-
training, or postcollegiate principal develop- mented less than 30% of the skills.
ment trainings whose objectives overlap those Given the importance of principal leader-
of Capturing Kids Hearts Campus By Design. ship when implementing interventions, we
Implementation of a School-Wide Adolescent Character Education and Prevention Program 83
100
90
Percent Principal Implementation
of the CKH-CBD Process 80
70
60
50
40
30
20
10
0
C3 C2 C8 C6 I7 I1 I5 I4
Control () or Intervention () School
FIGURE 2
Posttest level of principal support for the CKH-CBD process. The level of principal support (implementation
of CKH-CBD skills) was determined using observational measures.
investigated how principal support for the 4 demonstrates that this is indeed the case. In
intervention is related to teacher implementa- intervention school I4, low implementation of
tion of CKH-CBD skills in the classroom. As the CKH-CBD skills by teachers resulted in a
can be observed in Figure 3, there is a direct, low level of student acquisition of specific pro-
linear relationship between the level of support social skills.
provided by the principal and the degree to
which the process is implemented by teachers.
Analyses of Student Outcomes:
In the intervention schools, when principal
Withdrawal for Nonadherence Versus
support is low (school I4), there is a corre-
Intention-to-Treat
spondingly low level of teacher implementa-
tion. In schools in which principal modeling of The information in the preceding figures
the intervention skills is high (schools I1, I5, demonstrate that one of the intervention
and I7), there is a correspondingly high level schools implemented less than 35% of the
of teacher implementation of the interventions skills taught in the Campus By Design model
skills in the classroom. and may therefore be considered a nonadher-
The variability in the level of implementa- ing participant. A withdrawal for nonadher-
tion of the process in the classroom in inter- ence approach was considered for this
vention schools suggests that there will be intervention school exhibiting < 60% imple-
similar variability in student outcomes. Figure mentation of the intervention because doing so
84 Journal of Research in Character Education Vol. 9, No. 1, 2011
80
I7
I7
70
Percent Teacher Implementation
I5
I5
of the CKH-CBD Process
60
I1
I1
50
40
I4
I4
30
20
C8
C8
10 C6 C3
C3
C6 C2
C2
0
0 10 20 30 40 50 60 70 80 90 100
FIGURE 3
Teacher implementation versus principal support. The data demonstrates that there is a direct, linear relation-
ship between the percent of teacher implementation of CKH-CBD skills in the classroom versus the percent of
principal support for the intervention (R2=0.9398).
would allow the trial to provide a comparison not be avoided when nonadhering participants
of student outcomes resulting from optimal are dropped from the trial. In addition,
intervention versus control condition. Includ- Chaudhry and McDermott (2008) point out
ing the results from the nonadhering school that nonadherence is not only a problem in
would contaminate the intervention group research studies, but is also a major barrier to
and reduce the observed outcomes of the trial, obtaining desirable outcomes in nontrial sce-
thus making the intervention appear to be less narios. Low adherence to the intervention in
effective. In clinical trials, it has been pro- trials of educational interventions may indicate
posed that if the rules for withdrawing partici- that widespread use of the program may not be
pants are identified prior to study initiation, feasible unless developers address the reasons
then it is legitimate to remove nonadhering for non-compliance (i.e., participant knowl-
participants (Sackett & Gent, 1979). However, edge of what is required is lacking or fol-
Friedman, Furberg, and DeMets (1998) argue low-up strategies are ineffective). For the
that an intention-to-treat approach should be following evaluation of student outcomes, an
used in research studies because adherence, or intention-to-treat approach was used; there-
lack thereof, is itself a response to the interven- fore, the outcomes of all eight schools were
tion. They state that the potential for bias can- included in the analyses.
Implementation of a School-Wide Adolescent Character Education and Prevention Program 85
80
60
I1 I5
50
40
I4
30
20
C6
10 C3
C8
C2
0
0 10 20 30 40 50 60 70 80
FIGURE 4
Student acquisition of prosocial skills versus teacher implementation. The data demonstrates that there is a
direct, linear relationship between the percent of student acquisition of prosocial skills versus the percent of
teacher implementation of CKH-CBD skills in the classroom (R2 = 0.9229).
Composite Sub-Category
100 Scores for Scores for PMPersonal Morality
Pro-Social Pro-Social
sense of justice/fairness
Outcomes Outcomes
90 respect
Pr-SPro-Social Behaviors
80 caring concern for others
teamwork
Control ( ) vs. Intervention ( ) Schools
helping others
70 sharing
60 CCCommunicative Competencies
communication skills
attentive listening
50
CZCitizenship
40 democratic values
PSProblem Solving
30 consequential thinking
behavioral adjustment
20 conflict resolution
10
0
Pre- Post- PM Pr-S CC CZ PS
Test Test Post-Test
FIGURE 5
Pretest and posttest composite scores for prosocial outcomes for students in the control and intervention
schools are provided on the left-hand side of the figure. The posttest subcategory scores for personal morality
(PM), prosocial behavior (Pr-S), communicative competencies (CC), citizenship (CZ), and problem solving
(PS) are provided on the right-hand side of the figure.
APPENDIX
Mean Campus Enrollment and Number of Discipline Referrals
(During the 3 Years Preceding the Study) for Schools Originally Interested in Participating in the Study
SD/Mean #
Discipline
Mean Campus Mean # Discipline Referrals Variability
Condition School # Enrollment Referrals ( SD) (expressed as %) Stratum
ing relationships that promote open communi- data is unstable. Next steps in research include
cation. Implementing the EXCEL Leadership measuring outcomes in elementary and middle
Model through the Capturing Kids Hearts school students, and working with a different
Campus by Design process results in a culture unit of analysis (using both intervention and
in which students and staff model leadership control participants within the same school)
qualities, show respect for others, share and despite the possibility of a spillover effect
celebrate each others successes, demonstrate between participants.
caring concern for others, communicate effec- When implemented with fidelity, the Cap-
tively, listen attentively to others, have a sense turing Kids Hearts Campus by Design model
of justice and fairness, help others, resolve increases prosocial behaviors (protective fac-
conflicts, and work together as a team.
tors) in students and decreases the number of
The limitations of the study are directly
discipline referrals (risk factors) in interven-
linked to the statistical constraints placed on
tion schools. Continued support for the process
school districts when attempting to conduct a
will help sustain the behaviors that enhance the
randomized, controlled trial using schools as
the unit of randomization. In order to minimize learning environment and equip students for
costs, (1) a limited number of schools were success in school and after graduation.
used, and as a result of the associated statistical
constraints, (2) principals had to be willing to Acknowledgment: The author is grateful
participate, (3) only schools with stable base- for invaluable research discussions with Drs.
line data were eligible for the study, and (4) Thomas Lickona, Matt Davidson, and Vlad
schools were recruited from only two states, Khmelkov (SUNY-Cortland); Vic Battistich
New York and California. The results do not (late) and Marvin Berkowitz (U. Missouri-St.
necessarily demonstrate outcomes that would Louis); Jessaca Spybrook (U. Western Michi-
occur in schools in which principals are disin- gan); and Victor Willson (Texas A&M Uni-
clined to participate or for which the baseline versity).
88 Journal of Research in Character Education Vol. 9, No. 1, 2011
emotional and behavioral disorders. Boston, ory. In K. Glanz, F. Marcus Lewis, & B.Rimer
MA: Allya and Bacon. (Eds.), Health behavior and health education.
Hallfors, D., & Godette, D. (2002). Will the Princi- San Francisco, CA: Josey-Bass.
ples of effectiveness improve prevention prac- Resnick, M. D., Bearman, P. S., Blum, R. W., Bau-
tice? Early findings from a diffusion study. man, K. E., Harris, K .M., Johnes, J., Tabor, J.,
Health Education Research, 17(4), 461-470. Beuhring, T., Sieving R. E., Shew, M., Ireland,
Holtzapple, C. K. (2009). Controlling for baseline M., Bearinger, L. H., & Udry, J. R. (1997). Pro-
covariates to improve the statistical power of tecting adolescents from harm. Findings from
efficacy trials involving whole school prevention the National Longitudinal Study on Adolescent
programs. 17th Annual Meeting of the Society Health. JAMA, 278(10), 823-832.
for Prevention Research. Retrieved from http:// Resnick, M. D., Ireland, M., & Borowsky, I. (2004).
www.preventionscience.org/ Youth violence perpetration: What protects?
SPR_09_program_COMPLETE.pdf (abstract What predicts? Findings from the National Lon-
#224, pg 63) gitudinal Study of Adolescent Health. J. Adoles-
Holtzapple, C. K. (2011). Measuring behavioral cent Health, 35, 424e1-424e10.
outcomes associated with comprehensive char- Sackett, D. L., & Gent, M. (1979). Controversy in
acter education programs: A practical approach counting and attributing events in clinical trials.
to using fewer schools in school randomized N. Engl. J. Med. 301:1410-1412.
controlled trials. Journal of Research in Charac- Sherwood, R. (2003). It all began with a handshake.
ter Education, 9, 57-69. The Effective Schools Project Journal, 9, 6-11.
Irvin, K. L., Tobin, T. J., Sprague, J. R., Sugai, G., Sprague, J. R., Sugai, G., Horner, R., & Walker,
& Vincent, C. G. (2004). Using office discipline H.M. (1999). Using office discipline referral
referral data to evaluate school-wide discipline data to evaluate school-wide discipline and vio-
and violence prevention interventions. Oregon lence prevention interventions. Oregon School
School Study Council (OSSC) Bulletin, 42(2), Study Council (OSSC) Bulletin, 42(2), 1-18.
1-18. Spybrook, J., Raudenbush, S. W., Liu, X., & Cong-
Kam, C. -M., Greenberg, M. T., & Walls, C. T. don, R. (2006). Optimal design for longitudinal
(2003). Examining the role of implementation and multilevel research: Documentation for the
quality in school-based prevention using the Optimal Design software.
PATHS curriculum. Prevention Science, 4(1), Spybrook, J., Raudenbush, S.W., Liu, X., & Cong-
55-63. don, R. (2009). Optimal design for longitudinal
Lickona, T., & Davidson, M. (2005). Smart & good and multilevel research: Documentation for the
high schools: Integrating excellence and ethics Optimal Design(version 2.0) software.
for success in school, work and beyond. Cort- Retrieved from http://www
land, NY: Center for the 4th and 5th Rs (Respect .wtgrantfoundation.org/resources/overview/
& Responsibility). Washington, DC: Character research_tools/research_tools
Education Partnership. St. Pierre, T. L., & Kaltreider, D. L. (2004). Tales of
Litwin, M. S. (1995). How to measure survey reli- refusal, adoption, and maintenance: Evi-
ability and validity. San Francisco, CA: SAGE. dence-based substance abuse prevention via
Marzano, R. J. (with Marzano, J. S., & Pickering, D. school-extension collaborations. Am. J. Evalua-
J.). (2003). Classroom management that works. tion, 25(4), 479-491.
Alexandria, VA: ASCD. Tseng, V., & Seidman, E. (2007). A systems frame-
Norman, P. (1991) Social learning theory and the work for understanding social settings. Am. J.
predication of attendance at screening. Psychol- Community Psychol, 39, 217-228.
ogy and Health, 5, 231-239. U.S. Department of Education (2009). Smaller
Office of Juvenile Justice and Delinquency Preven- Learning Communities (SLC) Awards Database.
tion. (2004). OJJDP Model Programs Guide, Retrieved from http://slcprogram.ed.gov/
version 3.0. Retrieved from http://www2 cgi-bin/mysql/
.dsgonline.com/mpg/prevention.aspx? slcawards.cgi?showds=s&showstructure=
continuum=prevention &showstrategy=&searchss=&showdemopercent
Perry, C. L., Baranowski, T., & Parcel, G. S. & =&showdemo=&showstates=%&andor=
(1991). How individuals, environments and AND&searchforschoolname=&andorprofile=
health behavior interact: Social Learning The- AND&searchprofile=&showcohort=&sortby=
90 Journal of Research in Character Education Vol. 9, No. 1, 2011