Quality Early Learning — Key To School Success

A First-Phase 3-Year Program Evaluation Research Report for

Pittsburgh’s Early Childhood Initiative (ECI)

FINAL REPORT BY:

Stephen J. Bagnato, Ed.D., NCSP,

Professor of Pediatrics and Psychology
Director, Early Childhood Partnerships
Director, SPECS Program Evaluation Research Team
Children’s Hospital of Pittsburgh
The UCLID Center at the University of Pittsburgh

OCTOBER 2002
Quality Early Learning—-Key to School Success

A First-Phase 3-Year Program Evaluation Research Report for

Pittsburgh’s Early Childhood Initiative (ECI)

FINAL REPORT BY

STEPHEN J. BAGNATO, ED.D., NCSP

JANELL SMITH-JONES, PH.D
GEORGE MCCLOMB, PH.D
JENNETTE COOK-KILROY, M.ED

EARLY CHILDHOOD PARTNERSHIPS

SPECS PROGRAM EVALUATION RESEARCH TEAM
CHILDREN’S HOSPITAL OF PITTSBURGH
THE UCLID CENTER AT THE UNIVERSITY OF PITTSBURGH

OCTOBER 2002
 T A B L E O F C O N T E N T S

Acknowledgments 1
Author Note 2
Chapter 1: 3
What is the research base on quality early childhood interventions for young children at developmental risk?
Chapter 2: 6
What is Pittsburgh’s Early Childhood Initiative (ECI)?
Chapter 3: 11
What are the expected success outcomes for ECI?
Chapter 4: 14
How does the SPECS team document the success of ECI?
Chapter 5: 29
What are the research design, questions, and analysis methods for the ECI evaluation?
Chapter 6: 36
How successful was ECI in its first three years?
Chapter 7: 42
Did children benefit from participation in ECI?
Chapter 8: 58
Did families benefit from participation in ECI?
Chapter 9: 66
Did ECI programs achieve quality?
Chapter 10: 73
Was community leadership responsible for the success of ECI programs?
Chapter 11: 90
What are the lessons learned for future ECIs?
Appendix A: 99
Research references
Appendix B: 104
What statistical analyses and results underscore ECI outcomes and conclusions?
Appendix C: 139
Samples of dependent measures in the SPECS evaluation research battery
Appendix D: 155
Sample quality mentoring plan for ECI collaborative program consultation
Appendix E: 159
Letters of correspondence among ECIM, ECI neighborhood leadership councils and SPECS
Appendix F: 174
Organizational capacities of the SPECS Evaluation Team?
Appendix G: 178
Professional profiles for the UCLID faculty on the SPECS evaluation research team?
 A C K N O W L E D G E M E N T S

The SPECS evaluation team is privileged to work with remarkable people in Pittsburgh’s ECI communities. Providers,
teachers, and parents show enormous dedication. Children in the programs inspire with their eagerness and joy in
learning. Business, corporate, and foundation leaders have our respect for their vision and their commitment to school
readiness and our role in ECI. Perhaps most of all, we are humbled to work with the people associated with the ECI
programs in urban neighborhoods, especially the community leaders who have shown such unwavering commitment and
creativity. ECI would have been impossible without the unique talents of these partners:

Greater Braddock Early Childhood Network THE SPECS TEAM

Robert Grom, CEO, Heritage Health Foundation
Wilkinsburg Early Childhood Initiative
Leon Haynes, Executive Director
Primary Care Health Services Homewood Early Childhood Initiative
Wilford Payne, Executive Director
Sto-Rox Early Learning Network
Father Regis J. Ryan, Executive Director
East Liberty Early Childhood Initiative Network
Carl Redwood, Program Director, Kingsley Association
SouthSide Early Childhood Initiative
Hugh Brannan III, Executive Director, Brashear Association
Stephen J. Bagnato, Ed.D., Director
George McClomb, Ph.D., Director, Community Strand
Thanita Adams, MA, MSW, Site Liaison
Jennette Cook-Kilroy, M.Ed., SPECS and Program Strand Coordinator
Heidi Feldman, M.D., Ph.D., Health Strand Coordinator
Elizabeth Fuchs, BA, Coordinator
Ken Jaros, Ph.D., Family Strand Coordinator
Margie Matesa, M.Ed., Site Liaison
Connie Nojeim, BA, Site Liaison
Eleana Shair, M.Ed., Data Management Coordinator
Julia Slater, M.Ed., School Transition Strand Coordinator
Janell Smith-Jones, Ph.D., Child Strand Coordinator
Kristy Stefero, BA, Research Assistant
Hoi Suen, Ph.D., Statistics and Research Design, Penn State
Carol Whitacre, BS, Early Literacy Strand Coordinator

Highlands Early Childhood Initiative

Nancy Kuritzky, Program Director

Hill District Early Childhood Initiative

James Henry, Executive Director, Hill House Association

Steel Valley Early Childhood Initiative

Penny Dykes, Program Director

Stephen J. Bagnato, Ed.D., Director

George McClomb, Ph.D., Director, Community Strand

1
 A U T H O R N O T E

Dr. Bagnato and his SPECS Evaluation Research Team were honored to be the recipient of the 2001
University of Pittsburgh Chancellor’s Distinguished Public Service Award for their community-based
program evaluation research in the Early Childhood Initiative (ECI) and their leadership in forging
innovative methods of University-Hospital-Community collaboration across Pennsylvania for young
children and families at developmental risk in their Early Childhood Partnerships program (ECP).
The SPECS research on ECI is supported by grants from The Vira and Howard Heinz Endowments, Children,
Youth, and Families program (A7681) (1997-2004); and The Maternal and Child Health Bureau, Department
of Health and Human Services, Health Services Resource Administration (5T73MC00036-06), Leadership
Education in Neurodevelopmental Disabilities, the UCLID Center at the University of Pittsburgh (1995-2005).
Requests for reprints and inquiries about SPECS/ECI or ECP can be directed to: Dr. Stephen J. Bagnato,
Ed.D., Professor of Pediatrics & Psychology, Director, Early Childhood Partnerships, Children’s Hospital of
Pittsburgh, The UCLID Center at the University of Pittsburgh, 3705 Fifth Avenue, Pittsburgh, PA 15213; email:
steve.bagnato@chp.edu.

2
 C H A P T E R 1

WHAT IS THE RESEARCH BASE ON QUALITY EARLY

CHILDHOOD INTERVENTIONS FOR YOUNG
CHILDREN AT DEVELOPMENTAL RISK?

A RESEARCH OVERVIEW
Early childhood educators and researchers have long understood the importance of providing young
children with quality early childhood education (Azar, 1999; Guralnick & Bennett, 1987; NICHD Early
Childcare Research Study 1998; Ramey & Ramey, 1992;1998). Recent findings by the National Institute of
Child Health and Human Development (NICHD) child care study emphasize that child development is
significantly influenced by the quality of child care rather than the amount of time spent in childcare
outside of the home (Azar, 1999). Children in high quality day care programs performed better on tests of
language and cognitive skills (Azar, 1999; NICHD Early Childcare Research Study, 1998). In addition, high
program quality has been an important element in preparing children for the transition to elementary
school. Many studies show that the quality of children’s child care before they entered school continue to
affect their development at least through kindergarten and perhaps through third grade (Peisner-Feinberg,
Burchinal, Clifford, Culkin, Howes, Kagan, Yazwjian, Byler, Rustici, Zelazo, 1999).
Identifying factors related to high quality have been the subject of various early childhood research
studies (NICHD Early Childhood Care Research Study, 1998; Peisner-Feinberg, et al, 1999). The NICHD study
found that lower staff ratios, higher levels of caregivers education and training are associated with higher
scores of child development (NICHD, Early Childhood Care and Research Study, 1998). Other studies have
emphasized the needs for early literacy activities to be incorporated into early childhood environments
to facilitate language development for future school achievement (Anderson, Hiebert, Scott and
Wilkerson, 1985; Chall, Jacobs, & Baldwin, 1990; Snow and Tabors, 1996). Programs related to basic
cognitive skills (language, and math) as well as children’s behavioral skills in the classroom (thinking/
attention skills, sociability, behavior problems and peer relationships) (Peisner- Feinberg, et.al., 1999).
Finally, the emotional climate of child care classrooms as well as individual children’s relationships with
their teachers are important predictors of children’s outcomes (Peisner-Feinberg, et al, 1999).
Unfortunately, child care programs have many social and economic issues which negatively impact the
quality of care delivered to young children (Bryant & Maxwell, 1997; Fujiura & Yamaka, 2000) Issues such
as children in poverty, welfare reform, increased community violence and the increased number of
children with mental health, physical health and other special needs within early childhood settings
delimits the ability of programs to provide high quality care to all young children (Bryant & Maxwell, 1997;
Center for the Study of Social Policy, 1991; Fujiura & Yamaka, 2000). Hence, despite increased awareness
of the importance of high quality care, there appears to be considerable variability across child care
programs. Quality standards such as staff/child ratios, classroom size and staff training vary from state to
state (NICHD Early Child Care Research Study, 1998). These variations appear to occur more frequently
with child care programs serving younger children. Programs serving older children are more likely to
meet common standards of child care (NICHD Early Childhood Research Study, 1998). Children with
special health/mental health and developmental needs are limited in the access they may have to high
quality inclusive programs (Bailey, McWilliams, Buysee,& Welsey, 1998, Welsey & Keyes, 1998)

3
 IMPORTANCE OF INTERAGENCY PARTNERSHIPS
Improving the quality of early care and education requires reforms that extend beyond a single classroom
or program to community partnerships and linkages (Buysee, Wesley, Skinner, 1999). Researchers and
scholars in all the interdisciplinary fields that emphasize early childhood education advocate for system
changes that enable agencies, schools, and public and private organizations to pool human and financial
resources and to form innovative partnerships for integrated services. Such partnerships are viewed as
the most effective and efficient vehicle to augment their capacity and to integrate their resources to serve
all infants and young children including those at developmental risk or with developmental disabilities and
their families across the early childhood period (0-8 years).
Some of the most promising of these collaborative ventures in community settings has occurred within
the federal Head Start Program and in the school-linked healthcare services and mental health services
movement as well as in various states’ integrated technical assistance networks focusing on young
children (Ramey, 1999; Takanishi & DeLeon, 1994; Melaville & Blank, 1997; Bagnato, 1999). Each of these
programmatic efforts addressed the specialized needs of children with developmental disabilities or
chronic medical and mental health problems.
Despite these few model development efforts, the benefits of the few field-validated University-Hospital-
Community partnerships have not been universally realized in the regular early childhood education
system. Advocates stress the need for broader initiatives for all young children and families and the
professionals who support them (Hurd, Lerner, & Barton, 1999). Three areas of need are most prominent:
(1) continuing professional development training and ongoing consultation for early childhood teachers,
caregivers and administrators; (2) ongoing consultation regarding “best practices” in early care and edu-
cation; and (3) the integration of consultation and services to facilitate the management of young children
with challenging behaviors and special medical and educational needs in regular early childhood settings.

QUALITY EARLY LEARNING AND SCHOOL READINESS

The national debate about preventing school failure for young children at developmental risk has renewed
interest in the quality, cost, efficacy, and outcomes of early care and education programs in the United
States (Bryant & Maxwell, 1997; Christian, Morrison, & Bryant, 1998; Clifford, Peisner-Feinberg, Culking,
Howes, & Kagan, 1998; Gil & Reynolds, 2000; NICHD, 1999). The accumulated research results of thirty
years of studies in early childhood intervention indicate clearly that young children at developmental risk
from impoverished circumstances face progressive declines in their patterns of developmental, behavioral,
and learning skills and an early and continuing future of school failure in the absence of structured early
care and education experiences which can enhance developmental and early school success (Barnett,
1995; Bryant & Maxwell; Campbell & Ramey, 1995; Farran, 2000; Marcon, 1999; Schweinhart & Weikart,
1997).
Unfortunately, much of the debate about the value of early childhood intervention programs for children of
poverty surrounds not the issue of quality intervention, itself, but rather the cost of quality (Clifford, et.al.,
1998). It is clear, but not universally accepted, that comprehensive early care and education programs are
necessary in order to prevent school failure for children at developmental risk, because the cost of such
intensive programs exceeds the typical cost of daycare.

4
Much interest and debate surrounds the issue of accountability and its assessment in early childhood
intervention programs (Bagnato, Neisworth, & Munson, 1997; Meisels, Bickel, Nicholson, Xue, & Atkins-
Burnett., 2001; Meisels, Burnett, et.al., in press;). Advocates in the fields of early childhood and early inter-
vention eschew the tendency to extend downward both the academic standards and traditional testing
methods that pervade school-age practices. It is urgent for the field to conduct research on both assess-
ment and early care and education practices that are developmentally-appropriate and rigorous in docu-
menting child progress and the acquisition of precursor skills or early school success.
Finally, the early childhood fields must present evidence-based research on those elements of early care
and education practice that best promote positive child outcomes, especially for children at developmen-
tal risk and with developmental delays/disabilities (Head Start Bureau, 2000). Two areas of focus are ger-
mane to the current study: the impact of ongoing, onsite consultation and mentoring on program quality
improvements, and the implementation of “best practice” standards to establish and maintain program
quality.

RESEARCH ON EFFECTIVE EARLY CHILDHOOD INTERVENTION

FOR CHILDREN AT DEVELOPMENTAL RISK
Ramey and Ramey (1998) summarized the major experimental studies in the fields of early childhood edu-
cation and early intervention since the early 1970’s that have resulted in measurable beneficial outcomes
for children at developmental risk. From their analysis, they extracted 7 common elements of effective
intervention programs that have been associated with initial and long-term positive outcomes for children
and families. The seven core features are: (1) longitudinal interventions starting in infancy and monitored
through functional benchmarks; (2) intensive, comprehensive, and individualized programs and supports:
(3) integral parent program participation; (4) high program quality and frequent monitoring; (5) direct child
interventions; (6) community-directed programs and integrated services; and (7) follow-through of child
and family supports and program evaluation into the primary grades.

5
 C H A P T E R 2

WHAT IS PITTSBURGH’S EARLY

CHILDHOOD INITIATIVE (ECI)?

A DESCRIPTIVE OVERVIEW
In 1994, the Heinz Endowments, a nationally renowned Pittsburgh philanthropy organized the business,
corporate, agency, and foundation sectors in an ambitious effort to expand quality early care and
education programs and options for nearly 8,000 unserved children in 80 high-risk neighborhoods. The
overarching mission of ECI has been to foster preschool and school success for children of poverty whose
typical retention and special education placement rates in the early grades have ranged between 18 and
40% in Pittsburgh. The Early Childhood Initiative (ECI) has been a unique, collaboratively designed and pri-
vately funded joint venture with various Pittsburgh urban neighborhoods to nurture the development of
diverse early care and education options for young children. Based on the results of both previous local
and national research, a consortium of business, community, and foundation leaders designed the goals,
approach, and expected outcomes (“key performance indicators”) of ECI around a business-oriented
operational plan. Despite revisions to the original business and operational plan necessitated by
transformations in various state and national policies and unanticipated events (e.g., welfare-to-work;
full-day vs. part-day care; infrastructure costs), ECI began enrolling children in September 1998 in diverse
programs in 8 neighborhoods. In 1997, the first ECI participants were 25 children enrolled in the HUD
community of Hawkins Village.
The design and implementation of the ECI plan were based on various core efficacy principles summarized
in previous research (Ramey & Ramey, 1998): On-site consultation and mentoring to promote NAEYC
program quality standards, parent involvement, community-based decision-making, linkages to schools
and other community resources, child-centered programming, and involvement in an ongoing program
evaluation process to improve quality. In order to receive funding, the leadership councils of community
agencies that operated the ECI programs produced cohesive business and operational plans that
demonstrated adherence to these core principles to the ECI Oversight Committee, a council of business,
foundation, university, community, and corporate representatives who sanctioned each plan. ECI was
designed as a six-year longitudinal effort.

6
The following figure provides a schematic of the major components of the Early Childhood Initiative (ECI) model.

Pittsburgh's Private & Public Sector Stakeholders & Leaders:

Foundations, Businesses, Communities, Universities, Schools, Service Agencies

ECI Intervention ECI ECI Evaluation

ECIM
Consultation
ECI
Community SPECS
Programs

The genesis of ECI was the collaboration among the major public and private sector stakeholder groups in
the Pittsburgh region, particularly the Heinz Endowments and the business leaders of the major corpora-
tions in the region. With this nexus, an operational and business plan for ECI was generated; counsel was
provided by the various community leadership groups, including the university, agencies, schools, and
local neighborhood leaders in the urban sectors.
The basic operational plan that was implemented included a management group for ECI entitled “Early
Childhood Initiative Management” with oversight by the stakeholder group. ECIM provided the consulta-
tion to community agencies that created their own ECI programs and networks.
In October of 2000 the Early Childhood Initiative was downsized as the original steward withdrew prema-
turely and precipitously its commitment to the stakeholder consortium. Heinz-funded Early Childhood
Initiatives continue to evolve in Pittsburgh, Erie, York and Lancaster. Several programs previously support-
ed by ECI have maintained operation on a smaller scale.

OUTCOME BENCHMARKS
ECI is conceived as a local “natural experiment” whose ultimate objective is to enroll all unserved children
in high-risk urban neighborhoods into high quality early care and education programs that reflected diverse
options. Thus, the ECI research is a type of population study in which all children are included in the
evaluation and no unserved or “untreated” group was allowed; each child and program is its own control.
The original ECI plan designed by the business-corporate-foundation consortium established several “key
performance indicators (KPIs)” as outcome benchmark categories, culled from the research literature and

7
which reflected a business approach to monitoring goal-attainment. The KPI categories (specific KPIs
were later refined and transformed by the SPECS Evaluation Team) were as follows: Child, family, program,
community, and early school success.

COMMUNITY-BASED DECISION-MAKING AND PROGRAM OPTIONS

Various ECI program options were available in ECI neighborhoods based on local preferences established
by community consensus from the community leadership councils: newly created early care and educa-
tion centers, previously existing providers, Early Head Start/Head Start centers, early literacy programs,
family child care homes, inclusive early childhood and early intervention programs.

COLLABORATIVE CONSULTATION FOR PROGRAM QUALITY

The original ECI operational plan established the policy that program quality as measured by movement
toward NAEYC Standards and eventual accreditation would establish the focal “intervention” yardstick
within and across all ECI program options. Each community program had some autonomy to establish its
own ECI methodology of intervention beyond this criterion. However, in order to monitor adherence to
NAEYC standards and to mentor program administrators and teachers on implementation of best practices,
the ECI oversight group instituted a quality control vehicle. The ECIM team consists of local experts from
county and state government, professional organizations, agencies, and university groups who provided
ongoing weekly supervision, monitoring, and mentoring to the caregivers in each ECI program. ECIM
standardized the quality monitoring by using status and feedback data from independent observational
ratings on the series of early childhood program quality measures published by Harms and Clifford (Early
Childhood Environment Rating Scale [ECERS], Infant/Toddler Rating Scale [ITERS], Family Day Care Rating
Scale [FDRS]). Throughout the three-year operation of ECI, all programs received 2 to 4 hours of ECIM
consultation per month. The collaborative consultation process provided by the ECIM team was conceived
for program evaluation research purposes as the “ECI treatment” and its impact on child progress,
program quality and other benchmarks is being evaluated by the SPECS Evaluation Team. Thus, ECIM
consultation and mentoring is the primary independent variable in the SPECS evaluation research design
that is hypothesized to have an impact on child outcome and other dependent variables.
ECIM implemented an ongoing model of onsite consultation and mentoring for program quality, “collabo-
rative consultation”, based on the functional consultation approach which was field-validated in previous
special education and psychology research. This approach involved a combination of flexible didactic
sessions with teachers and caregivers about specific NAEYC standards, collaborative goal-planning to
improve various programmatic dimensions, modeling of “best practices” by consultants, observations of
caregiver behaviors, collaborative feedback to caregivers and supervisors, and follow-up observations
and documentation. Such a collaborative approach attempts to operationalize the “scaffolding” concept
proposed by Vygotsky (1978) that is posed as essential for learners to proceed to the next level of skill
acquisition and applied knowledge under the guidance of capable mentors (see sample of the
Collaborative Consultation Quality Mentoring model in the Appendix).

8
 CORE ELEMENTS OF THE ECI MODEL
Six overarching programmatic features are common to all ECI program options and distinguish the
ECI “treatment”: (a) ongoing consultation to improve program quality; (b) monitoring regarding the
implementation of NAEYC standards, practices, and eventual accreditation; (c) diverse forms of parent
participation; (d) early care and education routines guided by ongoing child assessments and feedback;
and (e) community-based leadership to marshal creative interagency supports for children and families in
ECI program settings; (f) longitudinal program evaluation for program quality enhancement, child
curriculum planning, and outcomes research.

WHO ARE THE ECI CHILDREN, FAMILIES AND PROGRAMS?

Children and Families
In the current descriptive and statistical study, developmental outcome data on 1350 urban preschool
children and families who participated in the Early Childhood Initiative (ECI) for the longest periods of time
were analyzed. The detailed regression analysis was conducted on n=834 children and families with
complete data sets on all dependent outcome measures. Average length of intervention was 12.3 months
(SD = 9.7; range= 7.4 to 22.6 months). Approximately 86% of the children enrolled in ECI programs were
classified as “at-risk” using Anna E. Casey (1994) guidelines for distressed communities. Developmental
delays were identified in 14% of the ECI children based on the Commonwealth of Pennsylvania Early
Intervention and Special Education Standards (1991) (e.g., 1.5 SD below average or 25% delay of chrono-
logical age in one or more domains) and using pre-intervention developmental rates on the DOCS.
National incidence rates of developmental delay range from three to eight percent (Fujiura & Yamaki, 2000).
Average chronological age of the boys (53%) and girls (47%) enrolled in ECI for this analysis is 3.01 years
(SD = .82 yrs, range= .59- 4.00 yrs.). Ethnic mix of children in the urban ECI communities included
African-American (73.4%), Caucasian (24.3%), and Asian and Hispanic (2.3%) representation. Median
yearly family income for the ECI children is $13,024. Reviews of school district web data within the urban
ECI communities show the following averages: low income (61%); free or reduced-price lunches (68%); and
limited English proficiency (66%).

Programs, Communities, and Caregivers

Each ECI community leadership council made independent decisions about the type of program option
using existing and new providers in their local area. Thus, wide diversity in program options is apparent
within and across the eight ECI neighborhoods of Braddock, Highlands, Hill, Homewood, Steel Valley, South
Side, Wilkinsburg, Sto-Rox, and East Liberty. The summary averages of the proportion of children partici-
pating in each type of early care and education option encompasses Early Head Start and Head Start
settings (36%), inclusive early intervention and early childhood classrooms (34%), family daycare homes
(16%), toddler rooms (12%), and infant rooms (2%).
Teachers and caregivers in ECI programs have a wide range of professional preparation and experience.
Across infant/toddler classrooms, preschool classrooms, and family or group daycare homes, providers
have an average of 5.89 years of experience in early care and education (SD= 5.56, range = 0-27 years).
Sixty-six percent have 0-5 years experience and 21% have 6-10 years of experience (5%= 11-15 years;

9
5%=16-20 years; and 3%= 20+ years). Across all program types, providers show the following ceiling lev-
els of professional preparation: <1%= MS; 30%= BS; 28%= Associate; and 42%=High School. Providers in
family daycare settings showed the lowest levels of preparation: 81%= High School and 19%= Associate.

Summary of Program Quality Data for ECI Settings.

Program quality data for each early care and education setting in ECI that enrolled the children was
collected using the SPECS-designed Program Quality Profile for Early Childhood Settings (PQP) (PQP;
Cook-Kilroy, Bagnato, Smith-Jones & Matesa, 1998). The PQP was developed to operationalize the “best
practice” standards of the revised developmentally-appropriate practices (Bredekamp & Copple, 1997).
Concurrent validity studies with the ECERS and its family of instruments reveal correlation coefficients of
.63 to .91 with an aggregate association of .83 over two evaluation time points.
The PQP is an authentic observational scale that surveys developmentally appropriate practice (DAP)
standards across infant to kindergarten transition settings using a scoring protocol that documents raw
scores reflecting percentages of the total possible score in each of seven domains. The PQP was
completed over two evaluation time points by SPECS team specialists (interrater reliability was r= .89
within a one-week period). PQP domains include curriculum, equipment and materials, physical
environment, teaching and care of children, health, relationship with parents, and staff & administrative
issues. Over a 12-month period, ECI program settings revealed a total percentage change of 48.6 % to
75.5% on the PQP. Thus, over a one-year period of quality consultation, ECI programs (58%) on average
reached and exceeded the cut-off of 70% on the PQP that signifies achievement of NAEYC quality
standards. The ECIM collaborative consultation to programs focused on specific practices in each of
the seven PQP clusters to encourage improvements in quality. ECIM used the ECERS-family of
instruments to guide their consultation with program staff and providers. SPECS used linked data from the
ECERS-family instruments with the PQP as an independent instrument to evaluate longitudinal changes in
program quality.

10
 C H A P T E R 3

WHAT ARE THE EXPECTED SUCCESS

OUTCOMES FOR ECI?

KEY PERFORMANCE INDICATORS

As indicated in the original ECI Business Plan, the SPECS Evaluation Team is responsible for revising and
refining the KPIs that are an integral part of appraising the success of ECI. This second revision of the KPIs
is based on a further refinement of the “benchmarks for success” to make the KPIs more objective,
measurable and sensitive to the missions of ECI. Moreover, this revision is based on the pilot outcomes
from our first year of data collection in 1998. Attainment of all ECI evaluation benchmarks in the 6 Key
Performance Indicator (KPI) categories outlined below will be quantified and monitored by normative (e.g.,
both local and national) or criterion-referenced measures using percentiles, standard scores, goal-
attainment indices (e.g., percentages), and statistically derived profiles that chart the significance of
individual patterns of developmental progress over a 3-5 year time period. In addition, external criteria
such as group consensus measures and extant data from the schools (i.e., kindergarten retention rates;
special education placements) will be used as comparative benchmarks to analyze the efficacy of ECI.
Finally, we are in the process of collecting randomly selected data from non-ECI programs with similar
socioeconomic status (SES) profiles on all six variables in order to derive a “contrast group” for compar-
ative purposes. The final determinations of success will be based on inter- individual and intra-individual
descriptive and statistical analyses, especially developmental growth curve analyses and associated
regression analyses that identify the interrelationships among child, family, program quality and intensity,
community, health/nutrition, early literacy and early school success benchmark clusters.

CHILD BENCHMARKS
1. 85% of the ECI children will enter and succeed in Kindergarten without educational support services.
2. After at least 2 years of continuous ECI programming, ECI children will show at least low average
overall competencies (percentiles of 16-50) in basic developmental skill areas that align with the
Pennsylvania Revised Academic Standards, including problem-solving, expressive and receptive
vocabulary, basic concepts, early literacy (pre-reading, math, writing), socialization, and self-control
behaviors.
3. As a result of participation in ECI programming, ECI children will show individual patterns of progress
in 6 major developmental skill areas that exceed by 50% the expectations projected for them before ECI
programming began.
4. Those children participating in ECI-funded programming for the longest periods of time will show the
greatest progress and the highest success rate in Kindergarten and first grade.
5. 80% of those ECI children showing evidence of developmental delays (difference of 1 Sd below
average) at the beginning of ECI programming will demonstrate average capabilities after, at least,
1.5 years of ECI participation.
6. After 3 years of ECI programming, ECI children will demonstrate significant increases in social skills
and self-control behaviors coupled with significant decreases in problem behaviors.

11
 PARENT/FAMILY BENCHMARKS
1. Increased time participating in their child’s early care and education program and activities.
2. Increased use of positive parenting skills and discipline in the home.
3. Increases in the number of educational materials and toys used in the home.
4. Increased time spent in weekly educational and enrichment activities with their child both in the home
and in the community.
5. Increases in the number of parents reporting and demonstrating improved personal competence in
their play, discipline, teaching, and knowledge of their child’s abilities.
6. Decreases reported in the overall amount of family stress reported as a result of participating in ECI
programming and support services.
7. Increases in the amount of social support reported by parents as a result of participating in ECI
programming and services

PROGRAM BENCHMARKS
1. Improvements in the physical arrangements of the early care and education setting.
2. Increases in provider knowledge of normal child development expectancies and indicators of
developmental and behavioral problems.
3. Improvements in the availability and use of educational materials and toys.
4. Improvements in the use of family-centered methods to increase and enhance parent participation in
the early education program.
5. After 3 years, 85% of ECI programs will consistently use an organized developmentally-appropriate
curriculum of goals and methods to guide teaching, child care, and child progress monitoring.
6. After 3 years of ECIM consultation, 85% of ECI programs will have achieved an overall “high quality
status” ranking on the NAEYC measure of quality benchmarks.
7. After 3 years of ECIM consultation, ECI program administrators will have developed collaborative and
working interagency supports for delivering early intervention, and developmental and behavioral
healthcare services to their children as evidence of a “moderate level” of program intensity.

12
 COMMUNITY BENCHMARKS
1. For 85% of the ECI children transitioning into kindergarten, direct communications will have been
established between the parent, the early childhood provider and the teacher and elementary school
building principal about the child’s strengths and needs and the families priorities for their child.
2. Achievement of a criterion of 75% for the number of neighborhood elementary schools that actively
support ECI goals across the 80 communities and written procedures for a cooperative transition
process between ECI and school entrance.
3. Completion of a collaborative process of social consensus by a focus group of community and
public school leaders and a written document that rank orders the benchmarks for kindergarten
entrance and success.
4. After 5 years of ECI involvement, ECI communities will show a 50% increase in the number of formal
and informal interagency or inter-organizational networks that will have developed to support the ECI
programs in the neighborhoods.
5. After 5 years of ECI involvement, 80% of the neighborhood partners will have reached consensus on
their individual future goals for ECI expansion and a plan for implementation.
6. Parents who participate actively and for the longest periods of time with ECI will show significant
individual increases in their knowledge and use of formal and informal community resources for
social support.
7. By the end of the fifth year of ECI, neighborhood consortia advisory boards will have achieved a
criterion of 70% accomplishment of the goals on their performance plan submitted to ECIM.
8. By the end of the fifth year of ECI, ECI neighborhood consortia advisory boards will demonstrate a
moderately high level on their self-efficacy measures that reveal the strength of neighborhood
confidence, commitment and resources regarding their ECI programs.
9. After 3 years of ECI program operations, ECI neighborhood consortia will report a criterion level of
60% regarding linkages between their program and various formal and informal interagency
supports of their programs.

EARLY SCHOOL SUCCESS BENCHMARKS

1. 85% of ECI children will demonstrate at least low average competencies (16-50%) compared to
national norms on a measure of basic literacy and school skills (BSSI) by the end of their
kindergarten year.
2. 85% of ECI children will reach a criterion of 80% accomplishment of the “kindergarten success”
curricular competencies on the DOCS upon entrance into kindergarten.
3. Less than 10% of ECI children will be identified as needing special education support services at
entry into first grade.
4. Fewer than 15% of ECI children will be retained in Kindergarten compared to historical average
grade retention rates of approximately 25% for school districts associated with ECI communities.

13
 C H A P T E R 4

HOW DOES THE SPECS TEAM

DOCUMENT THE SUCCESS OF ECI?

After a national competition in 1996, the Heinz Endowments and the Early Childhood Initiative Management
Council composed of business, corporate, foundation, and community members chose an interdisciplinary
research team from Children’s Hospital of Pittsburgh and the UCLID Center at the University of Pittsburgh
to conduct the longitudinal evaluation of the Early Childhood Initiative (ECI). The SPECS Team (Scaling
Progress in Early Childhood Settings) is the independent program evaluation and research group funded
by the Heinz Endowments and other funders to evaluate the outcomes of ECI. SPECS is directed by
Dr. Stephen J. Bagnato, Ed.D., a nationally known developmental school psychologist and early childhood
specialist at both Children’s Hospital and the University of Pittsburgh School of Medicine. Dr. Bagnato is
Director of both the Early Childhood Partnerships program at Children’s Hospital and the Developmental
Psychology Training Core for the UCLID Center (University, Community, Leaders, and Individuals with
Disabilities). He is Professor of Pediatrics and Psychology at the University of Pittsburgh School of
Medicine. SPECS has received a continuation grant (in 2001) from Heinz to continue the evaluation
research of the local and state-wide Heinz Early Childhood Initiatives in Pittsburgh, Erie, York, and
Lancaster from 2001 to 2004.
Working in collaboration with ECI communities, The SPECS team is one component of a community-based
consultation, training, and research collaborative (Early Childhood Partnerships) formed by Dr Bagnato
with faculty, staff, and fellows from Children’s Hospital of Pittsburgh and The UCLID Center at the University
of Pittsburgh, an interdisciplinary leadership training institute funded by the federal Bureau of Maternal
and Child Health which specializes in the impact of at-risk circumstances and neurodevelopmental
disabilities on child, family, program, health, and early school success outcomes, as well as community
needs, resources, and supports.
The SPECS Team is conducting a broad-spectrum, longitudinal appraisal of the outcomes and success of the
Heinz Early Childhood Initiatives over a five-year period. ECI has implemented a comprehensive model of
intensive consultation to upgrade the quality and scope of early care and education programs, Head Start,
and early literacy programs that enrolled 1350 high-risk children and families, in 48 early childhood programs,
and 8 communities (25 neighborhoods) throughout Pittsburgh. The central mission of ECI is to prevent school
failure for high-risk children and families by enhancing both their experiences in high quality, comprehensive
early childhood education programs and supportive communities, and their patterns of developmental and
behavioral progress when they enter their kindergarten through third grade school years.
Perhaps the primary long-term objective of The Early Childhood Initiative (ECI) is to function as a catalyst
for the formulation of an integrated, seamless, comprehensive, and interagency partnership system of
early care and education and support services for all high-risk children and families.
The SPECS research design, evaluation, and analysis strategies enable ECI to monitor the impact of their
comprehensive programmatic methods on five categories of interrelated child, family, program, school,
and community outcome benchmarks. SPECS progress benchmarks and measurement strategies match

14
with the Key Performance Indicators of ECI and with the new Pennsylvania Academic Standards. The
SPECS Team uses assessment techniques that rely on the collection of authentic information in natural
home, preschool, and community contexts.

SPECS EVALUATION METHODOLOGY

In July of 1997 after a national competition, the SPECS Evaluation Team of Children’s Hospital of Pittsburgh
and the University, Community, Leaders and Individuals with Disabilities (UCLID) Center at the University of
Pittsburgh under the direction of Dr. Bagnato was awarded the multi-year program evaluation research
grant by the Heinz Endowments to conduct the longitudinal evaluation of the quality, impact, and outcomes
of ECI. The SPECS evaluation model and research design were created based on the following
requirements established by the original ECI business and operational plan:

• Assess all children in ECI programs without exclusions for research purposes.
• Collect data in the child’s natural settings.
• Use assessment and evaluation procedures that are compatible with NAEYC standards.
• Ensure ongoing collaboration among the teachers, caregivers, and supervisors and the SPECS
Evaluation Team in the assessment process.
• Establish longitudinal evaluation procedures that provide both formative and summative feedback
in the major KPI areas to caregivers, parents, administrators, and community leadership teams for
quality improvement purposes.
• Match performance data to the KPIs and the Pennsylvania Academic Standards.
• Use data analysis procedures that will determine those combinations of KPI benchmarks that
account for child progress and early school success.

Based on these parameters, the SPECS Evaluation Team created and implemented a model entitled
Authentic Assessment and Program Evaluation in Early Childhood (Bagnato, 1997, 2000; Bagnato,
Neisworth, & Munson, 1997; Neisworth & Bagnato, 2000).

AUTHENTIC ASSESSMENT AND PROGRAM EVALUATION IN EARLY CHILDHOOD

The SPECS research design and evaluation methodology enable the team to monitor the outcomes and
impact of the comprehensive consultation and mentoring model of ECIM and the ECI programming in each
community using the KPIs as benchmarks. The SPECS Evaluation Team uses assessment and evaluation
strategies that rely on the collection of repeated formative and summative authentic data in the natural
home, preschool, and community contexts. SPECS techniques in the “authentic model” are innovative in
five ways; (1) they match the developmentally-appropriate quality and best practice standards of both
NAEYC (Bredekamp & Copple, 1997), and the Division for Early Childhood of the Council for Exceptional
Children (Sandall, McLean, & Smith, 2000); (2) they emphasize a check-and-balance system of convergent

15
and multi-source observations and appraisals of child developmental and behavioral competencies
overtime in everyday routines by teachers and caregivers who know the child best; (3) they do not rely on
“table-top testing” methods which remove the children from their natural developmental ecology, but
rather more appropriate and natural developmental observations and ratings; (4) they sample and feedback
information about each child’s acquisition of early learning skills within the programs’
developmental curriculum that are teachable and predictive of resiliency and early school success; and (5)
they offer ongoing feedback to teachers, parents, and community leadership council members about
children’s progress and needed enhancements to upgrade program quality and community cohesion.

GENERAL OBJECTIVES AND ORGANIZATION OF THE SPECS RESEARCH

SPECS activities are organized under a two-phase authentic program evaluation research agenda within
five operational strands that are interrelated: Child, Family, Program, Community, and Early School
Success.
PHASE 1 (1997-2000) uses a regression design and contrast group studies to analyze the first 3-year results
and outcomes for ECI. The primary objective is to determine the extent to which participation in quality
early care and education programs for the longest periods of time results in child success in preschool and
school. The SPECS strands and descriptions follow:
• Child Strand... focuses on longitudinal child progress (“developmental growth curves”) and outcomes
for developmental and behavioral competencies (cognitive, language, social, motor, self
care, social skills and self-control behaviors); links established with precursors of child-related
outcomes in early literacy and early school success strand that align with the Pennsylvania
Academic Standards (1997);
• Community Strand…Conducts qualitative studies on the evolution of more cohesive community
resource networks to support each neighborhoods’ ECI program. The focus of these studies is
individual profiles of community self-efficacy and workplan performance and, the extent to which
communities implement creative plans to link formal and informal interagency resources to
enhance their programs for children and families.
• Program Strand…conducts focused research on specific ways that programs increase the
quality, scope and intensity of support services for all children, with a focused emphasis on how
all types of ECI programs (e.g. infant/toddler, family daycare, centers, early literacy groups, and
Early Head Start/Head Starts) include children with developmental delays and challenging
behaviors into their routines.
• Family Strand…profiles parent involvement in their child’s program, parent preferences for and
satisfaction with their various early care and education program arrangements as these relate to
increasing parenting competencies, decreasing parent stress, and greater family social supports.
• Early School Success Strand…targets research outcomes studies in two areas: the impact of
quality early learning on the acquisition of early learning competencies at preschool and the sub-
sequent achievement and success of children in kindergarten through third grade.

PHASE 2 (2001-2004) involves the continuation of SPECS phase 1 research methods; will use path analysis
and structural equation modeling approaches to determine the particular combination of multiple factors
and ECI treatment “dosages’ responsible for child success in the ECI-DP (Demonstration Project for
Wilkinsburg and Braddock) and Heinz Pennsylvania ECIs in Erie, York, Lancaster, and Central PA.
16
 SPECS EVALUATION RESEARCH DESIGN AND SPECIFIC AUTHENTIC MEASURES
SPECS is based on an authentic assessment and program evaluation conceptual framework and associat-
ed methodologies. This unique, non-traditional approach blends norm-based, curriculum-based,
observation-based, and qualitative data collected in natural settings by typical caregivers to provide a
more representative and real-life portrait of progress and outcomes. SPECS accomplishes the general
purposes of formative (quarterly ongoing quality improvement feedback to providers, parents, and the
community) and summative (e.g., yearly progress and impact data regarding child, family, program,
community, health benchmarks).
Exhibit 1 characterizes the conceptual and practical limitations of an inauthentic assessment model in
early childhood research and practice (Bronfenbrenner, 1979) which results in “strange” child behavior
when unfamiliar adults and unfamiliar situations and constricted occasions are the contexts for the child.
In contrast, authentic assessment and evaluation approaches and methods (see Exhibits 2-4) rely on
natural observations in everyday routines; converge data across settings from multiple caregivers; link
program goals, content, and expected outcomes; emphasize intra-individual child progress comparisons
for formative purposes and both normative comparisons for summative analyses; and align with accepted
professional standards for developmentally-appropriate assessment, especially National Association for
the Education of Young Children (NAEYC) and the Division for Early Childhood (DEC) of the Council for
Exceptional Children.

Exhibit 1 Exhibit 2

Inauthentic Assessment in Authentic Assessment

Early Childhood
“Much of developmental psychology (early
childhood assessment) as it now exists is the
science of the strange behavior of children
with strange adults in strange settings for the
briefest possible periods of time.”
(Bronfenbrenner, 1979)
in Early Childhood
• Natural observations in everyday settings
and routines vs. contrived testing
• Convergent, multisource data from caregivers
across settings
• Curriculum-based measures linked to
program goals, content & benchmarks
• Intra-individual child progress & Inter-Individual
normative comparisons
• NAEYC/DEC DAP Assessment Standards

Exhibit 3 Exhibit 4

Developmentally Appropriate Practice: Developmentally Appropriate Practice:

Revised (NAEYC, 1997)—Assessment
• Reliance on developmental observations
• Performance on authentic, not contrived activitties
• Integration of assessment & curriculum
• Adapting teaching for individual needs
• Communication with family
• Evaluation of the program
Revised (NAEYC, 1997)—Assessment
• Content is based on developmental goals
• Decisions based on multiple sources of data
• Teacher, parent, & specialist input
• Child progress based on past performance as
the reference, not group norms
• Goals across all developmental domains
• Ongoing observational assessment

17
The major features of the SPECS Authentic Evaluation Model underscore the use of a longitudinal
multivariate regression design with cross-sectional studies and the use of Hierarchical Linear Modelling
statistical analysis methods (HLM) or developmental growth curve analyses coupled with the use of a
modified-treatment contrast group comparison. The modified-treatment group consists of family daycare
and early childhood center programs and children that are not enrolled in the ECI venture and thus do not
receive enhanced ECIM consultation and mentoring for its staff (the ECI “treatment”). Children enrolled in
the contrast group share the same demographics as their ECI counterparts, comprise a group that is still
enrolled in other early care and education programs and are not randomly assigned, and therefore, cannot
be considered a true control group.
The SPECS Conceptual Design Model serves as the underpinning for the longitudinal, repeated measures
design using both HLM analyses and contrast group comparisons (Exhibit 5). In this model, ECIM
consultation/mentoring afforded to ECI program administrators and providers is the “treatment”. ECIM-
enhanced programming is hypothesized to exert a direct effect on program quality, particularly teacher-
child interactions and other quality dimensions. A direct effect on community participation with the
program or cohesion is also hypothesized. In contrast, the program (teachers and staff) will exert a direct
effect on family participation, child developmental/behavioral progress, learning, and school success.
Families and program staff will implement policies and practices that may affect child health and nutrition.
The HLM analysis strategies will use sophisticated regression techniques to determine the interrelation-
ships among these hypothesized effects involving child developmental growth curves-child progress
and later school success- associated with the impact of ECIM-enhanced programming on increases in
program quality/intensity, family participation, community cohesion, and health/nutrition status.

Exhibit 5

SPECS Evaluation Conceptual Model

Community Family
Health &
Participation Participation
Nutrition
Status

ECI
Consultation

Learning &
Program Quality
School
& Intensity
Competence

18
The outcomes of previous national research in both early childhood and early intervention supports this
conceptual “transactional” model of the potential multiple ecological effects on child developmental
progress from intervention programs. The most notable recent example is the NICHD study on child care
outcomes (1998) that used the same type of transactional model. Ramey and Ramey (1998) proposed 6
“developmental priming mechanisms” as the constant features in all the most successful early childhood
intervention programs (see Exhibits 6-9). These results were used as benchmarks to evaluate the impact,
quality, and outcomes/efficacy of the Early Childhood Initiative (ECI) and directly influenced the design of
our SPECS program evaluation research and the selection of dependent measures and expected outcomes
expressed as KPIs or evaluation benchmarks. The six constant features included: (1) the long-term bene-
fits of earlier interventions; (2) the lasting effects of more intensive interventions in which children and par-
ents were interactive agents; (3) the greater benefits of direct teaching programs on child progress; (4) the
necessity for more comprehensive and integrated and flexible intervention programs; (5) the need to tailor
interventions for individual children; and (6) the need to continue some supportive interventions overtime
and into the school years.

Exhibit 6 Exhibit 7

Early Childhood Intervention Early Childhood Intervention

Principles: Benchmarks for Evaluation Principles: Benchmarks for Evaluation
Critical “developmental priming mechanisms” for early chilhood • 3. Program Quality
intervention programs associated with positive child outcomes
- Children in high quality programs demonstrate higher
(Ramey & Ramey,1998; NICHD, 1998)
levels of cognitive, social, and language skills and better
• 1. Developmental Timing
school readiness and success than children in low quality
- Earlier and longer interventions have greater benefits.
programs; the more standards met, the better children did.
• 2. Program Intensity & Parent Participation
- Low quality programs in either home-based or
- More intensive programs (time, visits, sessions) produce
center-based settings were associated with poorer
larger positive outcomes than less intense programs.
developmental and school performance.
- More active and involved children and families show the
greatest developmental progress.

Exhibit 8 Exhibit 9

Early Childhood Intervention Early Childhood Intervention

Principles: Benchmarks for Evaluation
• 4. Direct vs. Indirect Learning Experiences
- Direct child teaching programs provide more enduring
benefits than parent training only programs.
• 5. Program Breadth & Flexibility
- Programs with more comprehensive, integrated, and
community-directed services and multiple modes of
fostering child and family progress produce larger
effects than narrow focus programs.
Principles: Benchmarks for Evaluation
• 6. Individual Differences & Program Impacts
- Children with low risk status show more benefits from
early intervention than children with higher risk.
- Different programs and supports are needed to produce
similar benefits for high risk children/parents.
• 7. Ecological Dominion & Envirnnmental
Maintenance of Development
- Positive effects of early intervention diminish with time in
the absence of continued environmental supports for
learning in school for high risk children.

19
SPECS uses a combination of both formative and summative evaluation strategies to document progress
and outcomes (Exhibit 10). As will be detailed in the Data Collection section, SPECS Evaluation Team
members collect assessment data from providers, parents, and community consortia representatives on a
repeated measures basis, primarily 2 to 3 times per year. Child status and progress data using specific
developmental and behavioral measures are collected quarterly (e.g., September, January, May). After
collection, and trouble-shooting checks on reliability, accuracy, and completeness, the SPECS Child Strand
Coordinator with the Information Database Manager generates computerized feedback letters for
teachers and parents written from the child’s viewpoint and voice (see Exhibit 11). These individualized
letters detail the child’s strengths and readiness for the next steps in early learning (i.e., specific goals from
the DOCS assessment measure). The teachers and parents then infuse these experiences into the child’s
typical daily routine at preschool and in the home to enhance opportunities for changes in learning and
behavior. The letters serve a curricular purpose of sensitizing the caregivers to specific developmental
competencies that match the expected ECI outcomes. In addition, twice each year the summative results
are communicated in group meetings to teachers and community consortia board members to provide
them with information on apparent changes in the quality of their programs and the changes in children for
continuous program improvement purposes. The reliability and validity of caregivers’ ongoing assessments
are ensured in two ways:
(1) SPECS team members supervise caregiver assessments during a set week each quarter when
individual modelling occurs and consultation is given. A series of three initial, booster, and folow up
training sessions with each teacher occurs until the reach mastery on observation and recording of
skills observed.
(2) Once each year, SPECS specialists conduct random, traditional, but authentic assessments on
individual children as a concurrent validation of teacher/parent assessments (i.e., Battelle
Developmental Inventory)

20
Exhibit 10 Exhibit 11 - Developmental Feedback Letter

SPECS
T1 AUTHENTIC ASSESSMENT T10 KEISHA DAVIS
FORMATIVE
EVALUATION HOW I AM LEARNING AND GROWING
JANUARY 1998 ASSESSMENT

Dear Teacher,
QUARTERLY COMPUTER FEEDBACK LETTERS: INDIVIDUAL GOAL PLANNING I am learning things all the time. Let’s go over what I’ve been
learning lately. Well, I’m able to do lots and lots of things just like
other kids my age. Here are some things from my last evaluation
that I’ve been learning and some things I want to learn next.
T1 QUARTERLY PROGRESS REASSESSMENTS & FEEDBACK T10
SUMMATIVE
Some Things I’m Learning To Do
September January May • speak in complete short sentences
• take turn with toys
• ask questions using words such as “who”, “what”, or “where”
• count two to three objects (saying the number as I touch the
object)
Exhibit 12 • draw a circle and a cross (0, +)
• share toys while playing with other children without being asked
• play a simple board game
• cooperate while playing with others
SPECS: Purposes & Goals • take turns in a conversation with others
• name a few letters of the alphabet
• Conduct authentic evaluation via caregiver • follow a three-part instruction
• Meet NAEYC DAP Standards
• Link assessment, teaching, progress/program Some Things I Want to Do Next
monitoring & evaluation • repeat from memory a few simple nursery rhyymes, prayers, or
• Implement ecological evaluation scheme songs
• Provide formative feedback and ongoing training to • correctly give my first and last name when asked
teachers, parents, & community about evaluation • use pronouns correctly
methods and progress • tell another person when I am happy, sad, afraid, or angry
• Ensure legacy for providers after ECI • use time words such as morning, afternoon, and nighttime
• count by rote from one to ten

DEPENDENT MEASURES, DATA COLLECTION PROTOCOL, AND TIMELINE

SPECS Evaluation strategies are organized currently according to the 5 outcome/benchmark areas or
“strands”: Child, Family, Program, Community, and Early School Success. The SPECS evaluation research
battery reflects the convergent and multisource approach described in the authentic assessment and pro-
gram evaluation model in the previous section. The following sections illustrate the content of the specif-
ic dependent measures that comprise this battery by strand area. The battery below is consistently used
in the core ECI research studies. The following section will provide specific information on the primary
measures in each strand (see sample measures in Appendix).

Child Strand
The Developmental Observation Checklist System (DOCS; Hresko, etal, 1994) is the primary child develop-
ment observation assessment instrument used in the SPECS battery (see Exhibit 13-14). DOCS is a unique
norm-based measure whose 475 developmental competencies are naturally-occurring child skills (e.g.,
recognizes the McDonald’s sign; finds the correct toy at the bottom of the toy box; can read simple signs)
which are teachable curricular competencies and predictive of school success and sensitive to the effects
of intervention. DOCS samples child skills in the following domains: Cognitive, Language, Social, Motor, and

21
Overall. DOCS was nationally (33 states) normed on nearly 1,100 children from birth to 6 years of age.
Concurrent, criterion, and predictive validity data are strong (.78-.94) as well as interobserver (parent-
teacher= .81) and rating-rerating reliability (.95). Our first 16 months of SPECS/ECI data on 910 children
reconfirms and provides additional support for the valid use of DOCS with a low SES and high-risk population.
For the current study, statistical analyses were conducted on the graduated scoring of the DOCS (e.g., 0= No-
not achieved; 1= Sometimes-getting there or emerging skill; and 2= Yes-fully achieved). Descriptive standard
score data reflect the national norms based only on the total “Yes” scores in each domain.
The manual for the DOCS shows concurrent validity data and interrater reliability data. Interrater
reliability data shows strong overall interrcorrelations between parents and teacher/caregivers (r =. 94).
Concurrent validity studies with various traditional measures of language, developmental, and cognitive
skills show moderate to high interrelationships (rs= .35 to .83). Since the DOCS scoring scheme was
modified for the purpose of the ECI statistical analysis, the SPECS team conducted generalizability
analyses on the DOCS to determine the reliability of the composite scores under the modified scoring
scheme. Based on the data from a sample of 90 ECI children rated by their caregivers who had been
trained in the use of DOCS, the G-coefficient was documented to be .972.

Summary
The DOCS is a comprehensive developmental assessment system composed of 475 items covering the birth
to 6 year age range, and normed on nearly 1100 children in 30 states. Developmental competencies on the
DOCS are organized into five major functional domains: cognitive, language, social, and motor, and an
overall developmental level. Raw scores and normative (percentiles, and standard scores: M= 100; SD= 15)
and age scores are derived in the five domains. The format of the measure recognizes the interactive or
transactional nature of development and the interrelatedness of child functioning among the domains
(i.e., language and social skills), and allows scoring of skills in more than one domain. Only the DOCS total
raw score was used in the current analysis1.
The DOCS uses an observational strategy of observing and recording (Yes, No) the natural occurrence of
“authentic” developmental skills reflective of each domain (i.e., recognizes the McDonald’s sign; finds the
right toy hidden in the toy box; opens containers to get something inside; identifies the letter B; follows
rules in group games). All DOCS items are important and teachable functional skills that align with content
in the developmental curricula of most programs.
SPECS Evaluation Team members have produced (with publisher permission) a computerized version of the
DOCS with simplified observational and recording forms that include an additional emerging skill (“getting
there”) scoring category for curricular planning and feedback purposes. Integral to the computerized
DOCS format is a letter written in the “child’s voice” (“Hi Mrs. Jones; I’m Jimmy; I’m 3 years old; next time
I need help doing these things”) that summarizes for the teacher and parent the achievements and next
step skills for the child. The SPECS Team generates such individualized letters for each child for the
teacher and parent each quarter (e.g., September, January, May).

22
Exhibit 13 Exhibit 14

Developmental Observation DOCS Sample Items

Checklist System (DOCS) • Uses 2-3 word phrases (2-3 yrs.)
• Tries to say nursery rhymes (2-3 yrs.)
• Norms = 1,094 children (0-60 mo.) in 30 states
• Recognizes McDonald’s sign (3-4 yrs.)
• Norm/curricular hybrid; authentic observations
• Counts 2-3 objects (3-4 yrs.)
• Expects an activity before it happens (0-11 mo.)
• Pretend play with common objects (3-4 yrs.)
• Repeats own behavior when has effect (0-11 mo.)
• Writes few letters of alphabet (4-5 yrs.)
• Talks in own language (0-11 mo.)
• Names 10 printed letters (4-5 yrs.)
• Responds to “come here” (12-23 mo.)
• Knows how many pieces cut in half (4-5 yrs.)
• Understands meaning of 10 words (12-23 mo.)
• Sits and listens to stories (12-23 mo.)
• Tries to describe something that happened (2-3 yr.)

The Preschool and Kindergarten Behavior Scales (PKBS; Merrell, 1994) is an observation rating scale that
emphasizes the focused assessment of social skills and problem behaviors in young children from 3 to 6
years (see Exhibit 15). The PKBS stresses social and self-control behaviors that are sensitive to the effects
of intervention and predictive of early school success (e.g., shares toys and other belongings; waits, takes
turns; plays with several different children; attempts new tasks before looking for help). PKBS is a
norm-based scale whose behaviors are curricular or instructional in content. It was nationally normed on
2,855 children and is appropriate for a variety of evaluative and clinical purposes. Validity and reliability
data on the PKBS are the strongest of any currently available preschool measure of social skills and
behavior (.81-.98).

Exhibit 15

Preschool and Kindergarten

Behavior Scale (PKBS)
• Norrns = 2,855 children (3-6 yrs.)
• Social Skills and Problem Behaviors
• Sits and listens to stories
• Shares toys and belongings
• Attempts new tasks before asking for help
• Takes turns and plays cooperatively
• Defies parent, teacher, or caregiver
• Acts impulsively without thinking

23
The Basic School Skills Inventory (BSSI-R; Hammill, 1998) is a norm-based curricular measure of early
learning and basic competencies that are predictive of school success (see Exhibits 16-19). The BSSI-R is
completed by teachers based on their observation, knowledge of children, and reviews of the childrens’
work performance and portfolios. The scale samples preacademic and academic skills is such areas as
reading, math, spoken language, writing, classroom behavior, daily living skills, and social skills. The
BSSI-R was normed nationally on over 1,000 children; its shows reliability and validity data that is adequate
for evaluative purposes (.64-.93).

Exhibit 16 Exhibit 17

Basic School Skills Inventory (BSSI) BSSI Subscale Samples

• Learning readiness skills for children
• Authentic teacher observational ratings
• Ages: 48 to 107 months (Pre-3rd grade)
• 6 Domains: Spoken Language; Reading; Writing;
Math; Behavior; Daily Living
• Standard and T-Scores (100/15;50/10)
• Functional skills/benchmarks for learning
• Graduated scoring; 0,1,2,3 (mastery)
• Norms = 757 children; 5 states
• PRO-ED
Spoken Language
• Uses complete sentences when talking
• Listens to and retells a story in sequence
• Initiates and maintains conversations with others
Reading
• Recognizes upper/lover case letters
• Names letters when sounds are spoken
• Has basic site vocabulary of 5 words

Exhibit 18 Exhibit 19

BSSI Subscale Samples BSSI Subscale Samples

Writing Classroom Behavior
• Writes from left to right
• Writes first name without a model • Makes friends easily
• Writes single letters when asked (b,h,m,t,a,e) • Takes turns
• Yses teacher feedback to improve learning
Mathematics • Can attend to activity for 5 minutes
• Counts objects in set of fewer than 10
• Counts aloud from 1-20 Daily Living Skills
• Understands concepts of 1st, 2nd, 3rd • Enters and exits school by self
• Assumes responsibility for own belongings

24
Family Strand
The Parent Behavior Checklist –Short Form PBC; Fox, 1995) is a parent-completed report scale designed
to determine parenting skills and knowledge and beliefs (20 items) in three core areas: Expectations (child
development); Nurturing (child care, interactions, teaching); and Discipline (behavior management)
(see Exhibits 20-23). The PBC was nationally normed on 3,000 parents and includes competencies that
are amenable to parent education and support. Validity and reliability data are moderately strong and
sufficient for evaluative purposes.

Exhibit 20 Exhibit 21

Parent Behavior Checklist (PBC) PBC Subscale Samples

• Authentic observations of parenting behavior Expectations
• Ages: 12-59 months
• Norms = 1,,140 children & mothers & fathers; • My child should be able to use toilet w/o help.
subsample of 70 with delays • My child should be able to ride a tricycle.
• 2.9 grade reading level • My child should be old enough to share toys.
• Parent-child interactions: 3 subscales:
Nurturing; Expectations; Discipline
• Graduated scoring: 4-1 (Always-Never)
• PRO-ED

Exhibit 22 Exhibit 23

PBC Subscale Samples PBC Subscale Samples

Discipline Nurturing
• I would spank my child in public for bad behavior. • I play make-believe with my child.
• I yell at my child for whining. • I read to my child at least once a week.
• I send my child to bed as punishment. • I praise my child for learning new things.

The Parenting Stress Index- Short Form (PSI; Abidin, 1995) is a parent self-report measure of perceived
parent, child, and family stress (see Exhibit 24). The PSI has been validated in numerous research and
clinical studies with disadvantaged and disabled populations. Stress indexes are generated in the areas
of Total, Child, Parent, and Life Stress (e.g., my child is so active that I am exhausted; I feel trapped by my
responsibilities as a parent).

25
Exhibit 24

1. When my child wants something, my child usually keeps trying to get it.
2. My child is so active that it exhausts me.
3. My child appears disorganized and is easily distracted.
4. Compared to most, my child has more difficulty concentrated
and paying attention.
5. My child will often stay occupied with a toy for more than 10 minutes.
6. My child wanders away much more than I expected.
7. My child is much more active than I expected.
8. My child squirms and kicks a great deal when being dressed or bathed.
9. My child can be easily distracted from wanting something.
10. My child rarely does things for me that make me feel good.
11. Most times I feel that my child likes me and wants to be close to me.

Family Support Scale (FSS; Dunst & Trivette, 1992) offers a survey of parent perceptions regarding the type
and extent of personal and social support that is provided to parents of children with special needs.

Program Strand
The Early Childhood Environment Rating Scale-Revised (ECERS; Harms & Clifford, 1998) and Infant
/Toddler Environment Rating Scale, and Family Daycare Environment Rating Scale. The ECERS, ITERS,
and FDRS system of program quality scales have a long and distinguished history in early care and
education. They are the most widely used scales for evaluative, and quality feedback purposes. The
ECERS system is criterion-referenced and is supported by extensive field research. Areas sampled by
direct, naturalistic observation include: personal care routines, furnishing and display for children,
language-reasoning experiences, fine-gross motor activities, creative activities, social development, and
adult needs.
Program Quality Profile for Early Childhood Settings (Cook-Kilroy & Bagnato, 1997) offers a graduated
observation rating scale of status and changes in the quality of various domains of program elements
including the responsivity of the learning environment, teacher-child interactions, parent participation,
curriculum planning, accommodations for special needs, the physical setting. The PQP was designed for
all types of early childhood settings and is based on the NAEYC revised standards.

Community Strand
Community Evaluation Battery: Efficacy Scales, Performance Scale (McClomb, 1997) The CEB is a SPECS-
developed evaluation system designed to document the capacity of community consortia board members
to organize their resources and efforts to accomplish workplan goals, to accomplish their goals with a
sense of personal and collective confidence and effectiveness, and to support their ECI programs. The
scale has been field-validated on 28 communities in high-risk neighborhoods using data over 2 timepoints
with 68 individuals.

26
Evaluation Timeline
The repeated measures timeline for the SPECS evaluation protocol is summarized in Exhibits 28-30. All
observation and rating scale assessments in the SPECS evaluation battery are repeated 2-3 times per
year based on the time frame established in each strand. Child assessments recur each September,
January, and May. Repeated assessments for each child form a “developmental growth curve” for indi-
viduals and groups that are analyzed using the Hierarchical Linear Modeling (HLM) and related statistical
techniques.

Exhibit 28 Exhibit 29

SPECS Evaluation Protocol and Timeline (1997-2002) SPECS Evaluation Protocol and Timeline (1997-2002)
MEASURE/DOMAINS TIMELINE MEASURE/DOMAINS TIMELINE
Sept Jan May Sept Jan May
Child Strand Program Strand
• DOCS X X X • PQPECS X X
• PKBS/SEEC/TRIAD X X X • Program Intensity X X
• D-Specs/Infant-Specs X X X • Consultation-Intensity X X
Parent/Family Strand Community Process Strand
• PBC X X • Individual Efficacy Survey X X
• PSI X X • Organizational Capacity & Performance X X
• FSS X X • Extant health records/improvements X X
Health/Nutrition Strand
• Extant health records/improvements X X

Exhibit 30

SPECS Evaluation Protocol:

K-1 Achievements

CHILD DOMAIN 3rd Quarter

• BSSI-R X
• Group Achievement Tests X
• ECI Key Performance Indicators X
• PA Academics Standards X

Over the six year longitudinal study, a maximum of 18 time points are projected for children who entered
ECI as infants. Based on current data for the first 33 months of ECI operation, children on average are
expected to enroll in ECI at approximately 3 years of age and will be assessed over 6-9 time points.
Assessments in the Family, Program, and Community strands are conducted 1-2 times per year and align
with the child assessments during those periods. The array of subdomains sampled by SPECS evaluators
are outlined in the exhibits (31-33).

27
Exhibit 31 Exhibit 32

SPECS Primary Domains: Dependent Variables
Child Strand
• Cognition/Basic Concepts
• Language
• Social-Emotional
• Motor
• Physical/Sensory
• Self-Regulation
• Social Skills
• Problem Behaviors
• Reading/Writing
• Mathematics
• Self=Care
SPECS Primary Domains: Dependent Variables
Parent/Family Strand
• Parent Expectations
• Discipline
• Nurturing/Teaching
• Stress
• Social Support
• Child Development
• Child Behavior
Program Strand
• C urriculum Planning
• Responsive Learning
Environment
• Developing Relationships
with Children & Parents
• Health, Nutrition, Safety
• Staff & Administration
• Program Intensity
• Consultation Support & Intensity
• Reading/Writing
• Mathematics
• Self=Care
Health and Nutrition Strand
• Immunizations
• Primary Care Access
• Health Insurance
• Diet/Nutrition
Knowledge
• Universal Health
Precautions
• Anemia & TB
Screenings

Exhibit 33

SPECS Primary Domains: Dependent Variables

Community Process Strand

• Individual Efficacy: Personal Goals,

Expectations, Perseptions@ ECI Successin Neighborhood
• Group Organizational Efficacy: Coalition Goals,
Expectations, Perceptions @ ECI Success in Neighborhood
• Indiviaual Neighborhood Strategies, Capacity, & Goal
Attainment Performance in ECI Implementation
• Mapping Community Resource Networks

CONFIDENTIALITY OF DATA AND SAFEGUARDS

The SPECS evaluation proposal was originally reviewed and accepted for implementation by the Human
Rights Committee (HRC) of Children’s Hospital of Pittsburgh in 1997. Yearly renewals after interim progress
reviews of the original HRC proposal have been conducted since 1997.
The original consent form for SPECS was constructed in collaboration with ECIM so that the language of
our hospital-based consent was simple and non-threatening for parents and was compatible with the
overall ECI consent from the steward. The consent form acknowledges that individual child information
will be shared with parents only and with their teachers with their agreement in the case of the individual
computer-generated letters from DOCS. To safeguard confidentiality, all child, family, and other SPECS
strand data are numbered by child/family or community or program so that individual identifiers are secure.
The R-Client software package and database maintains this security except when parents request
individually generated child data for advocacy purposes (i..e., to help parents secure child access to
early intervention support services due to suspected delays). All data is aggregated for statistical analy-
ses and individual community data is identified only by number that is known only to the Information
Database Manager and Principal Investigator (SJB).

28
 C H A P T E R 5

WHAT ARE THE RESEARCH DESIGN, QUESTIONS, AND

ANALYSIS METHODS FOR THE ECI EVALUATION?

RESEARCH DESIGN, QUESTIONS, HYPOTHESES AND VARIABLES

As stated previously, the SPECS research design is a longitudinal regression scheme and analysis comple-
mented by a constructed comparison group analysis, and a contrast group analysis. The SPECS research
design poses questions and hypotheses based on variables that were identified by the original ECI
Management (ECIM) Group and refined by the SPECS team. Two sets of Program Evaluation Research
Benchmarks serve as the criteria for monitoring impact, quality, outcomes, and efficacy: Key Performance
Indicators and Research Questions.

RESEARCH QUESTIONS
The following research questions guided the current first-phase analysis of ECI: 1) Are child developmen-
tal progress and enhanced developmental trajectories associated with participation in high quality (i.e.,
NAEYC standards) early care and education programs? 2) Is the ECIM collaborative consultation and men-
toring approach focusing on NAEYC standards associated with enhanced child development outcomes?
First of all, Exhibits 31-33 provide an overview of the research variables that underlie the SPECS/ECI
evaluation research in the benchmark outcome categories of Child, Family, Program, Community, and Early
School Success, and and various transactional hypotheses. For example, targeted child variables include
language and social/self-control behavior skills; family targets include participation in their child’s program
and parenting behaviors; program quality variables stress curriculum implementation and individualization
of care and education; community variables stress cohesion and group effectiveness in decision-making
and work performance. School transition will target success indicators such as lower retention rates or
special education rates. As indicated below, The HLM longitudinal analysis will target interrelationships
among variables such as hypothesized effects among high program quality, high parent participation,
supportive community networks, and greater child success in preschool and at kindergarten entrance.
Related to these variables, the following overarching research questions guide the SPECS evaluation of ECI
and the HLM longitudinal analyses.

29
SPECS Research Questions

• Does ECI-enhanced programming result in positive child developmental, behavioral, and early school
success outcomes that exceed maturational expectations?
• Does ECI-enhanced quality mentoring result in significant programmatic improvements in ECI
programs?
• If ECI children show positive developmental and behavioral progress patterns that exceed
maturational expectancies and greater school success, what specific programmatic, family, and
community dimensions predict these effects?

DATA PROCESSING AND STATISTICAL ANALYSIS

The Early Childhood Partnerships evaluation database (R-Client for Children) developed by Great Lakes
Behavioral Research Institute, is a windows-based client management information system that includes
data file storage and retrieval, reporting, and file transfer capabilities. The following data are collected and
maintained:
• Identifying data, basic demographic data
• Evaluation data
• Administrative data
• Dates related to events
Data are maintained in the database that is housed in a secure (password access) network drive on the
Children’s Hospital of Pittsburgh server. The SPECS evaluation team can access information on the
database via standard reports or customized reports written by the team’s Information Manager. Children’s
Hospital of Pittsburgh Computer Information System (CIS) is responsible for daily backup of information on
the designated drive. In addition, the SPECS Team Information Manager creates a backup on the local
drive of the SPECS Team computer designated for the database.
Data sets are exported in ASCII file format and given to the statistician for statistical analysis. All data sets
exported for analysis are stripped of identifying information. A case number is assigned to each client
registered in the database. All programs and neighborhoods are given codes. All coded data fields have
built in verification. All collected data forms are reviewed by SPECS team researchers before data entry.
Processed data are also reviewed by the researchers for accuracy. Missing and incomplete data are
recorded in the database.

STATISTICAL ANALYSIS
For the overall analytic design of SPECS, there are basically five categories of potential analyses: 1) Initial
psychometric analyses of assessment procedures through the Generalizability method and other validation
processes as appropriate; 2) if feasible, initial optimization analyses to identify the most cost-effective
combination of assessment methods; 3) second though fifth year annual ordinary least square regressions
to evaluate the impact of ECI on programs, families, and communities; and the constructed comparison
group analysis; 4) second through fifth year annual HLM analyses to evaluate the impact of ECI, programs,
families, and communities on child outcomes; 5) second through fifth year annual HLM analyses to evalu-
ate the impact of ECI, programs, families, and communities on child growth.

30
1. Generalizability Theory and Analysis
An important potential threat to the validity of any of the outcome measures is the existence of a large
amount of random measurement errors which reduce the validity of the outcomes measures, it can
obscure the observation of impact of ECI on program, family, community, and child outcomes; much in the
same way that a great deal of static noise on the radio drowns out the music we wish to hear. Therefore,
it is important to identify the potential sources of measurement errors early on and take steps to minimize
them. The Generalizability analyses are designed to identify potential sources of measurement error and
to evaluate the reliability and validity of any assessment procedure. This method is different from the more
common classical reliability analysis methods (e.g., Cronbach alpha, test-retest reliability) in that the
classical methods are appropriate only for standardized objective tests. The Generalizability approach
can accommodate these standardized testing situations as well as other more complex naturalistic
observational assessment procedures, including the multi-faceted, authentic, and observational perform-
ance assessments, judgment-based assessment, and convergent assessments used in this project.
Through this analytic methods, we seek to identify the potential sources of error and the reliability for e
each of the outcome measures proposed, identify the sources of error and reliability for other potential
alternative measures, and devise convergent evidence of validity for these measures. Through these
analyses, we examine the relative quality of different methods assessing program, family, community, and
child outcome measures.

2. Optimization
In general, assessment is a sampling, probing process. The more performances we sample, the more
raters used, the more items and dimensions we explore and so on, the better the quality (reliability and
validity) of the assessment outcome measures. A classical expression of this idea for objective testing is
something called a Spearman-Brown prophecy which essentially states that in a standardized objective
test, the more items there are in a test, the more reliability the overall score. However, the more items,
more raters, and the more performances would also imply the higher cost of assessment in terms of
expense, rater time, child time spent on assessment, development time, and so on. There needs to be a
balance somewhere to gain the optimal quality of data at the lowest possible cost. In a complex
multidimensional assessment setting such as the convergent assessment used in this project, reliability
and validity can be maximized through many different ways: increasing the number of raters, using
composite scores only, increasing the number of probes, and so on. Each of these strategies will imply
different costs and each will lead to a different improvement of the quality of data. If it is feasible to
gather all the necessary data, a series of optimization analyses (most likely using a method called the
branch-and-bound integer programming method) can be conducted to final the optimal assessment design
that will yield outcomes measures with the best possible and validity while keeping cost to a minimum.

3. Ordinary Least Square Regressions

In an experimental design, the data analysis procedure often follows logically the data collection design.
The data analysis procedure is also divided into two aspects: 1) describing some impact or relationship
and 2) ruling out random chance as the explanation for the observed impact or relationship. The former
generally involves a descriptive statistical analysis and the latter and inferential statistical analysis.

31
A classical true experimental design is a data collection design and not a data analysis design. In this
design, the researcher deliberately manipulates a treatment variable such that some randomly selected
subjects obtain the treatment while others do not. The most common descriptive data analysis procedure
following this data collection design is to compare the mean scores on the outcome measure between
those who receive that treatment and those who do not receive that treatment. The most common infer-
ential data analysis procedure for the experimental data collection design is either a t-test or an analysis
of variance. That is, if the ECI evaluation project is to use a true experimental data collection design, some
randomly selected children would receive ECI treatment while others do not. We would then most likely
analyze that data by reporting the means and standard deviations of the experimental and control group
subjects so that we can conclude which group scores higher than another group. We would also rule out
chance by conducting t-tests for those comparisons involving only treatment vs. control groups and by
conducting analyses of variance for those comparisons involving several different treatments (e.g., the four
different types of early childhood programs).
The ECI project will not be a true experiment because subjects cannot be randomly assigned – children are
already in programs, communities, and families. We may randomly select programs but not children in the
programs to implement treatments. That is, once a program is chosen to receive ECI, ALL children in the
program will receive ECI with no within-program random assignment. In other words, we have a cluster
sampling situation. Also programs, families, and communities are not randomly chosen to receive or not
receive treatment. Rather, they are self-selected by proposing a particular set of activities. Therefore, we
do not have a classical true experimental data collection design in that there is not control or manipulation
of treatment on the part of the researcher and that there is no randomization of programs and subjects.
This situation is best described as a quasi-experimental design in that there are comparison groups.
Because of the lack of randomization, we are not sure if there are any initial differences in program out-
come measures, family outcome measures, community outcome measures, or child outcome measures
between ECI and non-ECI groups. The most often used inferential data analytic procedure for this situa-
tion is an analysis of covariance, which essentially examines whether we can rule out chance after
accounting for initial differences statistically. The corresponding descriptive data analysis procedure is to
compare the adjusted means scores of the ECI and the non-ECI groups; after adjusting for initial differ-
ences.
Overall, for inferential data analysis, t-test is a special case of analysis of variance and analysis of variance
is a special case of analysis of covariance. Similarly, for descriptive data analysis, comparing two means
is a special case of comparing more than two means and comparing unadjusted means is a special case
of comparing adjusted means.
The inferential data analytic portion of regression is primarily an analysis of variance (which may accom-
modate several treatment or predictor variables and several covariates or initial difference variables
simultaneously if desired). In other words, the inferential data analytic portion of regression is simply a
more flexible generalization version of what we use (i.e., t-test, ANOVA, ANCOVA, RMANOVA) in classical
true experimental designs or quasi-experimental designs. The descriptive data analytic portion of regres-
sion is primarily a prediction equation through which we define how the changes in the predictor variables
will lead to changes in the outcome (called criterion) variable. This again is basically an extension of the
descriptive data analytic procedure we use may describe the outcomes in this fashion: The experimental
group has a mean of X1 and the control group has a mean of X2. Alternatively, we can say the same thing

32
through: If you are in the experimental group, I expect your outcomes scores to be X1 and if you are in
the control group, I expect your outcomes score to be X 2. An analysis of variance is no more than an
extension if this: If you are in group A, I expect X1 ; in group B, I expect X2 and in group C I expect X3 and
so on. Regression is a further extension of this in that we can predict outcomes in the form of an equation
that may involve more than 1 treatment or predictor variable. Thus, we would say things like: If your
income is at this level, and you have receive Head Start, and that your parents are highly involved, I expect
X1 . On the other hand, if your income is at this other level, and you have nor received Head Start, but your
parents are highly involved, I expect X2 and so on.
Therefore, the general linear regression approach (also called GLM, the General Linear Model), with its
analysis of variance, inferential analysis and its prediction equation descriptive analysis, encompasses all
the analytic methods one might use for a true experiment or a quasi-experiment. As a data analysis
approach, it is appropriate whether one uses an experimental design, a quasi-experimental design, or
some other design. As such, the regression analytic method will be appropriate regardless of what we
decide as the data collection design.
The question is not so much what data analysis method we should use. Rather, it is what data collection
design we should use. We argue that we do not have a true experimental nor a quasi-experimental design.
This is because we do not have a uniform treatment for all treatment groups. In fact, what constitutes
treatment is different from program to program. The only “uniform” treatment from ECI is the assistance
of a consultant team (which we can manipulate for the purpose of experiment). Other “treatments” are all
locally determined and are unique to some extent from program to program. Therefore, instead of viewing
this as an experiment, We suggest viewing this as a combination of a quasi-experiment and a naturalistic
observation data collection design. The quasi-experimental portion is the manipulation of whether a
program receives the assistance of a consultant team. The naturalistic observation portions are the
collection of program feature and quality variables. Another way of looking at this data collection
approach is that we are collecting data to examine the naturally occurring variety of field experiments
being implemented by a variety of programs and communities. When viewed from this perspective, the
regression method becomes the most appropriate analytic procedure for these data. Through regressions,
we examine the impact of ECI consulting, ECI financial support, and quality program features on program
outcomes, family outcomes, community outcomes, and indirectly on child outcomes.

4. Constructed Comparison Group Analysis

For early childhood program evaluation, McCall et al. (1999) suggested modeling the pretest scores of
children as a function of chronological age. The slope of the linear model is derived through ordinary least
square regression. This slope is then used to estimate the expected change over the time period of
program intervention for each individual child. The expected change is added to the pretest score of the
child to produce an expected posttest score without intervention. The actual observed posttest scores can
then be compared against these expected scores to identify the impact of the treatment. To eliminate the
chance hypothesis, a one-sample dependent t-test would be used.
A fundamental deficiency of the McCall et al. method is its failure to take into account the potential error
of the slope estimate. To remedy this problem, we modified the “constructed comparison group” method
as follows: 1) We submitted the data on chronological age and DOCS scores at program entry (i.e., Time

33
point 1) to a polynomial regression analysis to identify the best prediction model using the ordinary least
square method as described by McCall et al. 2) We then applied the prediction equation to the chronolog-
ical age at later time points to derive the expected DOCS score for each child at each of these time points.
3) We then compared the expected DOCS scores without treatment (prediction of developmental change
without ECI participation) against the actual observed DOCS scores at each time point to determine the
effect of treatment (e.g.,, the ECIM consultation and mentoring approach) at these time points. We tested
for the statistical significance of the difference between the observed posttest score mean and the mean
of the expected no-treatment scores at each time point through a modified one-sample dependent t-test
as follow (see http://espse.ed.psu.edu/suen/proof.htm for the derivation of Equation 1):

xd
t=
s o2 + s p2 + s e2 - 2rop so sp
n

where xd is the mean difference between the observed posttest score and the mean predicted untreated
score, so is the standard deviation of the observed posttest scores, sp is the standard deviation of the
predicted untreated scores, se is the standard error of prediction, rop is the correlation between the
observed and the predicted scores, and n is the number of children involved in the program that is being
evaluated. The probability for this t-test is to be evaluated against n-1 degrees of freedom.

5. Hierarchical Linear Modelling (HLM) Analyses (Bryk & Raudenbush, 1987)

When we extend the regression analysis to examine the impact of ECI, program characteristics, family, and
community on the child, we will encounter a technical problem. In essence, children are grouped in
programs, families and communities. Children in the same program or community are more similar to one
another than are children between different programs or communities because the former receive similar
treatments. If we were to use the regression approach or the t-test or any of the analytic procedures
described earlier, we would have violated some of the underlying statistical assumptions and the results
may be misleading. The HLM analytic approach is basically a sophisticated form of regression in which
we model that data to account for the within-group and between group factors and the nested nature of
the data. Through this analysis, we obtain a more precise and appropriate description of the impact of ECI,
family, program quality variables, and communities on the child.
The HLM approach described above will accommodate change over time. Through this approach, we
would predict child outcome as a function of time, in addition to program quality changes, ECI support,
family, community and other factors. By analyzing data through an application of HLM to growth curve
analysis, the fact is that the spacing, number of time points, and exact starting point will not present
serious problems. With the growth version of HLM, however, we can plot the appropriate growth curve,
be it linear, quadratic, cubic, quartic, etc.
The HLM model of generating curves of developmental trajectories fits well with a developmental stage
transformation perspective rather than being misaligned with it. The HLM research design/statistical

34
model has been used in the early childhood and early intervention fields to chart longitudinal changes in
children’s developmental and behavioral status during programming. Buchinal, Bailey, and Snyder (1996)
published a research-based statistical procedure article that describes the steps and types of quantitative
and qualitative techniques that early childhood professionals can use to generate growth curves using
HLM and other associated strategies. The article specifically illustrates the use of the Battelle
Developmental Inventory in actual field-based intervention studies with samples of typical children and
those with delays and disabilities.

35
 C H A P T E R 6

HOW SUCCESSFUL WAS ECI IN

ITS FIRST THREE YEARS?

ECI COMMUNITIES AND PROGRAMS ACHIEVED

PROJECTED BENCHMARKS FOR SUCCESS
During the first-three years of ECI, ECI programs and communities achieved over 80% of the projected
benchmarks for success in the areas of child, family, program, community, and early school success. The
table below provides a global summary of the extent to which each of the 40 Key Performance Indicators
(KPIs) set by the original ECI consortium and refined, as authorized by the consortium, by the SPECS team,
was accomplished by the programs and communities: 30% were surpassed; 52% were achieved; and 18%
were not achieved. Eighty-two per cent of the KPIs were achieved or surpassed in the first 3 years of ECI.
The matrix reflects the overall results of an aggregate analysis which blends data from three different
sources of information: measurable progress on the SPECS battery of instruments; statistical analysis
of outcomes; and focus-group results using the community leadership councils within each ECI
neighborhood.
The following chapters of this report isolate and discuss the specific dimensions of these outcomes in
child, family, program, community, and early school success benchmark categories.

36
KPI# Key Performance Indicator Fell Short Achieved Surpassed Comments

CHILD BENCHMARKS

85% of ECI Children will enter K with- 98% of ECI children

1 out educational support services. X successfully promoted
to K and 1st grade

After at least 2 years programming, Median ECI developmental

ECI children will show at least low level was within the average
average competencies (16-50 per- to high average range
2 centiles) in basic developmental X
skill competencies.

<10% of ECI children will need special Less than 1% referred

education services upon entry into 1st for MDE or special
3 grade. X education evluations
and placements

<15% of ECI children will be retained Less than 2% retained

4 in K X in grade

ECI children will show individual Actual vs. expected progress

progress patterns in 6 developmental profiles for the average ECI
areas that exceed 50% the expecta- child moved from the 45th
5 tions projected for them before ECI to 59th percentile
X
participation.

Children participating in ECI the Children with longest

longest will show the greatest participation showed
6 preschool progress and highest X greatest actual progress vs
success rate in K & 1st grade maturational expectations

80% of ECI children with developmen- Only 1% pf ECI children con-

tal delays at the beginning of ECI will tinued to show d
7 demonstrate average competencies X elays at K entrance
after at least 1.5 yrs. in ECI

After 3 years of ECI, children will show Elimination of social

significant increases in social skill and skill delays and behavior
8 self-control behaviors AND significant X problems recorded after 18
decreases in problem behaviors months of ECI

FAMILY BENCHMARKS

Increased time participating in their Program reports

child’s program and activities and surveys
9 X

37
KPI# Key Performance Indicator Fell Short Achieved Surpassed Comments

10 Increased use of positive parenting PBC increases

skills and discipline in the home X

PBC increases
Increase in the number of educational X
11 materials and toys in the home

PBC increases
Increased time spent in weekly
educational and enrichment activities
12 X
with their child and in the community

Increases will be seen in # of Typical parent shows aver-

parents reporting and showing age nurturing behaviors,
improved personal competence in expectations for their ECI
their play, discipline, teaching, & X child after 2 years of ECI
13
knowledge of child’s abilities

Decreases reported in overall amount Parenting stress

of family stress reported as a result of rates reported within
14 being in ECI programming & support X average range
services

Increases in the amount of social Increases in social

support reported by parents as a support reported on
15 result of being in ECI programming X FSS
and services

PROGRAM BENCHMARKS

Improvements in physical PQP/ECERS

arrangements of early care & increases
education setting. Increase in
provider knowledge of normal
16 child develoopment expectancies & X
indicators of developmental and
behavioral problems

After 3 yrs, 85% of programs will con-

sistently use a developmentally appro-
17 priate curriculum of goals & methods X
to guide teaching, child care, & child
progress monitoring

After 3 years of ECIM consultation, 85% Average program quality

of programs will achieve an overall rating within 50-60% range-
high quality status ranking on NAEYC medium and medium high
measure of quality benchmarks
18

38
KPI# Key Performance Indicator Fell Short Achieved Surpassed Comments

After 3 years of ECIM consultation, ECI data only reflects a

ECI program administrators will have 2 year period.
developed collaborative working
interagency supports for delivering
19 early intervention & developmental X
and behavioral healthcare services
to their children as evidence of a
moderate level of program intensity

For 85% of children transitioning

into K, direct communications will
be established b/t parent, early
childhood provider & teacher and
20 X
elementary school principal about
child’s strengths & needs and the
families priorities for their child

COMMUNITY BENCHMARKS

Achievement of a criterion of
neighborhood elementary
schools that actively support ECI
goals across the 80 communities
21 X
and written procedures for a
cooperative transition process
between ECI & school entrance.

Completion of a collaborative
process of social consensus by
a focus group of community &
public school leaders & a written
22 document that rank orders the X
benchmarks for K entrance &
success

After 5 years of ECI involvement,

ECI communities will show a
50% increase in the number of
formal and informal interagency
23 or inter-organizational networks X
developed to support ECI
programs in neighborhoods

39
KPI# Key Performance Indicator Fell Short Achieved Surpassed Comments

After 5 years of ECI involvement, 80%

of neighborhood partners will have
24 reached consensus on individual X
future goals for ECI expansion and a
plan for implementation

Parents who actively participate for

the longest period of time with ECI will
show significant individual increases
25 in their knowledge and use of formal X
and informal community resources
fror social support

By the end of the fifth year of ECI, the

neighborhood consortia advisory
boards will have achieved a criterion
26 X
of 70% accomplishment of goals on
their performance plan submitted to
ECIM
By the end of the fifth year of ECI, ECI
neighborhood consortia advisory
boards will demonstrate a moderately
high level on self-efficacy measures
27 thatr reveal the strength of ngh confi- X
dence, commitment, & resources
regarding their ECI programs

After 3 years of ECI program opera-

tion, ECI neighborhood consortia will
report a criterion level of 60% regard-
28 ing linkages between their program & X
various formal and informal intera-
gency supports of their programs

EARLY SCHOOL SUCCESS BENCHMARKS

80% of children will demonstrate at 98% of ECI children

least low average competencies (16- successfully promoted
50%) compared to national norms on a to K
29 measure of basic literacy & school X
skills (BSSI) by the end of K

40
KPI# Key Performance Indicator Fell Short Achieved Surpassed Comments
85% of children will reach criterion
of 80% accomplishment of the kdg
30 success curricular competencies X
on the DOCS upon K entrance

85% of ECI children will enter K with-

31 out instructional support services X < 1%

< 10% if ECI children will need special

education services upon entry into
32 1st grade X
< 1%

< 15% of ECI children will be

33 retained in K X < 2%

41
 C H A P T E R 7

DID CHILDREN BENEFIT

FROM PARTICIPATION IN ECI?

A SNAPSHOT OF CHILD PROGRESS

All too often, young children growing up in high-risk situations, begin progressing at slower than expected
rates as infants and continue this pattern into elementary school. The consequence is children that are
less well-equipped to succeed in early educational settings because they enter behind their peers and lag
farther and farther behind over time. This means lagging behind on fundamental aspects of development
that include cognitive, language, motor (movement and coordination), and social behavior skills as well as
basic life skills
What were the children like that entered ECI programs and how did they do over time in relation to these
areas of development? When compared to peers across the U.S, most ECI children entered programs with
skills that were within the low average range for their age and progressed at age expected or slightly
accelerated rates of development. They, in turn, avoided expected skill losses so often evidenced by young
high-risk children.
From this larger group of children, 14% of youngsters began programs with skills significantly behind their
peers both in ECI as well as across the nation. These children entered programs with identified delays in
one or more areas of development. In spite of this, they made significant progress after approximately one
year of attending ECI preschool programs. Not only did these children develop skills at rates above and
beyond what would be expected by pure maturation, but also, overtime most of these children progressed
from at least one area of deficit to cognitive, language, movement, coordination, and social skills all in the
average range.
ECI programs promoted effective social skills and effective self-control behaviors that researchers suggest
are important elements of early school success (Merrell, 1994). Sharing, cooperation, listening and attend-
ing, self-control, and independence are but a few examples of the types of skills that were promoted AND
more importantly of the types of skills that children developed and maintained at levels comparable with
age mates across the country. Eighteen percent of ECI children began preschool with seriously delayed
social skills and/or clinical levels of problem behaviors (e.g., aggressiveness, high activity, anxiety, somat-
ic complaints). However, after one year, most of these children significantly increased their social skills
and decreased their problem behaviors to within normal developmental levels.
The success of ECI children in kindergarten and first grade was far greater than expected. By October
2000, 125 children transitioned to kindergarten and first grade and demonstrated age-appropriate academ-
ic and social skills on par with similar aged children across the U.S. Basic academics were assessed by
teachers in terms of speaking, reading, writing, and math skills and basic social skills were assessed in
terms of classroom behavior and daily living skills. Furthermore, the average grade retention rate of ECI
children was less than 2% and the average special education placement rate was less than 1% — figures
which sharply contrast to the retention and special education placement rates of school districts within ECI
communities. The average primary grade (K-3rd grade) retention and special education placement rates
for school districts within ECI communities were respectively 23% and 21%.

42
 QUALITY EARLY LEARNING ENABLED AT-RISK CHILDREN TO SUCCEED
ECI Children Avoided Expected Skill Losses, Progressed at Age-Expected or Accelerated Rates, and
Achieved at Average to Above Average Competency Levels

What do we know about the developmental progress patterns of high-risk children who do not
participate in preschool programs?

• Research (Barnett, 1995; Bryant & Maxwell, 1997; Farran, 2000; Marcon, 1999; Schweinhart & Weikart,
1997)) tells us that the rate young children in high-risk circumstances acquire skills
begins to decline after age 2 and continues to decline through kindergarten age; such declines are
cumulative so that they are typically 1.5 years behind their peers in basic school competencies. Even
though the children begin life developing at typical rates, their development begins to decline when
language and social competencies become important for the maturational advancements at about 2-3
years of age.

Research-based Developmental Declines for

High-risk Children Not in Preschool
130

120
Mean Normative Scores

110

100

90

80

70

60

50
1 2 3 4 5 6
Age in Years

How does the developmental progress of children in ECI programs compare to the progress of age
peers across the U.S.?

- Eighty-six percent of high-risk ECI children entered programs with skills within the low average
range for their age when compared to peers across the country, but progressed steadily at age
expected or slightly accelerated rates during their time in ECI.
- Children enrolled in ECI programs for longer periods of time did better, particularly in terms of
cognitive and language skills.

43
 - Children who participated for more than one year significantly increased their skills (p < .05) in
cognitive, language, motor (movement and coordination), and social development. Above and
beyond what would be expected by maturation alone, they showed significant progress in relation to:

* cognitive development which includes attention, memory, and problem solving

skills and concepts such as object permanence, cause and effect relations, let
letters, numbers, and time.

* language skills which includes acquiring and using language;

understanding and being able to follow requests, directions and discussion

* fine and gross motor skills which includes coordinating eye and hand
movements for eating, dressing, playing, drawing, and writing ; and
movement skills like crawling, walking, and running.

* social and self-help skills such as playing, making friends, following rules and
routines, and learning basic information like name, address, and age.

Developmental Score* - Developmental Score* -

Developmental Program Entry* 1+ Year in Programming*
Domain (Time 1) (Time 4)

Overall Development 102 (14) 107 (15)

Cognitive 100 (13) 103 (14)
Language 102 (14) 107 (15)
Motor 102 (13) 107 (15)
Social 105 (14) 110 (15)
*DOCS Mean Standard Score and Standard Deviation

44
 Developmental Progress Pattern: Full High-Risk Group TIME 1

Over 1+ Year (T1-T4) TIME 4

DOCS Mean Standard Score

150

130

110

90

70

50
Developmental Cognitive Language Motor Social
Developmental Domain

Are there progress differences among ECI children in Head Start centers, non-early care and education
classrooms, and family child care homes?

- ECI children that were enrolled in HeadStart classrooms began with slightly less well-developed,
but average skills and made significant progress (p< .05) over time in cognitive, language, motor, and
social skills. Their non-HeadStart ECI counterparts entered programs with age-expected skills and
maintained this level of skill competency over time. Children enrolled in family child care homes also
maintained age-expected competencies over time. This information was not compared in the
following graph and table due to notable differences in the number of children enrolled beyond
one year.

Developmental Area Score* - ECI HS Score* - ECI HS Score* - ECI Non-HS Score* - ECI Non-HS
Time 1 Time 4 Time 1 Time 4

Overall Development 96 (13) 101 (13) 104 (14) 104 (12)

Cognitive 94 (14) 98 (11) 102 (13) 100 (11)
Language 96 (14) 103 (13) 104 (13) 104 (13)
Motor 97 (14) 102 (13) 104 (13) 104 (13)
Social 99 (14) 107 (13) 106 (13) 107 (13)

*DOCS Mean Standard Score and Standard Deviation

45
 ECI Non-HeadStart and HeadStart Child
Progress Patterns Over 1+ Year
150
140
130
DOCS Mean Standard Score

120
110
100
90
80
70
60
50
Overall Cognitive Language Motor Social
Development
Developmental Domain

Non-HeadStart Time 1 Non-HeadStart Time 3 HeadStart Time 1 HeadStart Time 3

What is the progress pattern for children entering ECI programs with developmental delays?

- Fourteen percent of children entered ECI programs with mild, but significant delays (> 1 SD) in at
least one area of development, according to PA State Early Intervention Entitlement Act Standards
(1991)

- The ECI rate for delay exceeds the national incidence rate for developmental delay and disability
of 3-8% (Fujiura & Yamaki, 2000)

- Children with delays that were in programs for longer periods of time, demonstrated greater
developmental progress.

- After participating in ECI programming for at least 1 year, children with delays demonstrated notable
gains in their cognitive, language, motor and social skills, into average levels of developmental
performance.

- After 2 years of ECI participation, the children neither showed evidence of developmental delays
nor qualified for early intervention services.

46
Developmental Developmental Score* - Developmental Score* -
Domain Program Entry* (Time 1) 1 Year in Programming* (Time 3)

Overall Development 83 (4) 99 (14)

Cognitive 81 (5) 96 (13)
Language 83 (4) 100 (15)
Motor 83 (4) 100 (15)
Social 86 (5) 103 (14)

*DOCS Mean Standard Score and Standard Deviation

Developmental Progress Pattern of Children

with Delays Over 1 Year (T1-T3)
Time 1 Time 3
150

130
DOCS Mean Standard Score

110

90

70

50
Development Cognitive Language Motor Social

Developmental Domain

Do ECI programs promote effective social (interaction and independence) skills and self-control
behaviors?

- Children in ECI programs learned a number of important interaction skills, independence skills,and
self control behaviors. Eighty-two percent of high-risk children entered ECI programs with low
average skills for their age and continued to mature and advance their competencies over the
course of ECI.

47
 - Effective social skills and self-control behaviors contribute to children’s success during the early
school years. Some of the skills that ECI children developed and refined, that programs promoted,
and that were captured by these broad descriptors of social competence include:

* Learning to interact effectively with other children (e.g., making friends, sharing and ,
taking turns, showing self-control, compromising)

* Learning to interact effectively with adults (e.g., seeking help, cooperating, following
rules and instructions).

* Learning to be independent (e.g., working independently, trying new things independ-

ently, standing up for self and others).

- Most children in ECI programs demonstrated relatively low average levels of challenging behaviors
such as fighting, teasing, and disobeying both initially and over a period of time.

Developmental Developmental Score* - Developmental Score* -

Domain Program Entry (Time 1) 1 Year in Programming (Time 3)

Social Skills 104 (15) 92 (16)

Problem Behaviors 106 (14) 93 (17)

*PKBS Mean Standard Score and Standard Deviation

48
 Behavioral Progress Pattern of ECI Children
(Full High-Risk Group) Over 1 Year (T1-T3)
150
Time 1 Time 3
140
130
PKBS Mean Standard Score

120
110
100
90
80
70
60
50
Social Problem
Skills Behaviors
Developmental Domain

What is the progress pattern for children entering ECI programs with delayed social skills and/or
serious and clinically significant behavior problems?
- Eighteen percent of ECI children began preschool with delayed social skills and/or serious behavior
problems (e.g., aggressiveness, high activity, anxiety, somatic complaints) These problems would
have qualified the children for a DSM IV diagnosis for mental health services in Allegheny County.
However, after one year, most children significantly increased their social skills (p < .01) and
decreased their problem behaviors to within normal developmental levels so that they no longer
qualified for such diagnoses or services.

- At entry into ECI, the children showed behavior problems so severe (99th percentile) that they
exceeded the types of behavior problems demonstrated by nearly all their age peers in national
samples.

Developmental Developmental Score* - Developmental Score* -

Domain Program Entry (Time 1) 1 Year in Programming (Time 3)

Social Skills 73 (16) 93 (9)

Problem Behaviors 126 (17) 107 (23)

*PKBS Mean Standard Score and Standard Deviation

49
 Behavioral Progress Pattern of Children with Delayed Social
Skills and Challenging Behavior Group Over 1 Year (T1-T3)
150
Time 1 Time 3
140
130
PKBS Mean Standard Score

120
110
100
90
80
70
60
50
Social Problem
Skills Behaviors
Developmental Domain

Do children in ECI Programs show greater progress than those in non ECI Programs?

- Typically developing children in both ECI and non-ECI programs made consistent developmental
progress over a 1 year period of program intervention in cognitive, language, motor, and social skills.
However, ECI children entered programs with a higher level of developmental competence and
demonstrated a significantly greater number of cognitive, language, and social skills after 1
year than their non-ECI counterparts.

Developmental Developmental Score* - Developmental Score* - Developmental Score* - Developmental Score* -

Domain ECI - Time 1 ECI - Time 3 Non-ECI - Time 1 Non-ECI - Time 3

Overall Development 104 (14) 105 (15) 98 (16) 100 (18)

Cognitive 102 (14) 102 (13) 96 (15) 97 (16)
Language 105 (15) 106 (14) 99 (17) 100 (18)
Motor 105 (14) 106 (14) 102 (17) 104 (18)
Social 108 (14) 109 (14) 99 (16) 101 (19)

*DOCS Mean Standard Score and Standard Deviation

50
 ECI and Non-ECI Child Progress Patterns Over 1+ Year
150
140
130
DOCS Mean Standard Score

120
110
100
90
80
70
60
50
Overall Cognitive Language Motor Social
Development
Developmental Domain

ECI Time 1 ECI Time 3 Non-ECI Time 1 Non-ECI Time 3

- Children with delays identified at program entry in ECI and non-ECI programs made consistent
developmental progress over a 1 year period of program intervention in cognitive, language, motor,
and social skills. However, ECI children demonstrated significantly greater developmental gains in
language, and social skills after 1 year than their non-ECI counterparts (p>.05). These comparison
programs differed in that they were required by early intervention and Head Start performance
standards to include children with disabilities in their programs. ECI was not targeted primarily on
children with disabilities.

Developmental Developmental Score* - Developmental Score* - Developmental Score* - Developmental Score* -

Domain ECI - Time 1 ECI - Time 3 Non-ECI - Time 1 Non-ECI - Time 3

Overall Development 83 (4) 99 (14) 81 (9) 89 (14)

Cognitive 81 (5) 96 (13) 81 (10) 88 (13)
Language 83 (4) 99 (15) 81 (9) 90 (14)
Motor 83 (4) 99 (15) 83 (10) 93 (15)
Social 86 (5) 103 (14) 81 (9) 90 (15)

*DOCS Mean Standard Score and Standard Deviation

51
 Progress Comparisons for Children with Delays in ECI and Non-ECI Programs:
Progress Over 1 Year of Program Intervention
150
140
130
DOCS Mean Standard Score

120
110
100
90
80
70
60
50
Overall Cognitive Language Motor Social
Development
Developmental Domain

Time 1 ECI Time 3 ECI Time 1 Contrast Program Time 3 Contrast Program

• Typically developing children entered ECI programs with slightly higher levels of social competence
and slightly lower levels of challenging behavior when compared to their non-ECI counterparts.
However, children in ECI and non-ECI programs made consistent progress over 1 year period in both
social skills and self-control behaviors.

Developmental Developmental Score* - Developmental Score* - Developmental Score* - Developmental Score* -

Domain ECI - Time 1 ECI - Time 3 Non-ECI - Time 1 Non-ECI - Time 3

Social Skills 104 (15) 106 (16) 102 (15) 104 (17)
Problem Behaviors 92 (16) 93 (16) 95 (17) 95 (17)

*PKBS Mean Standard Score and Standard Deviation

52
 Behavior Progress Pattern of ECI and
Non-ECI Children Over 1 Year (T1-T3) ECI Time 1
ECI Time 3
150
Non-ECI Time 1
140
Non-ECI Time 3
130
PKBS Mean Standard Score

120
110
100
90
80
70
60
50
Social Skills Problem Behavior
Developmental Domain

What skills, learned by ECI preschoolers, are important precursors to success in kindergarten?

- Listed below are some of the early learning skills which ECI preschoolers learned that helped them
to smoothly transition and be successful in kindergarten.

Kindergarten Precursor Skills

• knowledge of numbers and counting
• everyday problem solving in natural situations
• expressing themselves through words and sentences
• waiting, sharing, and taking turns
• following directions and paying attention
• drawing and writing letters and numbers
• cooperating in group activities and games
• “reading”: singing, rhyming, and enjoying books
• remembering details of stories read aloud
• creative and pretend play
• making friends
• showing respect to adults and friends

53
 Preschoolers gained a notable number of kindergarten precursor skills during their last 6 months
in programming.

DOCS Kindergarten % Mastered Skills % Mastered Skills

Precursor Skill 6 Months Before Kdg. (Time 1) 3 Months Before Kdg. (Time 2)

Asks questions 76 84
Understands 72 85
Counts 60 77
Names letters 54 78
Writes letters 37 65
Names printed letters 26 50
Knows/“reads” common signs 51 70
Retells 49 64
Tells meanings of words 21 41
“Reads” 2 12
Total skills 45 63

Acquisition of Kindergarten Precursor Skills Matching PA 3rd Grade Academic Standards:

Achievement in Final 6 months of ECI
100 Time 1 Time 2

90
80
% Learning Skills Achieved

70
60
50
40
30
20
10
0
Asks Understands Counts Names Writes Names Knows/ Names Retells Tells “Reads” Total
Letters Letters Printed “reads” Items Meanings
Letters signs

54
Do ECI children succeed in kindergarten and first grade?

- Former ECI preschoolers continued their pattern of success into early elementary school. When
kindergarten and first grade teachers assessed these children on nationally normed measures of
early academic and social skills, assessment scores reflected skill development in the average
range.

Developmental Developmental Score* - Developmental Score* -

Domain Second Semester in Kindergarten Second Semester in First Grade

Overall Developmental 99 (24) 107 (13)

Spoken Language 99 (26) 97 (19)
Reading 98 (16) 110 (9)
Writing 99 (8) 109 (4)
Math 104 (17) 115 (9)
Classroom Behavior 98 (20) 100 (13)
Daily Living Skills 98 (18) 101 (10)

* More specifically, children demonstrated skills comparable to peers across the

country in the areas of speaking, reading, writing, math, behavior, and basic life skills.
* ECI children, both in kindergarten and first grade appeared strongest in math skills.
* Assessments by preschool providers were positively correlated with and predictive of
kindergarten and first grade teacher assessments and of children’s progress and
achievements in thinking, language, social skills, reading, math, writing, and daily
living skills.
* Children’s social skills at the end of ECI was predictive of early school success in
spoken language, reading, math, and classroom behavior.

55
 Progress Pattern of Former ECI Children in Kindergarten and First Grade
150 Kindergarten

140 First Grade

Mean Standard Score on BSSI-3

130
120
110
100
90
80
70
60
50
Overall Spoken Reading Writing Math Class Daily
Development Language Behavior Living

Developmental Domain

How do grade retention and special education placement rates for ECI children compare to historical
grade retention and special education placement rates for non-ECI children in associated school districts
within ECI communities?

- The average grade retention rate of ECI children was less than 2% and the average special
education placement rate was less than 1% — figures which sharply contrast to the retention and
special education placement rates of school districts within ECI communities. The average primary
grade (K-3rd grade) retention and special education placement rates for school districts within ECI
communities were respectively, 23% and 21%.

56
 ECI vs. Typical School District Grade Retention
and Special Education Rates
40
35
Percentage

30
25
20
15
10
5
0
District ECI Child District ECI Child
Retention Retention Special Special
Rate Rate Education Education

57
 C H A P T E R 8

DID FAMILIES BENEFIT FROM PARTICIPATION IN ECI?

A SNAPSHOT OF FAMILY PROGRESS

It is often hard for families in poverty to provide the many things for their children that help them grow and
thrive. One particularly challenging issue is accessing quality and affordable preschool services. Of the
many questions posed by this evaluation, one important one was “What is the impact on families of being
able to obtain affordable, high-quality, and community-based early care and education?” The evaluation
found that ECI families gained parenting skills and provided nurturing learning experiences for their
children at home as frequently as many other American families of varying socioeconomic levels.
Furthermore, families in ECI acquired appropriate expectations of what their children should be able to do
at different ages. ECI families experienced moderately low levels of stress directly related to parenting
their young children and slightly higher levels of support as parents. Appropriate parental expectations
were related to children progress in thinking, language and social skills. The nurturing aspects of parental
behavior was associated with success in reading and spoken language skills, especially, at the end of
kindergarten. ECI programs were a very important source of parenting support for families the longer that
they participated in the program.

Families Used Parenting Strategies and Provided Learning Experiences

At Home which Supported their Childrens’ Development.

How do ECI families compare to other U.S. families in their use of positive parenting strategies; what
kinds of strategies did ECI parents frequently use at home with their children?

- ECI families consistently used positive parenting strategies and activities that supported their
children’s growth and development. Furthermore, they provided nurturing learning activities at
rates comparable to that of other families across different socioeconomic levels from other large
urban areas.

- Eighty percent of parents gained more effective nurturing skills. The most frequently reported
nurturing skills used by ECI parents were,
* providing praise for learning new things
* reading and playing together with their children
* arranging play activities for their children, such as drawing,
coloring, and selecting toys.

58
Nurturing Behaviors and Learning Activities of ECI Families over Time

Raw Mean Score Families of Families of Families of Families of

1 Year-Olds* 2 Year-Olds* 3 Year-Olds* 4 Year-Olds*

ECI Families
at Program Entry 30.3 (7.7) 30.3 (9.4) 32.3 (9.3) 29.2 (7.9)
PBC-S Normative
Sample 30 31 31 31
ECI Families
1 Year Later 28.8 (9.2) 32.1 (6.0) 32.3 (8.1) 29.9 (2.9)

*PBC-S Nurturance Subscale Mean Raw Score and Standard Deviation

Nurturing Behaviors and Learning Activities

Program Entry
of ECI Families Over Time
PKBS Mean Standard Score

PBC-S Mean
47 Nurturance Score

42 1 Year Later

37
32
27
22
17
12
1 Year-Old 2 Year-Old 3 Year-Old 4 Year-Old
Age of Family's Child

59
Did parents in ECI have realistic expectations of their children’s abilities?

• Knowing what to expect of children in terms of their abilities and interests is pivotal for parents in their
roles as nurturers and teachers. As such, one question explored by this model was the extent to which
families had a sense of what skills were developmentally appropriate for their children.

• Both upon first enrolling their children in ECI programs and one year later, ECI parents’ expectations of
their children’s abilities were developmentally appropriate.

• Families of infants, toddlers, and preschoolers alike set developmentally appropriate expectations.

Developmentally Appropriate Expectations of Children’s Abilities in ECI Families over Time

Raw Mean Score Families of Families of Families of Families of

1 Year-Olds* 2 Year-Olds* 3 Year-Olds* 4 Year-Olds*

ECI Families
at Program Entry 19.2 (10.2) 27.6 (11.5) 41.7 (9.3) 40.2 (8.8)
PBC-S Normative
Sample 19 30 38 41
ECI Families
1 Year Later 19.8 (4.7) 34.8 (12) 41.7 (6.7) 42.8 (8.3)

*PBC-S Expectations Subscale Mean Raw Score and Standard Deviation

60
 Program Entry

PBC-S Norm
Developmentally Appropriate Expectations of 1 Year Later
Children's Abilities in ECI Families Over Time
45
PBC-S Mean Expectation Score

40
35
30
25
20
15
10
5
0
1 Year-Old 2 Year-Old 3 Year-Old 4 Year-Old
Age of Family's Child

Families Reported Relatively Little Stress Associated with Parenting

• To what extent did ECI families experience stress around parenting issues and did this level o
stress change over time?

- Families in ECI reported moderately low levels of stress (35th -38th percentiles) around parenting
issues during their participation in ECI programs when compared to typical parenting stress levels
(50th percentile) of urban families in the United States.

- ECI families’ level of parenting-related stress remained moderately low (i.e., did not change
significantly) over time.

Assessment Timeframe Stress Related to Stress Related to Negative

of Family Stress Challenging Child Behaviors* Parent-Child Interactions*

Program Entry 34.8 (29.6) 33.1 (27.8)

(Time 1)

1 Year Later 38.1 (31.22) 31.9 (27.4)

(Time 2)

*PSI Mean Standard Score and Standard Deviation

61
 Parenting Stress Levels in
ECI Families Over Time Program Entry
1 Year Later
100 Stress Levels Suggesting the Need
90 for Monitoring or Intervention

80

PSI Mean Percentile

70
60
50
40
30
20
10
0
Challenging Child Negative Interactions

Sources of Parenting Stress

- Children learned how to interact more effectively with others (i.e., children and adults) and
developed more skills around independence in families with more positive parent-child interactions
(p < 0).

Families Received Support in Parenting their Children, Particularly from

Grandparents, ECI Programs, and Family Doctors

To what extent was parenting support available for families in ECI; did the degree of support change
over time?

- Families in ECI reported slightly higher levels of helpful parenting supports (40-42) when compared
to other families across the United States (30).
- The degree of parental support remained relatively consistent (i.e., did not change significantly)
over time.
- Families with higher levels of parenting support engaged in more nurturing parenting activities
with their children (p < 0). These activities include:

* Providing praise for learning new things

62
 * Setting up play activities and playing on the floor together
* Getting books, reading, and taking walks together

- Families with lower levels of parenting support perceived their children as more challenging or
difficult to rear (p < .0) and had more stressful or negative interactions with their children (p < .0).

Overall Level of Parenting Support Over 1 Year in ECI Families

Assessment Timeframe
of Parental Supports
in Families Level of Parenting Support*

ECI Families: Program 39.4

Entry (Time 1)

ECI Families: 1 Year 41.9

(Time 2)

Normative Sample (FSS) 29.8 (10.5)

*Mean Raw Score

Overall Level of Parenting Support

Over 1 Year in ECI Families
100
90
(Overall Level of Parenting Support)

80
FSS Total Mean Raw Score

70
60
50
40
30
20
10
0
Program Entry 1 Year Later

Time of Assessment

63
What kinds of parenting supports did families find most helpful?

- When families first entered ECI childcare programs, they reported the following sources of
parenting support as most helpful:
* grandparents
* parent’s relatives
* family doctor
* child’s childcare program

Sources of support are listed in the order of helpfulness.

- After participating in ECI for one year, families reported ECI programs and staff as the most help
-ful sources of parenting support, followed by grandparents and family doctors, relatives, and
parenting partners/ spouses. Sources of support are listed below in the order of helpfulness.

* child’s childcare program

* child’s child care staff
* grandparents
* family doctor

64
 Source of Mean Score* at Program Mean Score* 1 Year
Parenting Support Entry (Time 1) Later (Time 2)

My Parents 3.5 (1.7) 3.3 (1.9)

My spouse or partner’s parents 2.4 (1.9) 2.2 (2.0)
My relatives 3.3 (1.4) 3.1 (1.5)
My spouse or partner’s relatives 2.2 (1.4)
My spouse or partner 2.2 (1.8) 1.9 (1.9)
My friends 2.8 (2.0) 2.7 (1.4)
My spouse or partner’s friends 1.3 (1.7) 1.3 (1.7)
My children 2.3 (2.1) 2.3 (2.1)
Other parents 1.9 (1.6) 1.9 (1.7)
Co-workers 1.2 (1.5) 1.4 (1.6)
Parent groups 1.0 (1.5) 1.0 (1.6)
Social groups or clubs 1.0 (1.6) .9 (1.6)
Church members 1.9 (1.8) 2.0 (1.9)
Family doctor 3.2 (1.5) 3.3 (1.4)
Early intervention program 1.5 (2.0) 2.6 (1.9)
Childcare program 3.0 (2.1) 4.0 (1.3)
Childcare staff 2.6 (2.1) 3.9 (1.2)
Other professionals or agencies 1.3 (1.9) 1.4 (1.8)

*FSS Mean Raw Score and Standard Deviation

Helpfulness of Parenting Supports Over Time for ECI Families Program Entry
1 Year Later
5.0
4.5
4.0
FSS Mean Raw Score

3.5
3.0
2.5
2.0
1.5
1.0
0.5
0
ts

ts

es

es

ts

rs

ps

ch

or

ff

ls
nd

nd

ub
re

io

a
en

en

en

ra

na
ke

ct
iv

iv

ou

ur

st
nt
ld
rie

rie

cl

og
do
at

at
ar

ar

ar

io
or

ch
gr

re
e
hi
el

al

ss
yp

yf

rp

rv

pr
-w
re

ca
yc

nt
's
yr

ci
's

te

fe
m

he

re
m

co
er
's

re

ild
er

so
m

In

ro
m

ca
er

rtn

ot

pa
rtn

ch

rp
rly
rtn

ild
pa
pa

he
Ea
pa

ch

ot

Source of Parenting Support

65
 C H A P T E R 9

DID ECI PROGRAMS ACHIEVE QUALITY?

A SNAPSHOT OF PROGRAM ENHANCEMENTS

Within the originally conceived ECI design and model, program quality was hypothesized to be the driving
force behind program impact and outcomes, especially child outcomes. Quality mentoring was viewed as
the vehicle through which programs would enhance their use of “best practices” and ultimately achieve
NAEYC quality standards.
The results of ECI over three years indicate clearly that children participating in high quality programs
showed developmental progress that exceeded maturational expectations. Fifty percent of ECI programs
met NAEYC quality standards after 18 months of engagement in the quality mentoring process by ECIM.
Improvements in program quality emerged in all best practice categories, including use of curricular
methods, environmental arrangements, and individualized instruction. Programs also showed increases in
the scope and level of intensity of services provided to high-risk children and families during the first three
years of ECI. ECIM consultation was predictive of positive overall developmental outcomes for children.
Similarly, ECIM consultation was positively related with program intensity and positively related to changes
in program quality at the end of ECI.

PROCESS AND OUTCOMES OF PROGRAM QUALITY AND INTENSITY

The model developed by the program strand was designed to provide a sampling of programs and
classrooms involved in the Early Childhood Initiative. The SPECS evaluation team implemented the
program evaluation model with 27 program settings throughout each ECI neighborhood. Types of programs
included family child care homes, inclusive center-based classrooms, and Head Start centers. All listed
programs were evaluated and followup program evaluations were completed after year of participation in
the Early Childhood Initiative (ECI).

66
Neighborhood Total Number Center FDC Head Start
of Programs

Braddock 7 3 4 0
Hill 3 2 1 0
Homewood 5 2 2 1
Highlands 4 1 0 3
Steel Valley 3 2 0 1
Sto Rocks 2 1 1 0
Wilkinsburg 3 1 2 0

TOTAL 27 13 10 4

Program quality was observed and evaluated using the SPECS Program Quality Scale for Early Childhood
Settings (PQP; Cook-Kilroy & Bagnato, 1999). The PQP was developed and field-validated based on the
revised NAEYC developmentally appropriate practice standards (Bredekamp & Copple, 1997). A mean
percent score was calculated for each sub section of the scale and a total mean percent score was
calculated for each individual program. If programs had multiple classrooms the scores were averaged to
obtain the mean percent score.
Based on the total mean percent score of the individual programs, programs were grouped into 3 quality
types, low (less than 50% of the quality markers were obtained) medium (50-70% of quality markers were
obtained) or high quality (70% or more quality markers were obtained.)

Program Quality Scores

100.0
90.0
80.0
Mean Percent Score

70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
LOW MEDIUM HIGH

Rating Level

67
 Time 1 and Time 2 Program Quality Evaluation
Time 1 Time 2
100
90
80
Mean Percent Score

70
60
50
40
30
20
10
0
Overall Curriculum Equipment Physical Teaching Health Relationship Staff and
& Materials Environment with Parents Administrative
Issues

Individual Categories

Changes in Program Quality - Individual Programs

(50% of ECI Programs Meet or Exceed NAEYC Standards) Time 1 Time 2

100
90
80
Mean Percent Score

70
60
50
40
30
20
10
0
A B C D E F G H I

Individual Programs
A SPECS Program Intensity survey was completed by center directors following the program evaluation

68
schedule of 1 time per year. A total of eight programs completed a Time 1 and Time 2 Program Intensity
Survey. Respondents completed surveys by choosing the item that best described their program in
specific categories (program scope, personnel, support services, involvement with communities,
involvement with families, programming). An overall mean percent score for each individual program was
calculated. A grand mean percent score for all 8 programs was calculated for each sub section.

Quality Mentoring Ensured Increased Program Enhancements in ECI

Did ECI Programs of All Types Show Enhancements in Quality and Intensity?
• 67% of ECI programs were of at least medium quality, 30% of ECI programs were of high quality
at the time 1 assessment
• 91% of the programs improved program quality after 1 year of quality mentoring
• There was an average growth rate of 13.6 percentage points.
• 50% of ECI programs met NAEYC standards in 18 months of ECIM quality mentoring.
• 70% of teachers enhanced their knowledge of child development expectancies and developmentally
appropriate practices.
• Improvements in program quality was associated with skill development in writing and overall
kindergarten success (p>.05).
• Length of time children were involved in medium to high quality ECI programs was associated with
positive developmental progress that exceeded maturational expectations (p>.01).

Program Quality Evaluation Time 1

Time 2
100.0
90.0
80.0
FSS Mean Raw Score

70.0
60.0
50.0
40.0
30.0
20.0
10.0
0
A B C D E F G H I
• The overall level of intensity of programs improved from Time 1Jto Time
K
2. L

Individual Programs

69
• Programs improved in the overall scope and intensity of their support services for children and
families except with the formal connection to community agencies. Failure to achieve this goal was
directly related to termination of ECI by the steward.
• The most significant gain in program intensity was demonstrated in the area of educational
programming (13.6% gain), which focused on individualized accommodations made by teachers
for each child’s early learning needs.

SPECS Program Intensity Profile

Category Mean Percent Score Mean Percent Score

TIME 1 TIME 2

Program Scope 28.1 34.4

Support Services 56.9 62.5
Personnel 71.9 78.1
Involvement with Community 67.2 46.9
Involvement with family 71.9 72.7
Programming 45.8 59.4

Overall 56.4 59.7

Changes in Specific Program Intensity Features

100
Time 1 Time 2
90
80
Mean Percent Score

70
60
50
40
30
20
10
0
Program Support Personnel Involvement Involvement Programming Overall
Scope Services with with
Community Family

Individual Categories

70
 Changes in Program Intensity: Individual Programs
Time 1 Time 2
100
90
80
Mean Percent Score

70
60
50
40
30
20
10
0
A B C D E F G H

Individual Programs

What were the process and outcomes of ECIM quality mentoring?

An Education and Consultation survey was completed by teachers and providers to rate the level of
intensity and satisfaction with ECIM consultation. Surveys were completed once per year following the
program evaluation schedule. Categories included, frequency of consultation, method of consultation,
content of consultation, satisfaction, and effectiveness of consultation. Teachers were asked to rate each
area based on their experience of the past year. Each teacher checked a box that most appropriately
reflected their experience with consultation.
Consultation varied slightly across neighborhood programs, however, programs received similar modes of
consultation and all consultation addressed issues related to improved program quality on the various
quality scales in the ECERS family of instruments.

71
 Questions Average Response

Frequency of consultation 2-4 hours monthly (1 hour per week average)

Most common used consultation strategy Observing and providing suggestions on

ways to improve

Types nof comments received Positive suggestions on ways to

from consultant improve teaching

How specific items are identified Items are addressed by the consultant

Items rated from strongly disagree to Items rated from strongly disagree to
strongly agree strongly agree

Availability of consultant Agree

Usefulness of equipment Strongly Agree

Usefulness of consultation to improve quality Agree

Usefulness in training to provide Agree

a quality program

Increased confidence in working with Agree

young children because of consultation

Overall benefit for participation in ECI Agree

How effective was ECIM quality mentoring?

• ECIM mentoring improved program quality and intensity.

• Teaching and caring for young children was the most frequently addressed topic of ECI
consultation that matched the NAEYC standards.
• Children with challenging behavior problems benefited from ECIM consultation to teachers (p>.05)
• Teachers and providers identified an overall benefit in participation with ECIM.
• Teachers and providers felt that ECI mentoring helped them improve quality
• ECIM Consultation was individualized to the areas teachers identified as needing increased support.

72
 C H A P T E R 1 0

WAS COMMUNITY LEADERSHIP RESPONSIBLE

FOR THE SUCCESS OF ECI PROGRAMS?

A SNAPSHOT OF COMMUNITY OUTCOMES

Today, early childcare and education programs are proceeding in several Greater Pittsburgh communities as
parents, community organizations and others demonstrate their commitment to the preparation of young chil-
dren for successful entry into public school. Despite frustrations experienced as participants in Pittsburgh’s
Early Childhood Initiative (ECI), several community-based organizations have pressed ahead with innovations
in early care and education. Their collective effort, referred to as the “community process”,
comprised a fundamental component of the ECI experience.

THE PRESENT

In Braddock, Pa., the Greater Braddock Early Childhood Network operates and manages the community’s
early education program. Some children, having progressed through the early education stage, have begun
to transition into the public school system. Bob Grom, Executive Director of the Heritage Health Foundation,
the lead agency for the Greater Braddock Early Childhood Network, observed that the community and its
board of directors are committed to the goals of its early education and childcare program.
Over in Wilkinsburg, the Wilkinsburg Early Childhood Initiative administers the community’s innovation in
early education. Hosanna House, Inc., under the directorship of Leon Haynes, serves as the lead agency
for WECI. Situated in a former public school building, the large, multi-purpose community center introduces
children and families to an array of educational and related family support functions.
In Homewood-Brushton, Primary Health Care Services under the directorship of Wilford Payne administers
the collaborative of more than seventy agencies and organizations comprising the community Early
Childhood Initiative. State of the art design and furnishings were on display when the Kelly Street Center
held its open house. Like the other centers, the Homewood initiative expects to make a difference in the
lives of young children and their families.
Down the river in Sto-Rox, Focus on Renewal, Inc., performs a similar function. Under the capable leader-
ship of Father Regis Ryan, FOR serves as the lead agency for the community’s early education initiative.
Housed in the appropriately named Butterfly Garden, the early education and childcare center seeks to
identify and overcome educational and childcare needs essential to successful entry into kindergarten.
These programs are some of the remaining community structures formerly comprising Pittsburgh’s Early
Childhood Initiative. From early 1998 to the fall of 2000, nine funded projects representing more than
twenty five communities were in operation. Similar community based efforts were also underway in other
communities including the Highlands, the Steel Valley, East Liberty, the Hill, and the Southside.
Geographically encompassing a massive area, programs stretched from Tarentum to Sto-Rox and from the
Hill to Homestead.

73
With the level of intensity, which characterized these programs, it would have been reasonable to expect
that ECI would become a resounding success. Instead, to the consternation of many, the communities, par-
ticipating agencies and organizations, and families with at risk children, it was terminated. Pressed to
account for that turn of events, some sought to attribute ECI’s disappointing fate to the “community
process”, i.e., the self-determination - local decision-making standard required of every program. In sharp
contrast, this evaluation arrives at a different conclusion. Commencing with the first ECI grant to the
Homewood-Brushton community in 1998, the SPECS Evaluation Team conducted an intensive evaluation of
the multiple “community processes” representing all of the funded projects. That evaluation has continued
to the present. Analysis of the data from that experience strongly suggests that collectively, these “com-
munity processes” not only fulfilled many of the original ECI goals, they surpassed them. No single out-
come is more telling in that regard, than the spontaneous emergence of the School Readiness Group, a
determined, grassroots community leadership coalition started by the CEO’s of several of the lead agencies
of the former ECI programs. This “super group”, as the SPECS researchers dubbed it, originated in the fall
of 2000. Confronted with the impending cut-off of funds, this coalition of former ECI staff, public school offi-
cials, agency representatives, other supporting organizations, family members, and parents banded
together to consider potential strategies that the communities might pursue to save their programs. That
this effort emerged in the face of adversity, has expanded its membership, implemented a work plan, and
has persisted to the present, demonstrates the core meaning of self determination and “ community
process”. Examination of the central importance of “community process” dynamics has been limited.
Hopefully, this evaluation will help to redress that imbalance.

WHAT WAS THE COMMUNITY PROCESS?

The Beginning
Pittsburgh’s Early Childhood Initiative was based on two assumptions:

• Early, quality-monitored education and childcare could improve the performance of at risk
children, ages 0-5, thereby enhancing their successful entry into public school.
• ECI programs had to be community based and guided by self-determination through neighborhood
decision-making and leadership.

Instructions in the Request for Proposal announcing the availability of ECI funding stipulated that all
applicants had to meet these requirements. (United Way Letter, 1999) A unique concentration of business,
philanthropy, public education, and community energies all directed to the task of bringing early education
to at risk children in the greater Pittsburgh Metropolitan Region. The trio of business, community, and
philanthropy promised to make a potent alliance with which to fight the insidious problem of under
prepared children.

74
 WHO AND WHAT COMPRISED THE COMMUNITY PROCESS?
• Communities extending from Tarentum to Sto-Rox participated in ECI.
• Community based organizations including multi-purpose centers, health care centers, family service
centers, foundations, and public facilities were among the agency - institutional participants.

Participating ECI Communities

Braddock Hill Rankin Upper Hill Rankin
N. Braddock Swissvale Middle Hill E. Pittsburgh Crawford
Roberts Homestead St. Clair Village W. Homestead Arlington
Munhall Arlington Heights Tarentum South Side Brackenridge
Stowe Township Harrison Township McKees Rocks Borough Homewood
East Liberty Brushton Lincoln East Hills Larimer
Lincoln- Lemington Garfield Heights Wilkinsburg
This inventory of ECI communities illustrates the breadth of human capital which was invested in the ECI
effort. Previous reports and documentation have tended to highlight the more commonly known communi-
ties while overlooking some of the lesser-known ones. All communities which had an equal stake in the ECI
effort. A goal of this evaluation is to draw attention to the commonalities which they experienced.

HOW WAS THE COMMUNITY PROCESS ORGANIZED?

• Initial instructions to ECI applicants specified the organizational model - a lead agency and a
community advisory board - which would be responsible for the community process in each funded
program. A summary profile of the characteristics of all ECI programs reveals the differences in
program approaches.
Variation in the structure and size of neighborhood coalitions, the primary function and prior early
education experience of auspice or lead agencies, the size and scope of program goals and objectives,
the historical legacies of communities, and their respective procedural dynamics constituted some
of the major differences.

ECI AND COMMUNITY LEGACIES

That community based programs and dynamics were central to the initial ECI effort was a strong incentive
to communities who considered applying. It was also consistent with the history and legacy of community
activity and development in Pittsburgh. The pioneering work of Action Housing and its urban extension
demonstrations in Homewood-Brushton, Perry Hilltop, and Hazelwood in the early 60’s set the stage for the
antipoverty program which was to come later with its Community Action Program (CAP) and stipulations
that the community action process had to be guided by boards, fifty one per cent of which had to be com-
prised of poor representatives (P.L. 88-452, 1964). The concept of community action was to be extended fur-

75
ther when Comprehensive Health Planning Agencies (CHPA) became a reality in the 70’s and local adviso-
ry boards had to have fifty one percent consumer representation (P.L. 89-748, 1966). Still later, the
Empowerment Zone era of public policy adopted similar emphasis on community representation in the
decision-making structures, which would guide and direct that effort ( HUD, 1994).
The more than forty years of community initiatives represented above left several lasting impressions on
the communities and, importantly, shaped the community environmental context into which the Early
Childhood Initiative was cast. Several of those characteristics are essential to an understanding of the
manner in which communities approached ECI..

WHAT WAS THE COMMUNITY PROCESS EVALUATION?

Developing an Evaluation Strategy
The objective of the community evaluation was to conduct a qualitative, evidence based analysis of the
relationship between community process and the attainment of ECI program goals and expected out-
comes. Although the “community process” concept enjoyed wide usage throughout ECI, an initial task of
the evaluation was to establish a reliable definition for it. The more common, intuitive notion of community
process had to be replaced with a definition, which was both operational and tangible. Toward that end,
“community process” was defined as the exercise of self-determination where local communities decid-
ed the goals and objectives for their neighborhoods, their families, and their children, and the processes
that would be followed in attaining them.
Sensitive to the uniqueness of the multiple localities involved in ECI , SPECS sought to capture and preserve
as much of that variation as possible in its evaluation effort. Foremost was the rejection of any intention to
compare neighborhoods in order to designate which one was best. Although a common practice in social
research, it would be counter productive in community research where factors such as time, place, and his-
tory, to cite but a few, made each community setting distinct and different. A “one-size-fits-all” evaluative
effort would conceal, rather than uncover the richness and variation in effort - the essence of “community
process”. With this conceptual understanding, three tasks awaited the evaluation effort.
Most immediate was the task of devising a way of measuring the concept. In general, scholarly and pro-
fessional literature at the periphery of the subject is voluminous. Volumes exist on community building,
community organization, citizen participation, neighborhood development, and consumer involvement, to
cite but a few. Empirical studies of the structures, processes, and outcomes of community organization
decision-making, however, were virtually non-existent. Consequently, a third problem was the absence of
a conceptual or systematic scheme with which to order the differences in “community processes”.
Given these shortcomings, the evaluation plan was both a challenge and an opportunity –the challenge of
conducting such a study – and the opportunity to focus attention on the importance of community variables.

Perception and Performance

The analysis of “community process” was directed at two variables considered central to the concept. The
first, perception, referred to those things that people believed to be true. To assess those beliefs, the study
designed an efficacy survey to be administered to ECI participants, staff and volunteers alike. The study

76
hypothesized that those perceptions or beliefs of participants would serve as prime motivators of their
behavior. The efficacy study was intended to measure their sense of self-confidence, their confidence in
the capacity of their community organizations, their confidence in the effectiveness of their community
leadership, and ultimately, their confidence in the potential of ECI. Buttressed by a Harvard School of Public
Health study of the effect of neighborhood efficacy in reducing neighborhood violence, the community
process evaluation anticipated that community efficacy might offer an important understanding of how
communities pursued their respective ECI plan. Commenting upon the research, the study’s director,
Harvard’s Felton Earls, M.D., Professor of Human Behavior and Development at Harvard’s School of Public
Health, observed that “it should be recognized that growth or maintenance of neighborhood efficacy
depends upon the commitment of individuals but also external supports that enable trust and cooperation
to flourish.” (p.2)
The centrality of self-efficacy as an indicator of individual and organizational capacity has substantial
documentation in the literature. Early formulation of the concept by Bandura (1977) preceded scores of
studies represented by (Kanter,1983: Hawley,1950,1986; Gist, 1987; Norlin and Chess, 1987; and Pearlmuter,
1998). Parsons’ (1951) classic work on systems theory and Warren’s (1963) studies of communities
contributed to extensive research addressing the relationships between efficacy, individual behavior,
human ecology, community systems, leadership, and organizational change.
With the assessment of collective efficacy as one arm of the evaluation, the analysis of performance
remained as the other. SPECS’ objective here was to ascertain whether programs accomplished what they
proposed to – whether they attained their self-stated goals. Cottrell (1983) fashioned the concept of
community competence as an indicator of a community’s ability to engage in collaborative problem
solving. A high degree of competence, the mark of a good community, was indicated by its ability to “get
things done” (Fellin, 2001).
That ECI programs were confronted with the challenges of “getting things done” was fundamental to the
community process. More important was the type of organizational and developmental change, which was
necessary. Burke and Litwin (1992) proposed that when human service organizations undertake to create
a new service delivery system, change their mission, or extend services to a never served population,
transformative change is involved. Contrasted with the more customary incremental change experienced
by human service organizations, the need for organizational creativity and visionary leadership become
intensified where transformative change is concerned. (Pearlmuter, 1998; Nadler (1982); Westley and
Mintzberg (1991), Guiterrez & DeLoris (1995), and Kanter (1991) treated these themes.
Performance evaluation or “getting things done”, measurement was based on the examination of program
activity data. Comparing program accomplishments to program projections allowed for the assessment of
each program against it, rather than against each other. Organizational characteristics, facilities develop-
ment, and enrollment were the indicators used in the assessment.
With the hypothesized link between perception and performance, the second phase of the evaluation
attempted to examine program accomplishments, their performance. Specifically, the study looked at
whether programs accomplished what they proposed to do – how well they accomplished their self -
stated goals.
A project status report, the second research tool, was used to gather selected program data. Comparing
program accomplishment to program projections allowed us to assess each program against itself, rather
than against each other as discussed earlier. Three indicators, organizational indicators, resource indica-
tors, and enrollment were used in the assessment.

77
 EFFICACY - HOW DID COMMUNITIES VIEW ECI?
The efficacy survey was intended as an assessment of commitment and capacity; commitment to the ECI
concept, and the capacity of self, community, and organization to achieve. Surveys were conducted dur-
ing the period from mid-March 1998 to early spring 2000 and were administered in a two stage, Time I –
Time 2 manner for each community.

Findings
Five issue areas were selected for inclusion in the final report because of their centrality to the evaluation.
These data were intended to provide insight into three dimensions, confidence, commitment, and capaci-
ty. The survey inquired about community perceptions regarding the commonality of goals, the biggest chal-
lenge confronting ECI, what should be changed first, the principal contribution from the community, and the
principal contribution from ECI. The findings revealed a pattern of high community identification with the
goals and objectives of ECI, initially, followed by a period of declining confidence and trust in ECI.

• Communities agreed that the most important goal of ECI was, and should be, to prepare children
to perform well in public school.

Perception of Goals

45 Expectations for Goals

40
35
30
25
20
15
10
5
0
Stengthen Prepare Early Increase
families children to do Education number of
well in school Early Education
programs

78
• The biggest challenges confronting ECI were that parents were too busy working to participate
and that programs were too expensive.

The Most Important Things that the Community

Brings into the ECI Program
Time 1 Time 2
100
90
80
70
Mean Percent

60
50
40
30
20
10
0
Close-up, Credibility Knowledge and Pride in Confidence in Ability to Representation Chance to make
day-to-day for ECI Experience community and community motivate and of the people a difference
knowledge from past its residents' representatives encourage who get things in the community
of the programs resources and resources more parent done in the
community involvement community

Things Community Brings to ECI

• Confidence in the effect of community representation and the adequacy of community resources slipped
as the steward’s commitment to ECI diminished.

The Most Important Things that

ECI Brings to the Community
Time 1 Time 2
100
90
80
70
Mean Percent

60
50
40
30
20
10
0
Technical Opportunity for Money Increased Opportunities Opportunities Public attention Expectation
Assistance community awareness for parental for children and support of change
participation of the importance involvement
in evaluation of early
and development education
programs

Things ECI Brings to Community

79
• ECI was increasingly perceived in a technical and mechanical manner rather than in the visionary,
systems and social change manner, which first attracted community interest.

What to Change First in Early

Childhood Education Programs
Time 1 Time 2
100
90
80
70
Mean Percent

60
50
40
30
20
10
0
Improve Increase Improve Increase Increase Increase Make programs Make programs
knowledge number of training for parental number of funding for less expensive easier to get to
about childcare childcare teachers and involvement programs current for parents
and early providers classroom programs
education and classroom aides
teachers
Things to Change

• Communities agreed that given the ability to do so, the area they would change first was improvement
in teacher training and knowledge about children and early education.

Performance
The performance evaluation divided the community process into three longitudinal stages, planning and
development, application and approval, and implementation and operation. These stages represented the
chronology, which applied to each successful ECI applicant. SPECS examined advisory board and commit-
tee minutes, program correspondence, plans, contracts, job applications, activity reports, business and
revised business plans and other related documents representing program activity from 1996 to 2000.
Attendance at community advisory board meetings of all of the funded programs throughout the period
1998 to 2000 coupled with meetings with the ECIM staff; the ECIM advisory board, and periodic meetings
convened by the steward inform this report.

80
NEIGHBORHOOD PROGRAM PROFILE

Lead # of Adv. Board Elected Regular Meeting

Program # facilities proposed Minutes
Agency Children Membership Officers Mtg. Dates Notices

100 CC; 404 Braddock 40 CDC;

Heritage Chair;
Braddock 500 Prospect 20 CDC; N. Braddock X X X X
Health Fdn. Vice Chair
100 CDC

Kinsley
East Liberty 197 - - - - -
Association

Hill House
Hill District 200 X - - - -
Association

Hosanna
Wilkinsburg 214 100 CCC; 60 CCH; 54 PreSchool X X X
House

Steel Valley AIU 387 293 CCC; 60 CCH; 34 Head Start X X X X X

Focus on
Sto-Rox 547 100 CCC; 40-60 CCH X X X X X
Renewal

Brashear
Southside 28 1 CCC; (28) X X X X X
Association

Highlands
Highlands 240 90 CCC; 36 CCH; 104 Head Start X X X X X
Family Ctr.

Primary
Care Health
Homewood 226 87 CCC; 80 CCH; 42 CCC X - X X X
Services
Inc.

Source: ECI Project Applications; Board of Directors Meetings

Did the program development stage account for differences in performance?

In its evaluation of Stage I, the planning and development stage, SPECS examined the dynamics that
occurred from the initial date of program interest to the submission date of a formal project application.
This investigation was to establish whether differences in development appreciably contributed to differ-
ences in performance. Data revealed that the first community effort occurred in December of 1996 with
subsequent starts occurring throughout the next two years through 1998. Differences in the length of time
programs expended in developing their proposals were also studied.

81
PERFORMANCE

Program 1st Meeting Application Date Time Interval Approval Date Time Interval

Braddock 1 Dec 1996 Oct 1997 10 mos. Jan 1998 3 mos.

Hill District 2 Feb 1997 Jan 1998 11 mos. Mar 1998 2 mos.

Steel Valley 3 Feb 1997 Jan 1998 11 mos. Mar 1998 2 mos.

Southside 4 Mar 1997 Jan 1998 10 mos. Mar 1999 14 mos.

Homewood 5 Mar 1997 Aug 1997 5 mos. Nov 1997 3 mos.

Wilkinsburg 6 Jun 1997 Feb 1998 8 mos. Mar 1998 1 mo.

Highlands 7 Feb 1998 Mar 1998 1 mo. Aug 1998 5 mos.

Sto-Rox 8 Mar 1998 Aug 1998 5 mos.

East Liberty 9 Nov 1998 Mar 1999 4 mos.

Source: “ECI Milestones, ECIM 9/15/00, Project Applications, Board Minutes”

• Programs ranged from one month, the shortest development time to eleven months, the longest.
• Differences in development time did not appear to account for characteristic differences in performance.

Did the program approval stage account for differences in performance?

Stage II included the events, which occurred after submission of a formal application but prior to imple-
mentation. As was the case with development time, differences in approval time occurred for various
reasons, which included revisions of program goals and objectives, modifications of budget projections,
and modifications in enrollment and facilities.

82
 Mean Organization Performance Times
8
7
6

Months
5
4
3
2
1
0
App/Time Apr/Time Before 3/98 After 3/98

Source: ”ECI Milestones“, ECIM 9/15/00, Project Applications

• The mean program development time, the time from the date of the first community meeting until the
submission of a formal application, was 8 months.
• The mean approval time for 8 of the 9 programs was 3.1 months (One program experienced an
exceptional 14 month delay awaiting program approval.)
• Programs, which applied before March 1998, had a mean approval time of 2.2 months.
• Programs, which applied after 1 March 1998, had a mean approval time of 4.6 months, more than
twice that of the earlier applicants.
• Neither development time nor approval time differences, in the main, appeared to account for
programmatic differences. However, the extended time differentials for some programs noted above
would serve as harbingers for philosophical and administrative changes, which would subsequently
affect the status of community programs.

83
Did the implementation stage affect program performance?
The Stage III assessment involved the implementation and administration of approved work plans. SPECS
examination of these data began to reveal a pattern of similar experiences across the community of ECI
programs. Stage III dynamics mainly addressed the completion of proposed facilities and the enrollment of
children, the latter hinging heavily upon the former. Commonly, the completion of tasks revealed an
increasingly confounding pattern of delay, ambiguity, and in some cases contradiction confronting the
community programs. All of the programs were affected by these phenomena with varying results.

Community Process Models

35%
30%
25%
20%
15%
10%
5%
0%
Staff/Directed Hybrid Volunteer Episodic/Ad-hoc
Volunteer Model Directed Model
Model Staff/Assisted
Model

Why were program implementation efforts stymied?

The SPECS evaluation revealed that to this point in the process, community programs were functioning in
environments in which they had the capacity to operate with a high degree of success. They were
working primarily within a consensual context where there was support and understanding of early
education and of the priorities of ECI. The high degree of programmatic successes, i.e., development of
new community coalitions, agreement of work plans, preparation of formal proposals, selection of lead
agencies, development and implementation of advisory boards other policies and procedures attendant to
the process of developing a complex organization had been quite effectively accomplished by most of the
programs. As the programs proceeded beyond this consensual environment, however, a new and different
pattern began to emerge. Heretofore, programs had been preoccupied with internal systems dynamics.
Unlike Stages I and II, Stage III directed primary concern to external systems, those agencies and
bureaucracies in the Greater Pittsburgh Region which were not necessarily consensual as they did not
attribute the same importance to early education. This dichotomy of internal vs. external system priorities
and incompatibility was a major obstacle to effective implementation and a fundamental flaw in the
original ECI design.

84
 Internal Organizational Performance
100
90
80
70
Mean Percent 60
50
40
30
20
10
0
Advisory Regular Standard Leadership Work Plan Implementation
Board Meetings Procedures Positions Schedule
Filled

Internal Performance Indicators

Source: Board Minutes, Performance Reports, Project Applications

External Organizational Performance

100
90
80
70
Mean Percent

60
50
40
30
20
10
0
Center Completion Administrative Staff ECI Students Enrolled
Recruitment/Approval

External Performance Indicators

Confronted with the necessity of securing operating space, either through construction or renovation,
community programs faced a host of “review and approval” actions over which they had no control. Often,
approval was either delayed, causing an impasse in program development or in some instances, not
forthcoming at all. SPECS evaluation found a response pattern that was often arbitrary, sometimes
contradictory, and always costly in terms of program delay.

85
• Commitments for instructional space for early education classrooms were routinely changed,
cancelled, or delayed.
• Site improvement and renovation plans requiring inspections, certifications and licensure were
commonly delayed and slow in materializing causing projects to miss key program deadlines.

How did management and policy changes affect the community process?
• Recurring business plan revisions, a changing ECI concept which communities saw as a contradiction
of the original plan, and a central decision making process from which they were excluded and the
impending threat of the termination of funding contributed to the communities’ loss of confidence in the
vision and promise which had originally characterized ECI.
• The impact of welfare reform policy provided the main emphasis on the need for full day education and
care, whereas the original ECI plan was for half-day coverage. The cost differential has been defined
elsewhere. There was no evidence that ECI sought, or, for that matter, could have obtained a waiver
for ECI families. The evaluation effort was left to confront the irony of the situation. Rather than
supplementing each other, ECI and TANF contradicted each other such that some of the children and
families at highest risk could not qualify for the program.
• The instruction that community programs were to apply for subsidies as a condition of child eligibility
further imposed a new and, in some cases, foreign technology on community based efforts. The
evaluation effort found no evidence of any attempt to either centralize this work in behalf of the
communities, or to provide training to equip them to administer it themselves.
• Lead agency executives formally expressed their discontent over the plight of ECI in a letter to the
Executive Director of the steward in the fall of 2000. Prompted by an effort by the steward to
interrupt the evaluation process and dismiss the evaluation team, the action by the community leaders
was suggestive of how far apart the steward and the communities had become. (See letters of corre-
spondence in the Appendix).

How did the community process change ? What is the School Readiness Group?

In the summer of 2000. executives of the lead agencies began to meet voluntarily to consider strategies,
which would keep ECI alive. This was the beginning formation of a grassroots organization, which would
eventually become the School Readiness Group. Despite termination of all of the original ECI Programs by
the steward in the fall of 2000, several programs persevered and continued to function. Two programs had
been selected by the Heinz Endowment for funding on a demonstration basis. The others were on their
own. Together they formed the School Readiness Group.

86
 COMMUNITY PROCESS EVALUATION CONCLUSIONS
Internal Dimensions
• The community process consisted of two dimensions, an internal, (neighborhood dimension), and an
external (greater Pittsburgh) dimension. ECI programs and policies which failed to recognize this
internal / external systems distinction placed expectations upon neighborhoods which were beyond
their control, conversely absolving other institutions and organizations in the greater Pittsburgh area
of responsibilities they should have been expected to fulfill.
• Throughout the ECI experience, “community process” was incorrectly perceived and treated by the
steward and ECIM as a one dimensional, neighborhood dynamic. This characteristic of ECIM methods
and policies had an arresting effect on each neighborhood program and ultimately on ECI’s cumulative
performance.
• In neighborhood after neighborhood, the pattern of delay and ambiguity on the part of Pittsburgh area
agencies and bureaucracies was manifest in the review and approval of engineering and renovation
contracts, commitment and allocation of space, applications for licensure, review and approval of
variances and building safety regulations, review and approval of program changes, and the review
and approval of the recruitment and selection of program staff. The subsequent requirement that
programs capture state subsidies further compounded administrative functions.
• Internally, neighborhoods demonstrated that they were significantly comparable in accomplishing
those tasks which were within their domain. This included proposal development time, proposal review
and approval time, determination of community need, identification of program goals, designation of a
governance structure, establishment of operating procedures, and the selection of leadership.
• Initially, neighborhood commitment to ECI goals and objectives was comparably high throughout all of
the neighborhoods as the SPECS Efficacy Survey revealed.
• Confidence in ECI waned in all of the neighborhoods as changing program requirements, and recurring
revisions in business plans were unilaterally determined as lead agencies were excluded from the
decision-making.
• References in ECIM documents of the problem of achieving consensus in the neighborhoods were not
supported by the SPECS evaluation. As an internal “community process” dynamic, consensus was one
of the most obvious of neighborhood characteristics. One program consisted of a coalition of 79
separate organizations. In another, the lead agency successfully convened the presidents of all four of
the area neighborhood councils, with the representatives of other major human service organizations,
a feat not previously achieved.
• New infrastructure, an original goal of ECI, occurred in varying forms in all of the communities.
Examples included new, home based day care, which raised the prospect of a new neighborhood
economy, new jobs for neighborhood residents, new affiliations with area schools and school boards,
and new collaborations with other agencies and programs serving the neighborhoods.
• Emergence of the School Readiness Group, a self initiated, coalition of several former neighborhoods
extends the foregoing comment by demonstrating the resiliency and stability of these organizations.
The SRG further demonstrates the capacity of organizational leadership as a feature of new
community infrastructure.
• Shared technology among the neighborhoods is another illustration of the creation and stabilization of
new community infrastructure.

87
 EXTERNAL DIMENSIONS
• Foremost was the absence of a summit agreement among critical “external actors” (local government
and agencies) demonstrating their common commitment to ECI and charging their respective bureau-
cracies to support ECI initiatives.
• The absence of a common, high-level leadership presence contributed to the absence of
centralization of authority and standardization of procedures, which affected the day-to-day
operations of neighborhood projects.
• External responses to neighborhood implementation initiatives affected the efforts of all ECI programs.
As a whole, these responses which affected review and approval of implementation contracts for lead
agencies, review and approval of renovation contracts, approval of space allocation, review and
approval of licensure applications, review and approval of building safety features, review and
approval of fire safety, and review and approval of job applications was often arbitrary and episodic.

LESSONS LEARNED
Opportunity Almost Lost
The ECI experience provides ample lessons for the future. Some of the more compelling are:

• The design and implementation of “community based” programs must assure compatibility between
rhetoric and reality. The early vision of ECI – the rhetoric found wide acceptance and commitment from
all of the participating communities. The rhetoric correctly interpreted need at the community level and
offered an appealing, “community process “ methodology of addressing the problem. The ensuing
sequence of business plan revisions and policy changes demonstrated that the capacity of the effort,
was not up to the promise.
• The perception of the “community process” as a one-dimensional, community-controlled dynamic
rather than an endeavor involving both internal and external systems the support and cooperation of |
numerous sectors of the Greater Pittsburgh Region was a fundamental flaw in ECI. The execution of
top-level agreements between city and county administrations and corporate and philanthropic
leadership could have done much to pave the way for the implementation efforts of the several ECI
programs. Greater standardization and more expeditious responses would have been forthcoming
from the several minor bureaucracies exercising review and approval power had the priority for ECI
been cleared from the top and made known throughout the respective bureaucracies. Does not mean
that the community can go it on its own.
• Similarly, a more accurate understanding and appreciation of the capacities of community leadership
could have contributed to a partnership which would have endured the difficulties occasioned by the
cut back in funding. The increasing exclusion of community leadership contributed to the hardening of
attitudes and the eventual loss of community confidence in ECI leadership.
• More effective use of experienced, professional community organizers at every stage of the
“community process” would have contributed to program outcomes. The Dickerson and Mangus Study
(August, 1999) commissioned by the steward offered a similar recommendation but there was no

88
evidence of its adoption. Only one of the funded programs had a professional community organizer on
staff. The School of Social Work at the University of Pittsburgh is one of the top social work schools in
the country and offers excellent accessibility to a number of accomplished professionals with broad
experience from years of work with communities throughout the Greater Pittsburgh Region. There was
no evidence that its resources were called upon.

89
 C H A P T E R 1 1

WHAT ARE THE LESSONS LEARNED FOR FUTURE ECIs?

CONCLUSIONS AS GUIDEPOINTS FOR ACTION

Despite the success of ECI in achieving important programmatic outcomes for children, ECI did not attain
the ambitious systemic changes envisioned by its originators due primarily to crucial flaws in design and
management. An independently commissioned report by the RAND Corporation (2002) detailed the flaws
with the design, administration, management, oversight, and process of implementing ECI. However, as
Max King, Executive Director of the Heinz Endowments observed (Pittsburgh Post Gazette, 2002), “it would
be one thing to say there were problems in the execution of ECI, but it’s quite another to say it was a waste
of time and money.” Nevertheless, ambitious “natural experiments” like ECI are essential for system
reform that is overdue, pressing, and vital to the welfare of our most vulnerable children,
families, and communities.
The question remains, then, what are the lessons learned from both the successes and the shortcomings
of the Early Childhood Initiative (ECI)? What specific recommendations can policymakers, advocates, and
system reformers glean from ECI and its research outcomes that apply to future early childhood and human
service system initiatives? Future initiatives are not just a possibility, but rather a reality both within
Pennsylvania and across the US.
The following chapter draws practical conclusions from the collaborative SPECS evaluation research with
Pittsburgh’s ECI community leaders; from regional systems of care reform efforts embodied in the
Allegheny County Children’s Cabinet and Birth to Five subcommittee; and a review of the systems of care
reform research literature to present guidepoints as a roadmap for future early care and education and
other systems reform efforts.

1. Prepare the Soil as a Solid Consensus for System Reform Efforts

Despite, arguably, rigorous, unprecedented, and admirable collaboration among the business, corporate,
foundation, and community sectors, the original design of ECI gave priority to the urgent implementation of
the venture before reaching a broad enough consensus from a host of important regional stakeholders.
Ambitious social reform efforts require an analysis of the elements of the “interagency” network that will
effect and, in turn, be affected by the reform. In the case of ECI, too little or no analysis was made of these
systemic issues that were crucial to embracing the ECI model and ensuring its overall success. Similarly,
the designers of ECI and the steward failed to anticipate the potential external forces not evident at plan-
ning that influenced the progress of ECI (i.e., welfare reform, infrastructure costs, lack of state buy-in).
Several examples are instructive:

• Communities faced insurmountable delays in gaining building code approvals because such agencies
were not part of the original stakeholder group for ECI.
• County agencies responsible for support services to families and children with challenging behaviors
and developmental delays or disabilities were not successfully drawn into a partnership with ECI
before implementation.

90
• School districts were viewed as important, eventual, partners in future phases of ECI by the original
designers of the initiative, but the school bureaucracy seemed a too complicated entity to collaborate
with ECI in its inception phase; thus, most school districts knew little about or were unprepared for ECI
and its graduates.
• Many existing providers of early care and education missed the opportunity to become part of ECI
or were not given sufficient or appropriate incentives to participate.
• Availability of quality mentoring was not broad enough due to the lack of integral participation by
already sanctioned state training and technical assistance groups.
Thus, ambitious system reform efforts that seek to “bring initiatives to scale” must gain broad-based
consensus or partnership, at inception, among a vast and typically uncoordinated array of interagency
stakeholders. These “partners” include but are not limited to state and county departments of health,
human services, and housing; the separate early childhood sectors (early intervention, Head Start, early
care and education); school districts; special education intermediate units; and state-wide training,
technical assistance, and quality mentoring groups (KURC, EITA, and HSQIC/DSQIC). System reform
experts across the US have advocated various approaches to coordinate education, health, and human
service system elements in moving toward a coordinated care system (American Psychological
Association, 1996; Melaville & Blank,1991).

2. Pilot Social Reform Initiatives Prior to Implementation

Urgency should not be the primary driver behind systems reform efforts. Similarly, ambitious “natural
experiments” require field-validation phases or pilot-testing before implementation no matter how worthy
or how sensible. Pilot-testing and field-validation of the ECI approach would have provided crucial
feedback about the some of the strong and weak dimensions of the original model. This formative
evaluation would have given planners practical feedback for improving the model and uncovering and
modifying any assumptions or strategies that were flawed or erroneous. This business-oriented “continu-
ous quality improvement” process buttressed by solid research supports would have benefited ECI.
Instead, ECI attempted to reach scale quickly (within 5 years) without the corrective benefit of a pilot-phase
(for example, 18 months). Moreover, in effect, ECI merged scaled implementation, field-validation, quality
improvement, and formative-summative research dimensions from inception. It is a testament to the
validity of the original design which was based on 30 years of early childhood outcome research and the
commitment and ingenuity of community leaders that ECI has been as successful as it has, albeit on a
smaller scale. Pilot-testing is crucial to all experiments, particularly social experiments. It is arguable that
early pilot-testing in ECI would have revealed many of its apparent shortcomings: the demand for full-day
programs; the cost assumptions and inequities; the lack of interagency supports for programs; the
comprehensive service needs of children and families who live in poverty; and the overly complex and
inflexible administrative and management structure for ECI under the original steward.

3. Choose a Capable and Committed Steward

National research on coordinated systems of care advocates for identifying management teams that are
independent, non-profit organizations with self-governing boards to direct new systems reform initiatives
(Melaville & Blank, 1991). These innovative interagency partnership efforts use “collaborative leadership

91
teams” (COLT) to manage such reform efforts. COLTS are composed of representatives of partner agen-
cies in the community each of whom has a critical stake in creating a coordinated care network. These
representatives share leadership responsibilities for the reform effort under the non-profit structure and
sometimes both contribute funds and empower their staffs to “work outside the box” of individual agency
institutional barriers to accomplish integrated care for children and families. Often, such efforts involve
blending elements of the health, human service, and education systems.
The ECI venture failed to establish such an independent, non-profit management group. The RAND report
(2002) cites this element as one of the major administrative and managerial shortcomings of ECI. Steward
is defined as…”one who administers anything as the agent of another or others…appointed by an
organization or [oversight] group to supervise and protect the affairs of that group”. (Webster, 1997). Our
research indicates that the original steward of ECI failed to administer adequately the affairs of ECI for the
ECI Oversight Group. Instead, the steward had glaring conflicts of interest that negatively influenced the
status, progress and future of ECI. These conflicts of interest were most notable in simultaneously raising
funds for ECI, managing its operation, coordinating the ECI “treatment” of quality mentoring, serving as a
funding conduit for the evaluation, and, eventually, usurping ultimate responsibility from a semi-independ-
ent advisory group for final decisions about the disposition of ECI. It is arguable that an independent,
non-profit management group operating under a legally sanctioned oversight board of COLT members
would have allowed ECI to operate in a more capable and committed manner according to its original
mandates and vision.
The original steward of ECI failed to operate and manage ECI in the necessary manner befitting an innova-
tive systems reform initiative; the steward had past experience in fund-raising, but lacked the
critical expertise for operating programs, particularly early childhood programs. Moreover, the steward’s
actions and decisions showed that they were not committed to the vision of the original ECI design. We
agree with RAND that fundamental conflicts arose between the steward and the advisory group and
original designers regarding the basic focus of ECI: expanding child care (“more slots for more kids”) or
creating quality early care and education options. We argue also that this fundamental disagreement
created skewed perspectives about the expected cost of ECI. National data in the cost of quality studies
indicate that the cost of such comprehensive school readiness programs range from $10,000 to $14,000 per
child, rather than the much lower costs ($4,000 to $7,000) associated with custodial daycare and child care
options. It is important to note that the standard cost reference at the inception of ECI was the state
stipend for child care and Head Start’s part-day, part-year model rather than the full-day model that was
necessary for the eventual success of ECI.
The appendix contains correspondence and other records that underscore the failure of the steward to
heed warnings about the impact of its actions on the quality concept in the various ECI communities. In
summary, the steward reneged on expectations to the communities that they would arrange or broker
various service supports and resources to help children and families in the ECI programs. These services
included inclusion support for children with delays and disabilities; behavioral support for children with
challenging behaviors; medical supports for children with health conditions, family resources, and
consultative assistance in writing proposals. The meager supports hastily arranged in 1999 proved to be
ineffective for most ECI programs. Early attempts at encouraging interagency partnership agreements
between the steward, ECI programs, and early intervention programs for children with developmental
delays failed to materialize and were abandoned. As concerns about the apparent overall cost of ECI rose,

92
cuts were made in these essential supports for ECI programs starting as early as September 1998 when
much larger enrollments began. Between January 1997 and September 1998, only about two dozen
children were enrolled in the home-based family child care arrangements within Hawkins Village.
Our analysis leads us to the conclusion that the choice of the original steward for ECI was a major strate-
gic error that compromised the scope, vision, and promise of ECI.

4. Commit to a Timetable for Success

The original conceptual design of ECI was set as a 6-year effort that would be continued through state
funding. Whether a field-validation demonstration project or a scaled effort, the five-year timeline for ECI
was a minimally sufficient timetable for achieving the ambitious objectives of ECI codified in the KPIs. In
the earliest stages of ECI, the institutional barriers and delays imposed by the apparently unexpected
infrastructure costs compromised the original timeline for ECI.
Most demonstration projects funded by the federal government are funded for 3 or 5 year periods and are
established as focused initiatives enrolling from 50-100 participants in one or two sites. The next phase for
full field-validation is continued funding for a replication of the original research model. ECI, as a scaled
initiative, far surpassed these typical parameters in terms of scope, number of children/families (n= 1350),
and sites (25 programs).
Despite the ECI timeline, the original steward failed to commit to the minimally sufficient frame for this
natural experiment and chose to terminate the initiative prematurely after only 25 months of operation
(September 1998-October 2000).
Our research recommends that system reform initiatives like ECI should have the following timeline and
scope requirements:
• Establish a field-validation initiative first to pilot-test the program
• Set the field-validation timetable at 3 years, and commit to its completion
• Overlap the field-validation with the full operation of the initiative by
gradually increasing program expansion to more children and sites
• Commit to an additional 5-10 year full field-validation of the systems
reform initiative to document its efficacy and outcomes

5. Expand Head Start and Existing ECE Programs as the Base

System reforms efforts like ECI can achieve better leverage, integration, capacity, and potential for
success by building on the strongest features of already existing entities. Our research recommends that
expansion of Early Head Start and Head Start and existing early care and education programs would be the
best strategy for achieving ECI’s long-term vision. In fact, some of ECI’s best outcomes were attained in
those communities that used Head Start as its base, most notably Braddock, Sto-Rox, and Steel Valley.
ECI has been the perfect opportunity to justify state funding for Head Start. Moreover, federal mandates
since 1999 have required Early Head Start and Head Start programs to make progress on three of ECI’s most
important benchmarks: partnerships with early care and education programs to expand full-day and
full-year options for children and families; ongoing program-based assessments and evaluations of child

93
outcomes; and curriculum modifications that promote early literacy and other “school readiness” compe-
tencies to nurture early school success for children at developmental risk.
The Head Start Performance Standards are increasingly aligned with the quality standards of NAEYC
(Bredekamp and Copple, 1997). Ongoing mentoring from program administrators and education
supervisors, is required in Head Start. ECI has the opportunity to build on a proven operational and
research track record for Head Start, but to help the communities to expand options for more children and
to reach important success benchmarks. By partnering with Head Start and existing partners in a
fundamental way, ECI can build a legacy by fostering the larger social objective of unifying the early
childhood field with common standards and benchmarks. Allegheny Intermediate Units Early Childhood
and Family Support Senibes is becoming an exemplar of the collaboration among Head Start, early care,
and interagency partners.

As Melaville and Blank (1991) promoted, creative interagency partnerships are the core of successful
human service programs that “work outside the box”. A few such partnerships emerged during the first 3
years of ECI. For example, Allegheny Intermediate Unit’s Early Childhood and Family Support Services,
specifically Allegheny County Head Start, deserves to be highlighted for its “leap of faith”in developing an
emerging collaboration with the Greater Braddock Early Childhood Network and other such ECI leadership
councils such as The Sto-Rox and Steel Valley programs. The attempt to blend Head Start and ECE
operations, especially in Braddock, was initially an exemplar of collaboration. This collaboration should be
credited with successfully preparing ECI children to succeed in their kindergarten programs as well as in
promoting the progress of children and their families while in ECI preschools. This creative Braddock-AIU
joint venture devolved as ECIM’s administrative changes had their negative impacts. However, the leader-
ship team in Heritage Health Foundation and their AIU partners are to be commended for their
positive efforts to make substantial changes in the human service system for young children at develop-
mental risk. These examples will provide a basis for future initiatives that will evolve toward the
integrated system that is needed.

6. Unify the “Unsystem” through Creative Partnerships

Early childhood education is not a unitary field. Because of its fragmentation, the field suffers from a
serious lack of political influence and social worth. The fields of early childhood encompass early
intervention and early childhood special education for children with developmental delays and disabilities
and are sanctioned by state departments of education. Next, Early Head Start and Head Start programs
have separate mandates and regulations for children at-risk and are managed by state satellites of
federal offices affiliated with US Department of Health and Human Services. Public and private early care
and education programs including family child care arrangements are another network with their own
regulations and are typically managed or licensed by the state departments of public welfare.
Credentialling for teachers and care providers vary widely across the three parts of the early childhood
field. Levels of experience range from high school education to master’s level training; in the case of ECI,
over 50% of the providers have only a high school education with an additional 22% having an associate
degree. Thus, the early childhood fields have suffered from a lack of social worth; high turnover rates; poor
pay; and a lack of professionalism that is required for development of the field. We believe that mergers
among the various parts of the early childhood field are necessary to create political influence and to
ensure professionalism within this important and growing specialty.

94
Within the Pittsburgh region, several examples of collaborations and integration among the three major
early childhood fields are apparent; for example innovative collaborations are observed among Allegheny
Intermediate Unit Head Start and Project DART, and associated early care and education programs in the
region. Within Pittsburgh City, the Board of Education Head Start, associated early childhood providers,
and the MOSAIC Early Intervention program have some integrated classroom sites as well with COTRAIC
Head Start. These provide nascent initiatives with varying quality that can be nurtured and expanded.
Beyond early childhood education, there is no cohesive and integrated network of human service agencies
that provide on-site support for programs which care for young children and families with pressing needs.
The lack of a single agency that is responsible for early care and education (like the schools for older
children) is one factor that hampers such integration; however, health and human services are more
experienced with traditional diagnosis- and clinic-based services and fail to understand the unique and
complex needs of infants, toddlers, and preschoolers and their families with medical, behavioral,
developmental, and family support needs. Regarding programs like ECI, children with challenging
behaviors who do not qualify for a mental health or related diagnosis are not afforded the programmatic
support that is necessary to help them adjust in center and family childcare options. Thus, many
vulnerable children “fall through the cracks” and are suspended from their programs. Our research
clearly indicates that there is no integrated network for early care and education with appropriate support
services that meet typical early childhood needs. The “unsystem” is a significant barrier to advancements
in the specialized field of early care and education.
Through our research, we recommend the following starting points in the region to begin to unify the field
and create a support system for early childhood:

• Provide participatory and monetary support to the Allegheny County Children’s Cabinet,
particularly the Birth-Five Subcommittee, in its strategic plan to transform the human service system
to provide more responsive support for all children and families in need.
• Convene, under foundation leadership, a “collaborative leadership team” (COLT) among the early
intervention, Head Start, and early care and education field representatives to explore regional
cooperative working agreements and ways to advance the integration within the Pittsburgh region.
• Expand and support the local collaborations that are emerging between intermediate units, Head
Starts, and early care and education providers
• Convene a working group at the state level of officials from the parallel Departments of Public
Welfare, Education (Bureau of Special Education), Head Start Office (Philadelphia), and Office of
Mental Health to explore inter-specialty integration of the early childhood field. Include
professional organization representation from Head Start, PAEYC, Early Intervention Technical
Assistance, Division for Early Childhood, PACA, and higher education representatives to further
advance the implications of this discussion.
• Gain, for future pilot initiatives, approval from county and state government officials with support of
local politicians, a suspension of the usual regulations that dictate eligibility for programs and
services so that reform efforts can work “outside the box” unimpeded by these barriers to access.

95
7. Establish Crucial Preschool-School Collaborations from the Start
It is arguable that public schools should be given the organizational and fiduciary responsibility for early
care and education. In several areas of the Commonwealth of Pennsylvania (for example, Lancaster City,
Braddock, Farrell, and Duquesne), natural connections, born of necessity, have been developed between
early care and education programs and school districts. Perhaps, the most frequent and notable linkages
are apparent between Head Start programs and school districts. In the Pittsburgh region, the integrations
among Woodland Hills School District kindergarten programs, Braddock 4-KIDS, and Allegheny
Intermediate Unit Head Start have been in process for 3 years. In the City, Head Start is operated by the
Pittsburgh Public Schools Board of Education.
The impact of ECI and similar local ventures across Pennsylvania (e.g., Erie, York, Lancaster) has
stimulated important linkages among school districts and early care and education programs. Moreover,
state and federal government mandates to develop uniform standards for learning outcomes has further
encouraged a downward extension to preschool curriculum standards. In several areas, curriculum
specialists for school districts and representatives of the early care and education sector have forged
sequenced curriculum objectives together (i.e., Philadelphia). There exists also a synergistic movement
to promote “developmentally appropriate practices” in kindergarten settings while simultaneously
encouraging preschool teachers and administrators to infuse a much greater emphasis on precursors to
early school success into all early learning activities and routines.
Many experts in the early childhood field express grave doubts, justifiability about the capacity and
compatibility of the public school’s assuming responsibility for early care and education programs. The
centralized, often overly structured, and normative focus of public education seems to clash with the
developmental character of preschool education. Yet, experiments such as the Chicago Longitudinal Study
(Reynolds, 2002) and the Inclusive Kindergarten program at 4-KIDS in Braddock (Bagnato, 2001)
demonstrate that these barriers can be overcome with important mutual benefits to children, parents, and
teachers alike.
Despite the misgivings, the future of early care and education seems to rest with public education (Dr.
Stanley Herman, personal communication, 2002). In this instance, some degree of centralization appears
important characterized by uniform leadership, quality benchmarks, learning standards, philosophies, and
professional credentialing and pay standards.
Based on our community research with broad stakeholders, we recommend the following next steps to
begin to forge the crucial interrelationships among school districts and diverse preschool programs:

• Nurture and study the developmental process of both urban and rural local partnerships among
school districts and preschool education programs (particularly Head Start) as natural experiments
that can function as prototypes to be adopted broadly to encourage state-wide reforms.
• Capitalize on already established fora like the Southwestern Pennsylvania Superintendents
Consortium, PAEYC, School Performance Network, PA Partnerships for Children, and the EPI
Center to explore the process and dynamics of school-preschool partnerships.
• Recognize that with integral community partnerships that school districts can provide the missing
ingredients of support and capacity for organizing and ensuring the success of ECE joint ventures.

96
 • Forge ready-made linkages among school districts and preschools by using the early literacy
movement to influence the development of cohesive curricula and content standards.
• Fund and pilot creative vehicles to smooth the transition of children from preschool to kindergarten,
including collaboratively designed transition procedures, and “child progress portfolios”
for parent-teacher communication (Bagnato, 2001).

8. Commit to Community-based Leadership Rather than Rhetoric

The SPECS results indicate that much of ECI’s shortcomings can be traced to the failure of the steward to
commit to the principle of community-based leadership and its collaborative process. In general, in those
communities in which community leadership was strongest and exerted its considerable authority and
ingenuity to overcome barriers erected by the steward’s attempt at centralized control, the better the
quality, progress, and outcomes of the ECI program. In those communities with less well-developed
leadership, the steward’s authoritarian style usurped the process and established numerous obstacles to
progress. In general, these communities took longer to gain acceptance of their proposals, enrolled fewer
children, and had difficulty maintaining quality in their programs. The promised support for these
communities to formulate and write their initial proposals for funding never materialized and placed them
at a considerable disadvantage compared to communities with more seasoned leadership. Research
demonstrates that a facilitative and collaborative management style promotes the attainment of objectives
in systems reform efforts.
It is clear that effective community-based leadership enabled many programs to succeed and thrive when
ECI’s promise was short-circuited. Throughout ECI’s short history, we observed and leaders’ reported that
a mismatch was apparent from early in the process between the rhetoric and the reality of the lack of
commitment to community-based leadership by the steward. Effective community-based leadership is the
equalizer and “social glue” in systems reform ventures like ECI.
The rise of the School Readiness Group (SRG; Grom etal, 2000) occurred out of necessary to rescue the
“dream” of ECI as it was dismantled prematurely. In contrast to the steward, SRG is the embodiment of
effective community leadership and management that is inclusive, collaborative, and entrepreneurial. SRG
created a home for all willing ECI neighborhood representatives. This cross-neighborhood consortium
uses the community collective to create negotiating power; forge common objectives; and to influence
local public policy. SRG was created from the necessity for survival and illustrates the type of integration
in community process that would have taken 10 years to achieve let alone, 3 years. Community-based
leadership was fundamental to ECI’s success.

9. Infuse Outcomes Evaluation Into All Early Care and Education Programs
The early care and education field has historically regarded assessment and evaluation as undesirable
and foreign to its philosophy and practices. Several events are beginning to melt this resistance and to
demonstrate the positive benefits of outcomes evaluation for both individual programs and the field. With
publication of the revised developmentally appropriate practice standards (Bredekamp & Copple, 1997), the
Division for Early Childhood Recommended Practices (Neisworth & Bagnato, 2001), and the Head Start
Performance Standards on assessment in early childhood settings, professionals and paraprofessionals
in all early childhood fields are striving to acquire specific competencies that will guide their use of assess-

97
ment to accomplish several purposes. The primary purposes (Bagnato, Neisworth, & Munson, 1997;
Neisworth & Bagnato, 2001; Bagnato & Neisworth, 2000) are identifying individual strengths and needs,
planning individualized instruction, and monitoring progress in development and learning. The importance
of monitoring progress has gained impetus with the recent federal and state mandates to ready children to
succeed in school.
The SPECS approach of “authentic assessment and program evaluation in early childhood settings”
(Bagnato, Suen, Brickley, Smith-Jones, Dettore, 2002) has been field-validated to operationalize the
developmentally-appropriate practice standards in the various sectors of the early childhood field.
Moreover, while placing emphasis on observation of naturally-occurring competencies of children in
everyday routines, the SPECS approach links both assessment and programming to curricular objectives
that are important precursors or “building blocks” for early school success. SPECS ensures a natural
assessment approach that aligns with the typical early childhood behavioral style of young children, yet, is
rigorous in capturing the early learning needs and accomplishments of children. SPECS allows teachers
to pinpoint each child’s particular skill needs, to design tailored plans for care and instruction, and to track
each child’s progress toward early learning objectives as a result of the their quality teaching. SPECS has
researched the match between developmental competencies as precursors to the Pennsylvania Academic
Standards at 3rd grade.
The positive benefits of infusing sensible and responsive assessment and evaluation methods into early
care and education programs is no where more evident than in the Pittsburgh’s Early Childhood Initiative
(ECI). Our SPECS methods and results demonstrate clearly that early childhood professionals can blend
child assessments naturally into everyday routines. With this ongoing assessment, they can know their
children better, plan quality learning activities, record successes, communicate and collaborate with
parents, and prove the results of their expert teaching and nurturing care. SPECS methods and results
advocate for the parents and their children by helping preschool and school teachers to rely on each other.
SPECS results give kindergarten and primary grade teachers the information they need to continue to
support the successes of children from quality preschools.

98
 A P P E N D I X A

RESEARCH REFERENCES
American Psychological Association (1994). Comprehensive and coordinated psychological services for children: A
call for service integration. Washington, DC: American Psychological Association, Task Force on Comprehensive and
Coordinated Psychological Services for Children Ages 0-10.
Annie E. Casey Foundation. (1994). Kids Count Data Book: State Profiles of Child Well-Being. Baltimore, MD: Author.
Bagnato, SJ. (1997). SPECS Evaluation of the Heinz Pennsylvania Early Childhood Initiatives. Pittsburgh, PA: Heinz
Endowments.
Bagnato, S.J. (1999). Psychology in education as developmental healthcare: A proposal for fundamental change and
survival. In R. Talley, T. Kubiszyn, M. Brassard, and R. Short (Eds.), Making psychologists in schools indispensable:
Critical questions and emerging perspectives (pp. 159-164). Washington, DC: American Psychological Association.
Bagnato, S.J. (2000). Developmental outcomes and impact of the Early Childhood Initiative (ECI): Statistical analysis
of the first 16 months. Pittsburgh, PA: Heinz Endowments.
Bagnato, SJ, Suen, HK, Brickely, D, Smith-Jones, J, Dettore, E (2002). Child developmental impact of Pittsburgh’s Early
Childhood Initiative (ECI) in high-risk communities: First-phase authentic evaluation research, Early Childhood
Research Quarterly, 17(4) (in press).
Bagnato, SJ., Neisworth, J.T., & Munson, S.M. (1997). LINKing assessment and early intervention: An authentic cur-
riculum-based approach (3rd ed.). Baltimore, MD: Brookes.
Bandura, A. (1977). Self – efficacy: Toward a unifying theory of behavior change. Psychological Review, 84, 191 – 215.
Bandura, A. (1982). Self – efficacy mechanism in human agency. American Psychologist, 37, 122 – 147.
Barnett, S.W. (1995). Long-term effects of early childhood programs on cognitive and school outcomes. The Future of
Children, 5(3), 25-50.
Brashear Association Early Childhood Initiative. Project Application. January, 1998.
Brashear Association Early Childhood Initiative Board of Directors and Task Force Meetings: ECI Steering Committee,
Mar 20, 1997; ECI Working Group, Apr 21, 1997, May 15, 1997, Jun 19, 1997, Jul 17,1997, Aug 28,1997, Sept 18, 1997, Oct
23, 1997, Nov 13, 1997, Jan 7, 1998, May 14,1998, Aug 4,1998; ECI Task Force, Apr 8,1999, Apr 20,1999 May 11,1999, Jun
2, 1999, Jun 16, 1999, Jul 7, 1999, Sept 15, 1999, Oct 5, 1999, Nov 3, 1999, Nov 17, 1999; St. Clair Task Force, Dec 7, 1999,
Jan 19, 2000; ECI Task Force, Feb 9, 2000, Feb 23, 2000; St. Clair Task Force, Mar 8, 2000, Mar 13, 2000, Mar 22, 2000,
Apr 7, 2000, Apr 20, 2000; ECI Task Force, May 4, 2000, May 25,2000; St. Clair Task Force , Jun 1, 2000; ECI Task
Force, Jun 13, 2000.
Bredekamp, S., & Copple, C. (Eds). (1997). Developmentally appropriate practice in early childhood programs.
Washington, DC: NAEYC.
Bryant, D., & Maxwell, K. (1997). The effectiveness of early intervention for disadvantaged children. In M.J. Guralnick
(Ed.), The effectiveness of early intervention. (pp. 23-46). New York: Academic Press.
Burke, W.W. & Litwin, G.H. (1992). A causal model of organizational performance and change. Journal of
Management, 18, 523 – 545
Byrd, O. Memo to United Way Board of Directors. Early Care and Education Budgets Former ECI Neighborhoods –
Period 10-1 through 12-31-00. August 23, 2000.
Campbell, F., & Ramey, C. (1995). Cognitive and school outcomes for high risk African-American students at middle
adolescence: Positive effects of early intervention. American Educational Research Journal, 32, 743-772.
Children’s Cabinet (2001). Children’s Cabinet System of Support Initiative: A Strategic Plan. Pittsburgh, PA: Allegheny
County Department of Human Services, Children’s Cabinet. May 6.

99
Children’s Cabinet (2001). A vision of Children’s Cabinet Services: New model’s committee report to the Children’s
Cabinet, Pittsburgh, PA: Allegheny County Department of Human Services, August 8.
Children’s Cabinet- Birth to Five Committee (2002). System of care for families with very young children, Birth to Five
committee report submitted to the full Children’s Cabinet. Pittsburgh, PA: Allegheny County Department of Human
Services, May, 2002.
Christian, K., Morrison, F.J., & Bryant, F.B. (1998). Predicting kindergarten academic skills: interactions among child
care, maternal education, and family literacy environments. Early Childhood Research Quarterly, 13, 501-521.
Clifford, D., Peisner-Feinberg, E., Culking, M., Howes, C., & Kagan, S.L. (1998). Quality care does mean better out-
comes. Retrieved March 15, 2002, from the University of North Carolina, Chapel Hill, National Center for Early
Development and Learning Web site: http://www.fpg.unc.edu/~ncedl/Pages/spotlt2.htm
Commonwealth of Pennsylvania. (1991). Early Intervention Standards. Harrisburg, PA: Bureau of Special Education.
Cook-Kilroy, J., Bagnato, S. J., Smith-Jones, J., & Matesa, M. (1998). Program Quality Profile for Early Childhood
Settings. Unpublished Assessment.
Cottrell, L. S., Jr. (1983). The competent community. In R. Warren & L. Lyon (Eds.), New perspectives on the American
community. Homewood, IL: Dorsey
Crick, J.E., & Brennan, R.L. (1983). Manual for GENOVA: A generalized analysis of variance system. ACT Technical
Bulletin, No. 43. Iowa City, IA: ACT.
Dickerson, L. & Mangus, P. Executive Briefing: The Revised Early Childhood Initiative Business Plan. August 1999,
Pittsburgh, PA.
East Liberty Early Childhood Initiative. Project Application. November, 1998.
Farran, D.C. (2000). Another decade of intervention for children who are low income or disabled: What do we know?
In J. P. Shonkoff & S.J. Meisels (Eds.), Handbook of early childhood intervention (2nd ed., pp. 511-548). Cambridge,
England: Cambridge University Press.
Fellin, Phillip. (2001). The community and the social worker. Itasca, IL: Peacock.
Fujiura, G.T., & Yamaki, K. (2000). Trends in the demography of childhood poverty and disability. Exceptional Children,
66(2), 187-199.
Gill, S., Reynolds, AJ., (1999). Educational expectations and school achievement of urban African American children.
Journal of School Psychology, 37, 403-424.
Gill, BP, Dembosky, JW, Caulkins, JP (2002). A “noble bet” in early care and education; Lessons learned from one
community’s experience. Pittsburgh, PA: RAND Education.
Gist, M. (1987). Self-efficacy: Implications for organizational behavior and human resource management. Academy of
Management Review, 12, 472–485.
Greater Braddock Early Childhood Network. Project Application. October 14, 1997, Revised January 8, 1998.
Greater Braddock Early Childhood Network Board of Directors Minutes: Aug. 10, 1998, Aug. 24, 1998, Sept. 14, 1998,
Oct. 19, 1998, Nov. 9, 1998, Nov. 30, 1998, Dec. 21, 1998, Jan. 11, 1999, Jan 14, 1999, Jan. 22, 1999, Feb. 22, 1999, Apr 12,
1999, Jul 12, 1999, Sept 13, 1999, Oct. 11, 1999, Nov.22, 1999, Jan. 10, 2000, Feb. 15, 2000, Mar. 27, 2000, May 8, 2000.
Grom, R. M. Letter to Edward Gagnadan, Executive Director, Allegheny County Housing Authority. March 4, 1998.
Pittsburgh, PA.
Grom, Robert M. Letter to George McClomb. SPECS. Community Activity Report. September 7, 2000.
Grom, Robert M. Letter to John Mitchell. Allegheny Intermediate Unit. November 2, 1998.
Grom, R.M. Letter to John Sava. United Way of Allegheny County. Contracts Delay. June 10, 1998.
Grom, R.M. Letter to Mardi Isler, ECIM. CY 1999 Budget Amendment Request. May 25, 1999.

100
Gutierrez, L., Glen aye, L., & Deloris, K. (1995). The organizational context of empowerment practice: Implications for
social work administration. Social Work, 40, 249 – 258.
Hammill, D.D., Leigh, J.E., Pearson, N.A., Maddox, T. (1998). Basic School Skills Inventory-3: A Readiness measure for
teachers. Austin, TX: PRO-ED.
Hawley, A. (1950). Human ecology: a theory of community structure. New York: Roland.
Hawley, A. (1986). Urban ecology. Chicago: University of Chicago Press.
Head Start Bureau (1999), Head Start Program Performance Standards and Other Regulations. Washington, DC:
Department of Health and Human Services, Administration on Children, Youth, and Families, Head Start Bureau.
Head Start Bureau (2000). The Head Start path to positive child outcomes. Washington, DC: Department of Health
and Human Services, Administration for Children and Families, Head Start Bureau.
Heritage Health Foundation. .Greater Braddock Early Childhood Network CY 2000 Budget Narrative, Year 3 Summary.
November, 1999.
Heritage Health Foundation. Greater Braddock Early Childhood Network CY 1999 Budget Narrative, Year 2 Summary.
November, 1998.
Highlands Early Childhood Initiative. Community Needs Assessment. 1998.
Highlands Early Childhood Initiative. Project Application. March, 1998.
Hill District Early Childhood Initiative. Project Application. March, 1998.
Hresko, W., Miguel, S., Sherbenou, R., & Burton, S. (1994). Developmental observation checklist system (DOCS).
Austin, TX: PRO-ED.
Isler, M. Letter to John F. Mitchell. ECI application review and comment letter. January 30, 1998.
Kanter, R. M. (1983). The change masters. New York: Simon & Schuster.
Kanter, R. M. (1991). Change–master skills: What it takes to be creative. In J. Henry & D. Walker (Eds.), Managing inno-
vation. London: Sage Publications.
Max King quote on first paragraph of Chapter 10 Lessons Learned. “It would be one thing to say there were prob-
lems in the execution of ECI, but it’s quite another thing to say it was a waste of time and money” (Pittsburgh Tribune
Review, 3/’29/02)
Long, Norton E. (1958). The local community as an ecology of games. American Journal of Sociology, 64, 3.
Marcon, R. A. (1999). Differential impact of preschool models on development and early learning of inner-city chil-
dren: a three-cohort study. Developmental Psychology, 35, 358-375.
McCall, R.B., Ryan, C.S., & Green, B.L. (1999). Some non-randomized constructed comparison groups for evaluating
age-related outcomes of intervention programs. American Journal of Evaluation, 2(20), 213-226.
Meisels, S.J., Bickel, D., Nicholson, J., Xue, Y., & Atkins-Burnett, S. (2001). Trusting teacher’s judgments: A validity
study of a curriculum-embedded performance assessment in kindergarten to grade 3. American Educational
Research Journal, 38(1), 73-95.
Meisels, S.M., Atkins-Burnett, S., Xue, Y., Nicholson, J., Bickel, D., & Son, S. (in press). Creating a system of account-
ability: The impact of instructional assessment on elementary children’s achievement test scores. American
Educational Research Journal.
Melaville, AI, Blank, MJ (1991). What it takes: Structuring interagency partnerships to connect children and families

101
with comprehensive services. Washington, DC: Education and Human Services Consortium.
Merrell, K. W. (1994). Preschool and kindergarten behavior scales. Brandon, VT: Clinical Psychology Publishing
Company.
Meyer, W. Memo to United Way Board of Directors. Interim United Way ECI Plan. October 7, 1999. Pittsburgh, PA.
Minutes of meeting with Nancy Kurensky, Director of the Family Center. Tarentum, PA. October 30, 1998.
Minutes of meeting with Valeta Ford, Coordinator for Implementation. Homewood, PA. January 23, 1998.
Minutes of the PCHS/Early Childhood Initiative Neighborhood Council
Meetings: Feb. 26, 1998, Mar.12, 1998, Mar.19, 1998, Mar.26, 1998, Apr. 23, 1998, May. 21,1998, June 25, 1998, Jul. 30,
1998, Dec. 2, 1998, Jan. 27, 1999
Nadler. D. (1982). Managing transitions to uncertain future states. Organizational Dynamics, 11 (1), 37-45.
NAEYC. (1994). NAEYC Position Statement: A Conceptual Framework for Early Childhood Professional Development.
Young Children 49 (3): 68-77.
Neisworth, JT, Bagnato, SJ (2000). Recommended practices in assessment (17-27. In Sandall, S, McLean, ME, Smith,
BJ, DEC recommended practices in early intervention/early childhood special education. Longmont, CO: Sopris West
Neisworth, J.T. & Bagnato, S.J. (2000). Recommended practices in assessment. In S. Sandall, M.E., McLean, & B.J.
Smith (Eds.). DEC recommended practices in Early Intervention/Early Childhood Special Education (pp. 17-27).
Longmont, CO: Sopris West.
New York Times. (August 14, 1997). Study of Chicago finds neighborhood efficacy explains reduction in violence.
NICHD Early Child Care Research Network (1999). Child outcomes when child care classes meet recommended stan-
dards for quality. American Journal of Public Health, 89, 1072-1077.
Norlin, J.M. & Chess, W.A. (1997). Human behavior and the social environment: Social systems theory. Boston: Allyn
& Bacon.
Parsons, Talcott. (1951). The social system. Glencoe, IL: Free Press.
Pearlmuter, Sue. (1998). Self-efficacy and organizational change leadership. Administration in Social Work, 22, 23-58.
Primary Care Health Services,Inc., Activity Report. November 10, 1999.
Primary Care Health Services, Inc., Early Childhood Initiative. Project Application. August, 1997.
Ramey, C.T., & Ramey, S.L., (1998). Early intervention and early experience. American Psychologist, 53(2), 109-120.
Reynolds, AJ, Temple, JA, Robertson, DL, Mann, EA (2001). Long-term effects of an early childhood intervention on
educational achievement and juvenile arrest; A 15-year follow-up of low-income children in public schools. Journal
of the American Medical Association, 285(18), 2339-2380.
Sandall, S., McLean, M.E., & Smith, B.J. (Eds.). (2000). Recommended practices in early intervention/early childhood
special education. Longmont, CO: Sopris West.
Schweinhart, L., & Weikart, D. P. (1997). The High/Scope preschool curriculum comparison study through age 23.
Early Childhood Research Quarterly, 12, 117-143.
Shera, W., & Page, J. (1995). Creating more effective human service organizations through strategies of empowern-
ment. Administration in Social Work, 19 (4), 1-15.
Shonkoff, JP (Ed) (2000). From neurons to neighborhoods: The science of early childhood development. Washington,
DC: National Academy of Sciences, Committee on Integrating the Science of Early Childhood Development, National
Research Council and Institute of Medicine, National Academy Press.
SPECS Evaluation: Steel Valley Early Childhood Initiative Profiles of Progress,September 1, 1998-May 12, 2000.
SPECS Evaluation: Summary Data of the Highlands Childhood Initiative for Community and Program Strand. January
1999-March 2000.

102
Steel Valley Early Childhood Initiative. By-laws. Not Dated.
Steel Valley Early Childhood Initiative. Project Application. January, 1998.
United States Congress. P.L. 88-452: The economic opportunity act of 1964. Washington: United States Government
Printing Office.
United States Congress. P.L. 89-749: Comprehensive health planning and service act. Washington: United States
Government Printing Office.
United States Department of Housing and Urban Development and United States Department of Agriculture. (1994).
National empowerment zones and enterprise communities. Washington: United States Government Printing Office.
United Way of Allegheny County. ECI the Early Childhood Initiative: 1999 Revised Business Plan. July 26, 1999,
Pittsburgh, PA.
United Way of Allegheny County. ECI the Early Childhood Initiative: Year 2000 Revised Business Plan.
United Way of Allegheny County. ECI Milestones, October 1994 through August 2000. Pittsburgh, PA.
United Way of Allegheny County. ECI Participating Facilities Update: Allegheny County Early Care and Education
Licensed Facilities. August 2000. Pittsburgh, PA.
United Way of Allegheny County. Overview. Publication date fall 1999.Vygotsky, L.S. (1978). Mind in society: The devel-
opment of the higher psychological processes. Cambridge, MA: Harvard University Press.
Warren, Roland. (1963). The community in America. Chicago: Rand McNally.
Westley, F., & Mintzberg, H. (1991). Visionary leadership and strategic management. In J. Henry & D. Walker (Eds.),
Managing innovation. London: Sage Publications.
Whitebook, M. & Bellm, D. (1996). Mentoring for Early Childhood Teachers and Providers: Building Upon and Extending
Tradition. Young Children 52(1), 59-64.
Whitebook, M., Howes, C., & Phillips, D. (1990). Who cares? Childcare teachers and the quality of the California early
childhood mentor teacher program. Washington D.C.: National Center for Early Childhood Work Force.
Wilkinsburg Early Childhood Initiative. Project Application. February, 1998.

103
 A P P E N D I X B

WHAT STATISTICAL ANALYSES AND RESULTS

UNDERSCORE ECI OUTCOMES AND CONCLUSIONS?

HOI K. SUEN & SANG HA LEE

PENNSYLVANIA STATE UNIVERSITY

STEPHEN J. BAGNATO
CHILDREN’S HOSPITAL OF PITTSBURGH
THE UCLID CENTER AT THE UNIVERSITY OF PITTSBURGH

This report represents the summative evaluation of the impact of the ECI model on child developmental
progress based on data gathered over 33 months (1997-2000).
As data collection progressed, it was necessary to adjust the evaluation data analytic design to
accommodate realistic programmatic conditions. The final impact evaluation design consists of three
analytical components. First, some data for a comparison group have become available. Hence, a first
series of analyses consist of comparisons between ECI children and children in the comparison group.
These analyses attempt to identify whether ECI has an impact on the child. Second, children’s develop-
ment was evaluated over time against their expected maturation without intervention. These analyses
attempt to assess the impact of ECI on child outcomes by using ECI children as their own control group.
Finally, the impact of consultation, community characteristics, program quality, and family/parental
variables on child development, after controlling for prior differences, was evaluated through a series of
OLS/HLM analyses. These analyses attempt to identify programmatic variables that might have a direct
impact on child outcomes.
This report is organized as follows: First, a summary of evaluation variables with available data is provid-
ed. Next, the results of psychometric analyses of the scores from various data collection tools and scales
are summarized. This will be followed by a summary of results and interpretation for each of the three
series of analyses described above. Finally, a number of specific questions that have been raised by the
overall UCLID evaluation staff are addressed.

104
 SUMMARY OF EVALUATION VARIABLES WITH AVAILABLE DATA
Over the past four years, due to various programmatic factors, a number of changes have been made with
the data collection design. Subsequent to these changes, data for a number of measures were available
for each child. Each measure for each child was taken a number of times. The number of time points varies
from child to child as a function of the duration of the child’s participation and the exact date of the child’s
enrollment. Hence, one child may have only 2 data points on a certain measure while another child may
have 6 measures. The first measure of one child may correspond to, for instance, the 6th measure of
another child in terms of calendar dates. Below is a summary of all the measures for which each child had
data for at least one time point:

Child measures:
• DOCS — Developmental Observation Checklist System composite and subscale scores: Each child has
been measured from 1 to 6 times.
DOCS has the following subscales:
o Cognitive development
o Language development
o Social skills
o Motivation
• DOC_delayed — Whether a child is considered developmentally delayed based on DOCS scores.
• PKBS — Preschool and Kindergarten Behavior Scale
o PKBS-social — Social subscale
o PKBS-behavior — Behavior subscale
• PKB_delayed — Whether a child is considered to have potential behavioral problems based on
PKBS scores.
• BSSI — Basic School Skills Inventory (BSSI) composite and subscale scores at the end of ECI.
BSSI has the following subscales:
o Spoken language skills
o Reading skills
o Writing skills
o Math skills
o Classroom behavior
o Daily living skills
• BSSI-K — Basic School Skills Inventory (BSSI) composite and subscale scores at the end of
kindergarten.

105
Family measures:
• FSS — Family Support Scale
• PBC — Parent Behavior Checklist
• PSI — Parent Stress Index
Program measures:
• PQP — Program Quality Profile
• PIP — Program Intensity Profile
• CON — Consultation Survey
These measures and a number of derivations from these measures formed the database for all the analy-
ses summarized in this report.

PSYCHOMETRIC ANALYSES OF SCALES

In 1999, a small sample of data became available for preliminary psychometric analyses. Subsequently, a
number of generalizability analyses were conducted for the available measures. Based on the results of
these analyses, a number of changes were made. Data for some of the earlier measures were found to
contain too much random measurement error and were thus discontinued. There was also evidence that
some other earlier measures were redundant and these were thus discontinued as well.
Table 1 provides a summary of the G-coefficients estimated for measures that were eventually kept and
used. The G- and D-study designs varied for each of these studies as a function of available data. The
G-coefficient in each case represents the reliability of using the data from a single administration using a
single rater on all the items.

Table 1
G-coefficients of data for measures that were kept

Scale # of items # of respondents # of time points G-coefficient

DOCS - Teacher Data (Time 1 and Time 2) 475 90 2 0.970

FSS - Parent Data (Time 1) 18 245 1 0.767

PBC - Parent Data (Time 1) 70 243 1 0.963

PKBS - Social Skills Teacher Data (Time 1 and Time 2) 34 85 2 0.833

PKBS - Problem Behavior Teacher Data 42 85 2 0.813

(Time 1 and Time 2)

PKBS - Social Skills Teacher and Parent Data 34 103 2 0.397

(Time 1)

PKBS - Problem Behavior Teacher 42 103 2 0.412

and Parent Data (Time 1)

PSI - Parent Data (Time 1) 24 240 1 0.906

106
The table above provides a summary of scales that were discontinued due to various reasons. DSPEC and
ISPEC data were found to be unstable and were discontinued in subsequent data collection activities.
SEEC scores, although with high degrees of reliability, were found to correlate with DOCS scores highly
with a correlation coefficient ranging from 0.83 to 0.92. Consequently, SEEC was considered redundant and
was dropped from subsequent data collection activities. Two measures of community efficacy have failed
to attain adequate levels of reliability to be considered potentially meaningful contributors to the analyses.
These measures were thus analyzed qualitatively and separately from the impact evaluation reported here.
The analyses of community and neighborhood efficacy are being conducted by the community strand of
the evaluation project. No reliability analysis was done for any of the program measures due to the
inherently small sample of programs.

Table 2
G-coefficients of data for measures that were dropped

Scale # of items # of respondents # of time points G-coefficient

DSPEC - Parent Data (Time 1) 19 172 1 0.845

DSPEC - TeacherData (Time 1) 19 197 1 0.896

DSPEC - TeacherData (Time 2) 19 100 1 0.837

DSPEC - Teacher Data (Time 1 and Time 2) 19 95 2 0.554

DSPEC - Teacher and Parent Data (Time 1) 19 121 2 0.417

ISPEC - Parent Data (Time 1) 24 45 1 0.929

ISPEC - Teacher data (Time 1) 24 46 1 0.982

ISPEC - Teacher and Parent Data (Time 1) 24 24 2 0.000

SEEC - Parent Data (Time 1) 74 68 1 0.972

SEEC - Teacher Data (Time 1) 74 77 1 0.976

SEEC - Teacher and Parent Data (Time 1) 74 36 2 0.894

107
 IMPACT EVALUATION RESULTS AND INTERPRETATION
Overall, three groups of analyses were conducted to evaluate the impact of ECI on various child outcome
measures. The exact design, method, results and interpretation of the analyses are summarized below.

ANALYSIS GROUP A: COMPARISONS AGAINST NON-ECI CHILDREN

The purposes of the first group of analyses are: 1) to compare the posttest child measure scores of ECI
children against those of non-ECI children after controlling for other factors including family characteris-
tics, duration of treatment, age, and prior differences; 2) to determine whether there was an interaction
effect between being developmentally delayed and the length of treatment on posttest child measure
scores; and 3) to determine whether there was an interaction effect between being identified as having
potential behavioral problems and the length of treatment on posttest child measure scores.
These were accomplished through a series of 21 multiple polynomial regressions, each examining the
degree to which ECI participation and other variables account for differences in one of the child outcome
measures. For non-ECI children, a group of children participating in a Headstart program in the CenClear
Center of Phillipsburg, Pennsylvania was used. Data on the CenClear children and their families we
re gathered along with data gathered for ECI children in a similar manner. The only exception is that data
for the program measures were not available for CenClear children.
A series of multiple regressions were conducted using all available data for both ECI and CenClear
children. Because of the absence of program measure data for CenClear children, these measures were
not examined in this series of 21 analyses. Although the ECI/CenClear combined sample has a total of 2,065
children, only a subset of these children were included in each analysis. For each of the analyses, only
children with available data on all the measures considered were included. Consequently, the sample size
for each of the 21 analyses ranges from as many as 949 to as few as 18. The sample size for each of the
analyses will be reported along with the results below.
The statistical method used for each of these analyses was a standard multiple regression with one
polynomial predictor variable – the quadratic function of age. This quadratic term was included because
preliminary analyses results had indicated a polynomial relationship between age and many of the child
outcome measures. In all analyses, the posttest score of a child outcome measure was used as the
criterion variable. Since children enrolled in the program at different times and may have exited program
at different times; further, the intake assessment of one child might take place after the exit assessment of
another child, there was no specific time point that can be considered pre- or posttest for all children.
Therefore, posttest score was defined as the score on the very last available assessment of the child. For
one child, it might be the 2nd assessment; for another, it might be the 6th. This definition of posttest has
two inherent problems: 1) variation in posttest scores may be due to length of program participation and
not simply having participated in the program; and 2) variation in posttest scores may be due to age differ-
ences at the time of program exit. Therefore, age at time of posttest and length of treatment were
included as predictors in these regressions to control for these two factors.

108
The results of each of the 21 analyses are summarized below. For each analysis, a table summarizing all
the significant predictors, their corresponding regression weights, standard errors, and t-test results is
provided. Additionally, the effect size of the statistical model in the form of R-squared value, as well as the
corresponding standard error of estimate and adjusted R-squared value are reported. A forward stepwise
procedure was used in each of the analyses. Finally, an interpretative narrative is provided to assist in
identifying potentially programmatically meaningful results.
ADVANCE SUMMARY OF FINDINGS FROM ANALYSIS GROUP A:
To re-iterate, the primary purposes of the first group of analyses are: 1) to compare the posttest child
measure scores of ECI children against those of non-ECI children after controlling for other factors
including family characteristics, duration of treatment, age, and prior differences; 2) to determine whether
there was an interaction effect between being developmentally delayed and the length of treatment on
posttest child measure scores; and 3) to determine whether there was an interaction effect between being
identified as having potential behavioral problems and the length of treatment on posttest child measure
scores.
On the question of comparing ECI children against CenClear children, the results indicated that, after
controlling for other factors, there were some differences. These include the following: 1) Children in ECI
scored 4.5 points higher on the DOCS language posttest, and 2) children in ECI scored 3.5 points higher on
the DOCS social posttest. To the extent that the two groups are comparable after controlling for the
variables of family characteristics, duration of treatment, age, and pretest scores, it can be concluded that
ECI has an impact on language development and social development as measured by DOCS. Whether
these differences may be attributed to other uncontrolled factors such as rural vs. urban setting, and so on
cannot be determined by these analyses.
On the question of whether there was an interaction effect between being developmentally delayed and
the length of treatment on posttest scores, there is some evidence that such interactions exist in the areas
of cognitive development and language development. Specifically, while children who were not delayed
can be expected to gain 1 DOCS cognitive posttest score point for every 7 additional days of treatment,
children who were delayed would gain that same 1 point for only 5 additional days of treatment. Similarly,
while children who were not delayed can be expected to gain 1 DOCS language posttest score point for
every 8 additional days of treatment, children who were delayed would gain that same 1 point for only 6
additional days of treatment.
Analyses of BSSI scores at the end of kindergarten were based on 18 cases only since only 18 children had
available data. The results of these analyses are unlikely to be generalizable and should be considered
tentative and inconclusive at this point.

109
Analysis A1: DOCS posttest composite scores as child outcome measure - Sample size: 949 Cases

Predictor Coef SE Coef T P

Constant 70.12 13.68 5.13 0.000

Docs_age 0.33426 0.02954 11.31 0.000

Docs_age2 -0.00009627 0.00000935 -10.30 0.000

PSUT_Pre 0.57977 0.02270 25.54 0.000

Docs_trt 0.26954 0.01855 14.53 0.000

PCDIRawT -1.0583 0.3319 -3.19 0.001

S = 60.83 R-Sq = 90.3% R-Sq(adj) = 90.2%

Interpretation: After accounting for differences in chronological age and differences in pretest DOCS
scores, no evidence of difference between CenClear and ECI, nor evidence of interaction effect between
being developmentally delayed and length of treatment, nor evidence of interaction effect between having
prior behavioral problems and length of treatment was found on DOCS posttest composite scores.
However, the length of treatment (DOCS_TRT) was found to be positively related to posttest composite
scores after accounting for age differences.
Each additional 3.7 days of treatment is associated with an increase of 1 DOCS posttest composite score
point. Further, parental distress—dysfunctional interaction (PCDIRawT) scores were found to be negative-
ly related to DOCS posttest composite scores. Each score point reduction in dysfunctional interaction is
associated with an increase of slightly over 1 DOCS posttest composite score point.

110
Analysis A2: DOCS posttest cognitive scores as child outcome measure - Sample Size 837 Cases

Predictor Coef SE Coef T P

Constant 34.953 9.904 3.53 0.000

Docs_age 0.13294 0.01764 7.54 0.000

Docs_age2 -0.00003753 0.00000541 -6.94 0.000

PSUCOGT_pre 0.612277 0.02920 20.98 0.000

DOCS_trt 0.14248 0.01117 12.76 0.000

DOCSTr*delay 0.05422 0.02275 2.38 0.017

FSST -0.16885 0.08278 -2.04 0.042

ESTOTT1 0.4595 0.2123 2.16 0.031

PCDIRawT -0.5438 0.1905 -2.85 0.004

S = 31.51 R-Sq = 88.1% R-Sq(adj) = 88.0%

Interpretation: After accounting for differences in chronological age and differences in pretest DOCS
cognitive scores, no evidence of difference between CenClear and ECI nor evidence of interaction effect
between having prior behavioral problems and length of treatment was found on DOCS cognitive posttest
scores.
However, the length of treatment (DOCS_TRT) was found to be positively related to DOCS cognitive posttest
scores after accounting for age differences. Each additional 7 days of treatment is associated with an
increase of 1 DOCS cognitive posttest score point. There was an interaction effect between length of
treatment and being developmentally delayed. While children who were not delayed can be expected to
gain 1 DOCS cognitive posttest score point for every 7 additional days of treatment, children who were
delayed would gain that same 1 point for only 5 additional days of treatment. Parental distress—dysfunc-
tional interaction (PCDIRawT) scores were again found to be negatively related to DOCS cognitive posttest
scores. Each 2 score points reduction in dysfunctional interaction is associated with an increase of
slightly over 1 DOCS posttest cognitive score point. Parental behavior-expectation pretest (ESTOTT1)
scores were found to be positively related to DOCS cognitive posttest scores. Each 2 additional points in
ESTOTT1 is associated with a gain of 1 point on the DOCS cognitive scale. Surprisingly, family support
scale (FSST) scores were found to be negatively related to DOCS cognitive posttest scores. Every 6 points
increase in FSST is associated with a 1-point drop in DOCS cognitive posttest scores.

111
Analysis A3: DOCS posttest language scores as child outcome measure

Predictor Coef SE Coef T P

Constant 4.888 9.026 0.54 0.588

Docs_age 0.13905 0.01570 8.86 0.000

Docs_age2 -0.00004184 0.00000486 -8.60 0.000

PSULANT_pre 0.61308 0.02950 20.78 0.000

DOCS_trt 0.12412 0.01002 12.38 0.000

DOCSTr*delay 0.05056 0.02067 2045 0.015

Group 4.540 2.081 2018 0.029

FSST1 -0.15009 0.07523 -1.99 0.046

ESTOTT1 0.4659 0.1944 2.40 0.017

PCDIRawT -0.4472 0.1732 -2.58 0.010

S = 28.62 R-Sq = 87.6% R-Sq(adj) = 87.5%

Interpretation: After accounting for differences in chronological age and differences in pretest DOCS
language scores, no evidence of interaction effect between having prior behavioral problems and length
of treatment was found on DOC language posttest scores.
However, children in ECI scored 4.5 points higher on the DOCS language posttest, after accounting for prior
differences and age. Further, the length of treatment (DOCS_TRT) was found to be positively related to
DOCS language posttest scores after accounting for age differences. Each 8 days of treatment is
associated with an increase of 1 DOCS language posttest score point. There was an interaction effect
between length of treatment and being developmentally delayed. While children who were not delayed
can be expected to gain 1 DOCS language posttest score point for every 8 additional days of treatment,
children who were delayed would gain that same 1 point for only 6 additional days of treatment. Parental
distress—dysfunctional interaction (PCDIRawT) scores were again found to be negatively related to DOCS
language posttest scores. Each 2 score points reduction in dysfunctional interaction is associated with an
increase of slightly over 1 DOCS posttest language score point. Parental behavior-expectation
pretest (ESTOTT1) scores were found to be positively related to DOCS language posttest scores. Each 2
additional points in ESTOTT1 is associated with a gain of 1 point on the DOCS language scale. Surprisingly,
family support scale (FSST) scores were found to be negatively related to DOCS language posttest scores.
Every 7 points increase in FSST is associated with 1 point drop in DOCS language posttest scores.

112
Analysis A4: DOCS posttest social scores as child outcome measure - Sample size: 837 cases

Predictor Coef SE Coef T P

Constant 38.260 6.209 6.16 0.000

Docs_age 0.12168 0.01119 10.87 0.000

Docs_age2 -0.00003604 0.00000351 -10.26 0.000

PSUSOCT_pre 0.48732 0.02504 19.46 0.000

DOCS_trt 0.078172 0.006286 12.44 0.000

Group 3.514 1.438 2.44 0.015

ESTOTT1 0.3167 0.1348 2.35 0.019

PCDIRawT -0.3790 0.1199 -3.16 0.002

FSST1 -0.11595 0.05213 -2.22 0.026

S = 19.87 R-Sq = 86.3% R-Sq(adj) = 86.2%

Interpretation: After accounting for differences in chronological age and differences in pretest DOCS
social scores, no evidence of interaction effect between being developmentally delayed and length of
treatment, nor evidence of interaction between having prior behavioral problems and length of treatment
was found on DOC social posttest scores.
However, children in ECI scored 3.5 points higher on the DOCS social posttest, after accounting for prior
differences and age. Further, the length of treatment (DOCS_TRT) was found to be positively related to
DOCS social posttest scores after accounting for age differences. Each 13 days of treatment is associat-
ed with an increase of 1 DOCS social posttest score point. Parental distress—dysfunctional interaction
(PCDIRawT) scores were again found to be negatively related to DOCS social posttest scores. Each 2.6
score points reduction in dysfunctional interaction is associated with an increase of slightly over 1 DOCS
posttest social score point. Parental behavior-expectation pretest (ESTOTT1) scores were found to be
positively related to DOCS social posttest scores. Each 3.2 additional points in FSST is associated with a
gain of 1 point on the DOCS social scale. Surprisingly, family support scale (FSST) scores were found to be
negatively related to DOCS social posttest scores. Every 8 points increase in FSST is associated with a
1-point drop in DOCS social posttest scores.

113
Analysis A5: DOCS posttest motor scores as child outcome measure - Sample Size: 917 cases

Predictor Coef SE Coef T P

Constant 47.634 5.971 7.98 0.000

Docs_age 0.15570 0.01311 11.87 0.000

Docs_age2 -0.00004215 0.00000397 -10.63 0.000

PSUMOTT_pre 0.47615 0.02541 18.74 0.000

DOCS_trt 0.085034 0.007815 10.88 0.000

DOC_delay -10.615 5.161 -2.06 0.040

FSST1 -0.08264 0.05728 -1.44 0.149

PCDIRawT -0.4791 0.1305 -3.67 0.000

S = 23.27 R-Sq = 90.1% R-Sq(adj) = 90.0%

Interpretation: After accounting for differences in chronological age and differences in pretest DOCS
motor scores, no difference in DOCS motor score was found between CenClear and ECI children, nor
evidence of interaction effect between being developmentally delayed and length of treatment, nor
evidence of interaction between having prior behavioral problems and length of treatment was found on
DOC motor posttest scores.
However, the length of treatment (DOCS_TRT) was found to be positively related to DOCS motivation
posttest scores after accounting for age differences. Each 12 days of treatment is associated with an
increase of 1 DOCS motivation posttest score point. Children who were developmentally delayed based on
DOCS pretest scores were found to score 11 points lower on the DOCS motivation posttest. Parental
distress—dysfunctional interaction (PCDIRawT) scores were again found to be negatively related to DOCS
motivation posttest scores. Each 2 score points reduction in dysfunctional interaction is associated with
an increase of 1 DOCS posttest motivation score point. Surprisingly, family support scale (FSST) scores
were found to be negatively related to DOCS cognitive posttest scores. Every 12 points increase in FSST
is associated with 1 point drop in DOCS motivation posttest scores.

114
Analysis A6: PKB posttest social scores as child outcome measure - Sample size: 715 cases

Predictor Coef SE Coef T P

Constant 65.495 3.986 16.43 0.000

PKB_age 0.017656 0.002124 8.31 0.000

PKBAT_pre 0.007286 0.002571 2.83 0.005

DOC_delay -7.928 1.369 -5.79 0.000

PKB_delay -10.581 1.549 -6.83 0.000

PCDIRawT -0.50435 0.08121 -6.21 0.000

S = 13.47 R-Sq = 23.2% R-Sq(adj) = 22.7%

Interpretation: After accounting for differences in chronological age and differences in pretest PKB social
scores, no difference in PKB social score was found between CenClear and ECI children, nor evidence of
interaction between being developmentally delayed and length of treatment, nor evidence of interaction
between having prior behavioral problems and length of treatment was found on PKB social posttest
scores.
Children who were developmentally delayed based on DOCS pretest scores were found to score 8 points
lower on the PKB social posttest. Children who were classified as having behavioral problems based on
PKB-behavior pretest scores were found to score 11 points lower on the PKB social posttest. Parental dis-
tress—dysfunctional interaction (PCDIRawT) scores were found to be negatively related to PKB social
posttest scores. Each 2 score points reduction in dysfunctional interaction is associated with an increase
of 1 PKB posttest social score point.

115
Analysis A7: PKB posttest behavior problems scores as child outcome measure - Sample size: 680 cases

Predictor Coef SE Coef T P

Constant 56.358 9.266 6.08 0.000

PKBBT_Pre 0.016943 0.008376 2.02 0.044

PKB_age -0.010117 0.004185 -2.42 0.016

PKB_delay 28.676 2.887 9.93 0.000

ESTOTT1 -0.5173 0.1947 -2.66 0.008

DCRawT1 0.4292 0.1183 3.63 0.000

S = 23.86 R-Sq = 22.3% R-Sq(adj) = 21.6%

Interpretation: After accounting for differences in chronological age and differences in pretest PKB
behavioral problem scores, no difference in PKB behavioral problem score was found between CenClear
and ECI children, nor evidence of interaction between being developmentally delayed and length of treat-
ment, nor evidence of interaction between having prior behavioral problems and length of treatment was
found on PKB behavioral problem posttest scores.
Children who were classified as having behavioral problems based on PKB-behavior pretest scores were
found to score 29 points higher on the PKB behavioral problem posttest. Parental distress—dysfunctional
interaction (PCDIRawT) scores were found to be positively related to PKB behavioral problem posttest
scores. Each 2.3 score points reduction in dysfunctional interaction is associated with a reduction of 1 PKB
posttest behavioral problem score point. Parental behavior-expectation pretest (ESTOTT1) scores were
found to be negatively related to PKB behavioral problem posttest scores. Each 2 additional points in
ESTOTT1 is associated with a drop of 1 point on the PKB behavioral problem posttest score.

116
Analysis A8: BSSI posttest (at end of ECI) spoken language scores as child outcome measure

Predictor Coef SE Coef T P

Constant -21.37 27.03 -0.79 0.432

PKB_delay -18.842 5.886 -3.20 0.002

ESTOTT1 1.4107 0.5519 2.56 0.013

DCRawT1 0.8650 0.2963 2.92 0.005

S = 14.05 R-Sq = 20.8% R-Sq(adj) = 16.7%

Interpretation: No difference in BSSI spoken language score was found between CenClear and ECI chil-
dren, nor evidence of interaction between being developmentally delayed and length of treatment, nor
evidence of interaction between having prior spoken languages and length of treatment was found on
BSSI spoken language posttest scores.
Children who were classified as having behavioral problem based on PKB—behavior pretest scores were
found to score 18 points lower on the BSSI spoken language posttest. Parental distress—dysfunctional
interaction (PCDIRawT) scores were found to be positively related to BSSI spoken language posttest
scores. Each 1.2 score points reduction in dysfunctional interaction is associated with a reduction of 1
BSSI posttest spoken language score point. Parental behavior-expectation pretest (ESTOTT1) scores were
found to be positively related to BSSI spoken language posttest scores. Each additional point in ESTOTT1
is associated with a increase of 1.5 points on the BSSI spoken language posttest score.

117
Analysis A9: BSSI posttest (at end of ECI) reading scores as child outcome measure - Sample size: 61 cases

Predictor Coef SE Coef T P

Constant -22.15 16.86 -1.31 0.194

Bssit_age -0.018266 0.007415 2.46 0.017

PKB_delay -5.947 2.892 -2.06 0.044

FSST1 -0.18674 0.08632 -2.16 0.035

NSTOTT1 0.4331 0.2343 1.85 0.070

S = 7.223 R-Sq = 32.0% R-Sq(adj) = 27.2%

Interpretation: After controlling for age difference, no difference in BSSI reading score was found
between CenClear and ECI children, nor evidence of interaction between being developmentally delayed
and length of treatment, nor evidence of interaction between having prior behavioral problems and length
of treatment was found on BSSI reading posttest scores.

Children who were classified as having behavioral problems based on PKB—behavior pretest scores were
found to score 6 points lower on the BSSI reading posttest. Family support scale (FSST) scores were found
to relate negatively with BSSI reading posttest scores. Each 5 score points reduction in family support
scale score is associated with an increase of 1 BSSI posttest reading score point. Parental behavior-
nurturing pretest (NSTOTT1) scores were found to be positively related to BSSI reading posttest scores.
Each additional 2.3 points in NSTOTT1 is associated with an increase of 1 point on the BSSI reading
posttest score.

118
Analysis A10: BSSI posttest (at end of ECI) writing scores as child outcome measure

Interpretation: There was insufficient evidence that BSSI posttest writing scores were related to any of
the variables investigated.

Analysis A11: BSSI posttest (at end of ECI) math scores as child outcome measure

Predictor Coef SE Coef T P

Constant -1.78 15.53 -0.77 0.909

Bssit_age 0.015636 0.007173 2.18 0.033

PKB_delay -8.603 2.769 -3.11 0.003

FSST1 -0.17201 0.07576 -2.27 0.027

S = 7.173 R-Sq = 30.3% R-Sq(adj) = 26.8%

Interpretation: After controlling for age difference, no difference in BSSI math score was found between
CenClear and ECI children, nor evidence of interaction between being developmentally delayed and length
of treatment, nor evidence of interaction between having behavioral problems and length of treatment was
found on BSSI math posttest scores.
Children who were classified as having behavioral problems based on PKB—behavior pretest scores were
found to score 8 points lower on the BSSI math posttest. Family support scale (FSST) scores were found
to relate negatively with BSSI math posttest scores. Each 6 score points reduction in family support scale
score is associated with an increase of 1 BSSI posttest math score point.

Analysis A12: BSSI posttest (at end of ECI) classroom behavior scores as child outcome measure

Predictor Coef SE Coef T P

Constant 43.583 1.387 31.43 0.000

PKB_delay -18.208 4.386 -4.15 0.000

S = 11.77 R-Sq = 18.1% R-Sq(adj) = 17.0%

Interpretation: BSSI classroom behavior scores were not found to be related to any of the variables inves-
tigated, except for being classified as having behavioral problems based on PKB behavior pretest scores.
Specifically, children who were classified as having behavior problems were found to score 18 points
lower on the BSSI classroom behavior posttest scale.

119
Analysis A13: BSSI posttest (at end of ECI) daily living scores as child outcome measure - Sample size: 62 cases

Predictor Coef SE Coef T P

Constant 15.77 10.70 1.47 0.146

PKB_delay -6.975 2.877 -2.42 0.018

ESTOTT1 0.6585 0.2561 2.57 0.013

S = 7.571 R-Sq = 18.7% R-Sq(adj) = 15.9%

Interpretation: BSSI daily living scores were not found to be related to any of the variables investigated,
except for being classified as having behavioral problems based on PKB behavior pretest scores and par-
ent behavior – expectation scores. Specifically, children who were classified as having behavior problems
were found to score 7 points lower on the BSSI daily living posttest scale. On the other hand, every 1.5
point increase in parent behavior—expectation scores is associated with an increase of 1 point on the
BSSI-daily living posttest score.

Analysis A14: BSSI posttest (at end of ECI) total scores as child outcome measure - Sample size: 80 cases

Predictor Coef SE Coef T P

Constant 57.6250 0.7927 72.70 0.000

PKB_delay -7.125 2.507 -2.84 0.006

S = 6.726 R-Sq = 9.4% R-Sq(adj) = 8.2%

Interpretation: BSSI posttest total scores were not found to be related to any of the variables investigat-
ed, except for being classified as having behavioral problems based on PKB behavior pretest scores.
Specifically, children who were classified as having behavioral problem during pretest were found to score
7 points lower on the BSSI total posttest scale.

120
Analysis A15: BSSI kindergarten (at end of kindergarten) spoken language scores as child outcome measure

Predictor Coef SE Coef T P

Constant -47.66 28.43 -1.68 0.114

NSTOTT1 2.2956 0.7406 3.10 0.007

PCDIRawT 1.2840 0.5263 2.44 0.028

S = 12.73 R-Sq = 44.3% R-Sq(adj) = 36.9%

Interpretation: BSSI kindergarten spoken language scores were not found to be related to any of the vari-
ables investigated, except for parent behavior – nurturing scores and parental stress – dysfunctional
behavior scores. Specifically, each point increase in parent behavior – nurturing score is associated with
2.3 points increase in BSSI kindergarten spoken language scores. Conversely, every point increase in 1.3
points increase in BSSI kindergarten spoken language scores. Caution: these results are based on a sam-
ple size of 18 children. As indicated by the substantial shrinkage in the adjusted R-square value, there is
no support for generalization of these results beyond the 18 children.

Analysis A16: BSSI kindergarten (at end of kindergarten) reading scores as child outcome measure

Predictor Coef SE Coef T P

Constant -16.25 15.15 -1.07 0.301

DOC_delay 21.204 8.891 2.38 0.031

NSTOTT1 1.3623 0.4763 2.86 0.012

S = 8.389 R-Sq = 54.8% R-Sq(adj) = 48.8%

Interpretation: BSSI kindergarten reading scores were not found to be related to any of the variables
investigated, except for being classified as developmentally delayed based on DOCS pretest scores and
for parent behavior – nurturing scores. Specifically, developmentally delayed children scored 21 points
higher on the BSSI-reading scale at the end of kindergarten than did non-delayed children. Further, each
point increase in parent behavior – nurturing scores is associated with 1.4 point increase in BSSI-reading
kindergarten score. Caution: these results are based on a sample size of 18 children. As indicated by the
substantial shrinkage in the adjusted R-square value, there is no support for generalization of these results
beyond the 18 children.

121
Analysis A17: BSSI kindergarten (at end of kindergarten) writing scores as child outcome measure - Sample size: 18 cases

Predictor Coef SE Coef T P

Constant -63.71 21.81 -2.92 0.012

FSST1 -0.022022 0.08351 -2.64 0.021

ESTOTT1 0.8781 0.4155 2.11 0.054

NSTOTT1 1.2405 0.2471 5.02 0.000

PCDIRawT 0.7729 0.2022 3.82 0.002

S = 4.025 R-Sq = 69.0% R-Sq(adj) = 59.5%

Interpretation: No difference in BSSI kindergarten writing score was found between CenClear and ECI
children, nor evidence of interaction between being developmentally delayed and length of treatment, nor
evidence of interaction between having prior behavioral problems and length of treatment was found on
BSSI kindergarten writing scores.
BSSI kindergarten writing scores were found to be negatively related to family support scale scores and
positively related to parent behavior – expectation scores, parent-behavior – nurturing scores, and
parental stress – dysfunctional interaction scores. Every 5 point increase in family support scale scores is
associated with 1 point drop in BSSI kindergarten writing score. Each 1.1 point increase in parent
behavior – expectation score is associated with 1 point increase in BSSI kindergarten writing score, each
1 point increase in parent behavior – nurturing score is associated with 1.2 point increase in BSSI
kindergarten writing score; and every 1.3 increase in parental stress – dysfunctional interaction score is
associated with 1 point increase in BSSI kindergarten writing score. Caution: these results are based on a
sample size of 18 children. As indicated by the substantial shrinkage in the adjusted R-square value, there
is no support for generalization of these results beyond the 18 children.

122
Analysis A18: BSSI kindergarten (at end of kindergarten) math scores as child outcome measure

Predictor Coef SE Coef T P

Constant -14.85 16.02 -0.93 0.368

NSTOTT1 1.6317 0.4995 3.27 0.005

S = 9.062 R-Sq = 40.0% R-Sq(adj) = 36.3%

Interpretation: BSSI kindergarten math scores were not found to be related to any of the variables inves-
tigated, except for parent behavior – nurturing scores. Each additional point in parent behavior – nurtur-
ing score is associated with an additional 1.6 points in BSSI kindergarten math score. Caution: these results
are based on a sample size of 18 children. As indicated by the substantial shrinkage in the adjusted R-
square value, there is no support for generalization of these results beyond the 18 children.

Analysis A19: BSSI kindergarten (at end of kindergarten) classroom behavior scores as child outcome measure

Predictor Coef SE Coef T P

Constant -66.24 18.59 -3.56 0.003

NSTOTT1 3.0057 0.4857 6.19 0.000

PCDIRawT 2.0658 0.5054 4.09 0.001

DCRawT1 -1.0162 0.3413 -2.98 0.010

S = 8.322 R-Sq = 76.4% R-Sq(adj) = 71.4%

Interpretation: BSSI kindergarten classroom behavior scores were not found to be related to any of the
variables investigated, except for parent behavior – nurturing score; parental stress – dysfunctional
interaction scores, and parental stress – difficult child scores. Each additional point in parent behavior –
nurturing score is associated with an additional 3 points in BSSI kindergarten classroom behavior score.
Each 1 point increase in parental stress – dysfunctional interaction score is associated with an increase
of 2 points on the BSSI kindergarten classroom behavior scale. Finally, each point reduction in parental
stress – difficult child score is association with 1 point increase in BSSI kindergarten classroom behavior
score. Caution: these results are based on a sample size of 18 children. As indicated by the substantial
shrinkage in the adjusted R-square value, there is no support for generalization of these results beyond the
18 children.

123
Analysis A20: BSSI kindergarten (at end of kindergarten) daily living scores as child outcome measure

Predictor Coef SE Coef T P

Constant 3.70 17.75 0.21 0.837

NSTOTT1 1.4149 0.5535 2.56 0.021

S = 10.04 R-Sq = 29.0% R-Sq(adj) = 24.6%

Interpretation: BSSI kindergarten daily living scores were not found to be related to any of the variables
investigated, except for parent behavior – nurturing scores. Each additional point in parent behavior –
nurturing score is associated with an additional 1.4 points in BSSI kindergarten daily living score. Caution:
these results are based on a sample size of 18 children. As indicated by the substantial shrinkage in the
adjusted R-square value, there is no support for generalization of these results beyond the 18 children.

Analysis A21: BSSI kindergarten (at end of kindergarten) total scores as child outcome measure

Predictor Coef SE Coef T P

Constant -4.39 17.46 -0.25 0.805

NSTOTT1 1.3897 0.4548 3.06 0.008

PCDIRawT 0.8847 0.3232 2.74 0.015

S = 7.817 R-Sq = 46.0% R-Sq(adj) = 38.8%

Interpretation: BSSI kindergarten daily living scores were not found to be related to any of the variables
investigated, except for parent behavior – nurturing scores and parental stress – dysfunctional interaction
scores. Each additional point in parent behavior – nurturing score is associated with an additional 1.4
points in BSSI kindergarten total score. Each 1.1 point increase in parental stress – dysfunctional
interaction scores is associated with an increase of 1 point on the BSSI kindergarten total scale. Caution:
these results are based on a sample size of 18 children. As indicated by the substantial shrinkage in the
adjusted R-square value, there is no support for generalization of these results beyond the 18 children.

124
 ANALYSIS GROUP B:
COMPARISONS AGAINST EXPECTED MATURATION OF ECI CHILDREN

An inherent limitation of the analyses in Group A above is the uncertainty of the comparability of CenClear
children. CenClear children are a group of children participating in a Headstart program in rural central
Pennsylvania. To the extent both ECI and CenClear children come from economically disadvantaged
backgrounds, these children are comparable. However, they are also different in many factors, including
such factors as urban/rural backgrounds, racial/ethnic and cultural backgrounds. Therefore, the results of
Group A analyses are not totally conclusive. More conclusive results would have been obtained through
a true experimental design in which children are randomly assigned to treatment and control groups. Such
an approach, while scientifically sound, would be neither ethical nor feasible in the case of ECI.
In order to compare ECI children against children who are clearly comparable, one solution is to compare
the ECI children against themselves. That is, we can compare the outcome measures of the children after
a period of program participation against some expected scores of the same children without intervention.
To accomplish this, we used a modification of the constructed comparison group method suggested by
McCall, Ryan and Green (1999). For early childhood programs in which there is a large variation of
chronological ages of children at intake, McCall et al. suggested modeling the pretest scores of children
as a function of chronological age. The slope of the linear model is derived through ordinary least square
regression. This slope is then used to estimate the expected change over the time period of program
intervention for each individual child. The expected change is added to the pretest score of the child to
produce an expected posttest score without intervention. The actual observed posttest scores can then
be compared against these expected scores to identify the impact of the treatment. To eliminate the
chance hypothesis, a one-sample dependent t-test would be used. A fundamental deficiency of the McCall
et al. method is its failure to take into account the potential error of the slope estimate. To remedy this
problem, we modified the McCall et al. method as follows:

1. We submitted the data on chronological age and outcome scores at intake (i.e., Time point 1) to
a polynomial regression analysis to identify the best prediction model using the ordinary least
square method as described by McCall et al.
2. We then applied the equation to the chronological age at later time points to derive the expect
ed outcome score for each child at each of these time points.
3. We then compared the expected outcome scores without treatment against the actual observed
outcome scores at each time point to determine the effect of treatment at these time points.
4. We tested for the statistical significance of the difference between the observed posttest score
mean and the mean of the expected no-treatment scores at each time point through a modified
one-sample dependent t-test as follows

xd
t=
s o2 + s p2 + s e2 - 2rop so sp
n

125
 where xd is the mean difference between the observed posttest score and the mean predicted
untreated score, so is the standard deviation of the observed posttest scores, sp is the standard
deviation of the predicted untreated scores, se is the standard error of prediction, rop is the cor
relation between the observed and the predicted scores, and n is the number of children involved
in the program that is being evaluated. The probability for this t-test is to be evaluated against
n-1 degrees of freedom.

The net effect of this method is to compare the posttest scores of ECI children against their scores they
would have obtained had there not been any program intervention after accounting for natural maturation.
We further extended this method to include all other potential predictors of posttest scores besides age
(e.g., family support, parental stress). In order to apply this method, the R-square value for polynomial
model needs to be substantially high so that the error around the expected scores is minimized.
After applying step 1 in the process described above, we found a strong quadratic function between
chronological age and DOCS composite pretest scores. We then applied the polynomial equation to the
chronological age of each child at the time of posttest to derive the expected DOCS composite posttest
score of the child under the scenario of no treatment and with only the effects of natural maturation. We
then compared the children’s observed DOCS posttest scores against their own expected scores.
Additionally, we charted the changes in observed and expected scores over a number of time points to
track the growth in terms DOCS composite scores of children both with and without ECI over a number of
time points. Since different cohorts of children had different numbers of data time points, we charted the
growth of 4 different cohorts separately: those with 3, 4, 5, and those with 6 observation time points. The
results are presented below in Analyses B1 through B4. For children with 3 and 4 observation time points,
we further divided them between those who would have been classified as developmentally-delayed
based on their pretest scores to determine the difference in growth trajectories between delayed and non-
delayed children, if any. The results of the sub-group analyses are presented in Analyses B5 and B6 below.
Only ECI children were used in this series of analyses since the purpose was to discern the effect of ECI.
Therefore, no CenClear data were included in any of the analyses in Group B. As was the case with
Analysis Group A, the size of the sample with available data differed from analysis to analysis. The
sample size of data for each analysis is reported along with the results below. As before, an interpretation
of the results is provided for each analysis.

ADVANCE SUMMARY OF FINDINGS FROM ANALYSIS GROUP B:

The purpose of Analysis Group B is to determine if ECI has an impact over and beyond what can be
expected by maturation using ECI children as their own control. This removed some of the uncertainties
regarding results from Analysis Group A but is limited to DOCS composite posttest scores only and not
other child measures since DOCS composite scores were the only measure found to be sufficiently
related to maturation to make these analyses meaningful.
After examining four different cohorts, the general trend is clear. Children participating in ECI progressed
at a faster rate than can be explained by their own natural maturation. The accelerated developmental
growth curve was found to be as much as 21 percentile points in 20 months for the cohort with the longest
participation period. In all cases with all cohorts, the departure from maturation expectation occurred

126
immediately and can be observed in the second observation time point, which took place about 4 months
after enrollment and baseline evaluation. After 1 year of ECI intervention, the difference between DOCS
scores and expectation became sufficiently large in all cases to enable us to rule out random error in the
estimation of the expected scores as the explanation. That is, the difference between observed and
expected scores after 1 year can no longer be explained by possible estimation error in the calculation of
the expected scores due to maturation.
Children who were developmentally delayed progressed at a substantially faster rate than their
non-delayed counterparts. However, their posttest scores remained in the lower end of age expectations
although the gap closed substantially in the course of time.

Analysis B1: Constructed comparison group analysis for 302

children with data for 3 DOCS observation time points.

Figure 1: Observed and expected DOCS scores for 302 children with 3 time points

900
observed
850
expected
DOCS score

800
upper limit
750 of expected
700
650
600
950 1150 1350 1550 1750
Age (in days)

Observed Expected effect

Age (in days) Difference N t-value p-value percentile
DOCS DOCS size

Time 1 1130 654.8 655.6 -0.8 302 -0.142 1.000 -0.01 50

Time 2 1253 706.3 701.6 4.7 302 0.817 0.414 0.37 64

Time 3 1375 746.4 740.7 5.7 302 1.049 0.295 0.51 69

Interpretation: A total of 302 ECI children had data for 3 DOCS observation points. The mean age at Time
1 was about 3 years 1 month, that at Time 2 was about 3 years 5 months, and that at Time 3 was about 3
years 8 months. During this 7-month period, the mean DOCS composite score increased from 655 to 746.
The expected DOCS score at Time 3 based on maturation alone was 741. The difference was not statisti-
cally significant. Hence, no treatment effect beyond maturation expectation can be concluded at Time 2
nor Time 3.

127
Analysis B2: Constructed comparison group analysis for 106
children with data for 4 DOCS observation time points.

Figure 2: Observed and expected DOCS scores for 106 children with 4 time points

900
observed
850
expected
DOCS score

800
upper limit
750 of expected
700
650
600
950 1150 1350 1550 1750
Age (in days)

Observed Expected effect

Age (in days) Difference N t-value p-value percentile
DOCS DOCS size

Time 1 1005 615.2 619.7 -4.5 106 -0.411 1.000 -0.20 42

Time 2 1130 672.3 673.0 -0.7 106 -0.066 1.000 0.04 50

Time 3 1255 723.1 719.6 3.5 106 0.328 0.744 0.20 58

Time 4 1381 773.5 759.7 13.8 106 1.535 0.128 0.93 82

Interpretation: A total of 106 ECI children had data for 4 DOCS observation points. The mean age at Time
1 was about 2 years 9 months, that at Time 2 was about 3 years 1 month, that at Time 3 was about 3 years
5 months, and that at Time 4 was about 3 years 9 months. During this 12-month period, the mean DOCS
composite score increased from 615 to 774. The expected DOCS score at Time 4 based on maturation alone
was 760. None of the differences found at Time 1, 2, 3, or 4 was statistically significant. Hence, no
treatment effect beyond maturation expectation can be concluded at Time 2, 3, nor 4.

128
Analysis B3: Constructed comparison group analysis for 53 children with data for 5 DOCS observation time points.

Figure 3: Observed and expected DOCS scores for 53 children with 5 time points

900
observed
850
expected
DOCS score

800
upper limit
750 of expected
700
650
600
950 1150 1350 1550 1750
Age (in days)

Observed Expected effect

Age (in days) Difference N t-value p-value percentile
DOCS DOCS size

Time 1 984 606.4 615.4 -9 53 -0.5371 0.703 -0.04 48

Time 2 1106 677.1 668.7 8.4 53 -0.5462 0.294 0.04 52

Time 3 1228 734.6 715.5 19.1 53 1.1081 0.136 0.11 54

Time 4 1361 778.3 758.8 19.5 53 1.4477 0.077 0.13 55

Time 5 1485 819.3 791.8 27.5 53 1.9733 0.027 0.22 59

Interpretation: A total of 53 ECI children had data for 5 DOCS observation points. The mean age at Time 1
was about 2 years 8 months, that at Time 2 was about 3 years 0 month, that at Time 3 was about 3 years 4
months, that at Time 4 was about 3 years 9 months, and that at Time 5 was about 4 years 1 month. During
this 17-month period, the mean DOCS composite score increased from 606 to 819. The expected DOCS
score at Time 5 based on maturation alone was 792. None of the differences found at Time 1, 2, 3, or 4 was
statistically significant. However, the difference found at Time 5 was significant. The general trend over
the 17-month period was one of increasing departure from maturation expectation. The mean DOCS score
of the cohort of 53 children was at the 48 percentile (in the bottom half) of their peers at Time 1. Their posi-
tions relative to the theoretically untreated peers continuously progressed to the 52nd percentile at Time 2,
to the 54th percentile at Time 3, 55th percentile at Time 4 and finally to the 59th percentile at Time 5. There
was clearly an intervention effect beyond what can be accounted for by maturation.

129
Analysis B4: Constructed comparison group analysis for 22 children with data for 6 DOCS observation time points.

Figure 4: Observed and expected DOCS scores for 22 children with 6 time points

900
observed
850
expected
DOCS score

800
upper limit
750 of expected
700
650
600
950 1150 1350 1550 1750
Age (in days)

Observed Expected effect

Age (in days) Difference N t-value p-value percentile
DOCS DOCS size

Time 1 1079 700.8 684 16.8 22 0.7076 0.243 0.17 57

Time 2 1197 778.5 729.7 48.8 22 2.2729 0.017 0.53 70

Time 3 1303 808.6 767.7 40.9 22 1.6108 0.061 0.48 69

Time 4 1438 840.9 807 33.9 22 2.0197 0.028 0.49 69

Time 5 1570 870.3 836.7 33.6 22 1.6606 0.055 0.56 71

Time 6 1689 896.9 857.31 39.59 22 2.4615 0.011 0.79 78

Interpretation: A total of 22 ECI children had data for 6 DOCS observation points. The mean age at Time 1
was about 3 years 0 month, that at Time 2 was about 3 years 3 months, that at Time 3 was about 3 years 7
months, that at Time 4 was about 3 years 11 months, that at Time 5 was about 4 years 4 months, and that
at Time 6 was about 4 years 8 months. During this 20-month period, the mean DOCS composite score
increased from 701 to 897. The expected DOCS score at Time 6 based on maturation alone was 857. All the
differences between observed and expected scores from Time 2 forward were found to be statistically
significant. The general trend over the 20-month period was one of continuous and stable departure from
maturation expectation. The mean DOCS score of the cohort of 22 children was at the 57th percentile
(better than median) of their peers at Time 1. Their positions relative to the theoretically untreated peers
changed to the 70th percentile at Time 2, to the 69th percentile at Time 3 and Time 4, to the 71st percentile
at Time 5 and finally to the 78th percentile at Time 6. There was clearly an intervention effect beyond
what can be accounted for by maturation. For this cohort, the effect appeared at Time 2 and remained
throughout.

130
Analysis B5: Separate constructed comparison group analyses for developmentally delayed (N=34) and non-delayed (N=268)
children with data for 3 DOCS observation time points.

Figure 5: Constructed comparison groups contrasting developmentally delayed

(N=268) vs. nondelayed children (N=34) with 3 time points.
900
observed
nondelayed
850
expected
DOCS score

800 nondelayed
upper limit
750 of expected
nondelayed
700 observed
delayed
650
expected delayed
600
lower limit of
950 1150 1350 1550 1750 expected delayed
Age (in days)

Observed Expected effect

Age (in days) Difference N t-value p-value percentile
DOCS DOCS size
Non-delayed

Time 1 1110 658 644.5 13.5 268 0.976 0.33 0.85 80

Time 2 1233 706.1 691.7 14.4 268 0.92 0.359 1.03 85

Time 3 1354 744.7 731.9 12.8 268 0.862 0.39 1.04 85

Delayed

Time 1 1289 629.5 743.3 -113.8 34 -25.095 0 -4.76 1

Time 2 1413 708.1 779.8 -71.7 34 -11.651 0 -2.92 1

Time 3 1615 760.5 810 -49.5 34 -6.927 0 -2.88 1

Interpretation: Of the of 302 ECI children who had data for 3 DOCS observation points, 34 would have been
classified as developmentally delayed based on DOCS scores at Time 1. The delayed children were
chronologically older than their non-delayed counterparts. The mean age of the 34 delayed children at
Time 1 was about 3 years 6 months, that at Time 2 was about 3 years 10 months, and that at Time 3 was
about 4 years 5 months. In contrast, the mean age of the 268 non-delayed children at Time 1 was about 3
years 0 month, that at Time 2 was about 3 years 5 months, and that at Time 3 was about 3 years 9 months.
The growth pattern of the non-delayed children was quite similar to that described in Analysis C1; i.e., there
was no significant departure from expectation at any of the time points. In contrast, the delayed children
were significantly lower than age expectation at all three time points. However, their observed scores
moved substantially closer to age expectation throughout the three time points. Specifically, their scores
at Time 1 were 114 points below expectation. This was reduced to 72 points at Time 2 and to 50 points at
Time 3.

131
Analysis B6: Separate constructed comparison group analyses for developmentally delayed (N=16) and non-delayed (N=90)
children with data for 4 DOCS observation time points.

Figure 6: Constructed comparison groups contrasting developmentally delayed (N=90)

vs. nondelayed children (N=16) with 4 time points.
900
observed
nondelayed
850
expected
DOCS score

800 nondelayed
upper limit
750 of expected
nondelayed
700 observed
delayed
650
expected delayed
600
lower limit of
950 1150 1350 1550 1750 expected delayed
Age (in days)

Observed Expected effect

Age (in days) Difference N t-value p-value percentile
DOCS DOCS size
Non-delayed

Time 1 967 617.5 600.9 16.6 90 1.757 0.082 0.66 75

Time 2 1094 667.4 656.7 10.7 90 0.974 0.333 0.48 68

Time 3 1218 718 705.4 12.6 90 1.155 0.251 0.64 74

Time 4 1343 769.9 747.5 22.4 90 2.54 0.013 1.33 91

Delayed

Time 1 1213 602.3 725.3 -123 16 -6.4 0 -3.53 1

Time 2 1333 700.1 764.8 -64.7 16 -2.806 0.013 -2.09 2

Time 3 1461 751.9 799.5 -47.6 16 -1.582 0.133 -1.77 4

Time 4 1591 793.8 828.3 -34.5 16 -1.228 0.237 -1.61 5

Interpretation: Of the of 106 ECI children who had data for 4 DOCS observation points, 16 would have been
classified as developmentally delayed based on DOCS scores at Time 1. The delayed children were
chronologically older than their non-delayed counterparts. The mean age of the 16 delayed children at
Time 1 was about 3 years 4 months, that at Time 2 was about 3 years 8 months, that at Time 3 was about 4
years 0 month, and that at Time 4 was about 4 years 4 months. In contrast, the mean age of the 90 non-
delayed children at Time 1 was about 2 years 8 months, that at Time 2 was about 3 years 0 month, that at
Time 3 was about 3 years 4 months, and that at Time 4 was about 3 years 8 months. The growth pattern of

132
the non-delayed children, when analyzed separately, was found to be different from that of the overall
group depicted in Analysis C2. Specifically, there was a substantial gain at Time 4 among the non-delayed
children. The non-delayed children in this cohort were quite proficient from the start, with a Time 1 DOCS
score placing them at the 75th percentile (i.e., top _) of all ECI children. Their position remained there and
the gap between their observed and expected scores remained stable through Times2 and 3. However, at
Time 4, they moved to the 91st percentile with the difference between observed and expected wider from
13 points at Time 3 to 22 points at Time 4. The delayed children in the cohort were significantly lower than
age expectation at Times 1 and 2. But the gap became statistically nonsignificant at Times 3 and 4. The
gap between observed and expected score reduced from 123 points at Time 1, to 65 points at Time 2, to 48
points at Time 3 and finally to 35 points at Time 4. In terms of percentiles, the delayed children were initial-
ly at the bottom 1 percent of their peers, but moved to the 2nd percentile at Time 2, to the 4th percentile at
Time 3 and finally to the 5th percentile at Time 4.

133
 ANALYSIS GROUP C:
IDENTIFICATION OF ECI PROGRAMMATIC VARIABLES
WITH A DIRECT IMPACT ON CHILD OUTCOMES

When combined, the results from Analysis Groups A & B provide strong evidence that a child’s participa-
tion in high quality ECI programs of all types has a statistically and functionally significant (educationally
relevant) impact on overall developmental competency, particularly as measured through DOCS scores.
However since ECI has only been implemented for a brief 33-month period instead of its projected 5-year
timeline , the picture is incomplete in that we cannot discern precisely what aspects of the program might
have led to the impact. This question is partially answered in the results of Analysis Group A. However,
since Analysis Group A did not include program measures, the answer is incomplete. In order to include
program measures (i.e., program quality, program intensity and amount of consultation), two changes in the
design for Analysis Group A became necessary. First, only ECI children would be included since program
measure data were available only for ECI children. Second, it is necessary to extend from the polynomial
regressions in Group A to the use of hierarchical linear modeling (HLM) techniques. This is because
program measures are nested measures in that children from the same program share exactly the same
program measure value, although their differ among themselves in child outcome measures and in family
measures.
The extended analyses were accomplished through a two-step process: First, an ordinary least square
polynomial multiple regression similar to those in Analysis Group A was done for each of the outcome
measures. The difference was that only ECI data were used and program measure variables were
included. This step was used as a preliminary screening process to determine whether an HLM analysis
might be warranted. If none of the program measures were found to be statistically significant for a
particular child outcome measure, the analysis for that outcome measures stopped at that point and we
revert back to the corresponding results for that child measure in Analysis Group A as the best identifica-
tion of explanatory variables for program impact. On the other hand, if any of the program measures were
found to be statistically significant for a given child outcome measure, we proceeded to an HLM analysis
to derive the more precise explanatory model for the impact.
The initial OLS analyses failed to show any of the program measures to be significant predictors of
PKB-social, PKB-behavior, BSSI-spoken language at end of ECI, BSSI-reading at end of ECI, BSSI-compos-
ite at end of ECI, or any of the BSSI scores at the end of kindergarten. Hence, the results showed in
Analyses A6, A7, A8, A9, and A14 through A21 represent the best explanatory model of the direct impact of
different family and child variables on the child outcome measure. (Note: the analyses of BSSI scores at
the end of kindergarten were based on a sample of only 18 children. The lack of significance for these
particular analyses might be attributed to the lack of statistical power.)
For DOCS composite scores, DOCS language subscores, and DOCS social subscores, program intensity
was found in the initial OLS analyses to be negatively related to child measure outcomes. However, when
these were submitted to the more precise HLM analysis processes, these relationships were not
confirmed. Hence, the results reported for Analyses A1, A3, and A4 remain the best explanatory models
for these measures. The initial OLS analyses also found BSSI writing subscores at the end of ECI to be
negatively related to program intensity and consultation but positively related to program quality. Similarly,

134
BSSI math subscores at the end of ECI were found to be negatively related to program intensity and pro-
gram quality. BSSI daily living subscores at the end of ECI were found to be negatively related to program
quality. However, subsequent HLM analyses failed to confirm any of these three patterns. Therefore,
again, the results reported for Analyses A10, A11, and A13 remain the best explanatory models for
these measures.
Finally, DOCS cognitive subscores, DOCS motor subscores, and BSSI classroom behavior subscores at the
end of ECI were all found to be significantly related to one or more program measures. Subsequent HLM
analyses for these three measures also confirmed the impact of these program characteristics on these
measures. The results of the three analyses are summarized below and are labeled Analyses C2, C5, and
C12 (corresponding to A2, A5, and A12).

135
Analysis C2: DOCS posttest cognitive scores as child outcome measure - Sample size: 363 cases

INITIAL OLS RESULTS

Predictor Coef SE Coef T P

Constant 32.12 13.03 2.47 0.014

Docs_age 0.28809 0.02350 12.26 0.000

Docs_age2 -0.00007270 0.00000703 -10.34 0.000

PSUCOGT_pre 0.36014 0.03925 9.18 0.000

Docs_trt 0.09970 0.01270 7.85 0.000

PCDIRawT -0.6172 0.2899 -2.13 0.034

PINTOTT1 -0.5998 0.1747 -3.43 0.001

S = 27.94 R-Sq = 93.7% R-Sq(adj) = 93.6%

SUBSEQUENT HLM RESULTS

STANDARD APPROX.

Fixed Effect Coefficient Error T-ratio d.f. P-value

For INTRCPT1, B0
382.719529 3.204276 119.440 10 0.000
INTRCPT2, G00

INTENSE, G01 -0.692647 0.247961 -2.793 10 0.019

For PCDIRAWT slope, B1

-0.861746 0.377658 -2.282 11 0.043
INTRCPT2, G10
For DCOS_TRT slope, B2
0.143618 0.027642 5.196 11 0.000
INTRCPT2, G20
For DOCS_AGE slope, B3
0.058312 0.014241 4.095 11 0.002
INTRCPT2, G30
For PSUCOG_F slope, B4
0.545754 0.063680 8.570 11 0.000
INTRCPT2, G40

Interpretation: An initial OLS analysis shows program intensity to be a potential significant (negative)
predictor of DOCS posttest cognitive score. A more accurate HLM analysis indeed confirmed the
existence of effects due to program intensity. Specifically, after accounting for the effects of age and prior
differences in DOCS cognitive scores, program intensity, parental stress – dysfunctional interaction were
found to be negatively related to, and length of treatment was found to be positively related to DOCS
cognitive posttest scores. Each 1.5 point drop in program intensity scores is associated a 1 point gain in
DOCS cognitive posttest scores. Each 1.1 point drop in parental stress – dysfunctional interaction scores
is associated with a 1 point gain in DOCS cognitive posttest scores. Finally, each additional 7 days of
treatment is associated with a 1 point gain in DOCS cognitive posttest scores.

136
Analysis C5: DOCS posttest motivation scores as child outcome measure - Sample size: 363 cases

OLS RESULTS

Predictor Coef SE Coef T P

Constant 43.962 9.424 4.67 0.000

Docs_age 0.29571 0.01915 15.44 0.000

Docs_age2 -0.00007955 0.00000572 -13.90 0.000

PSUCOGT_pre 0.21216 0.03554 5.97 0.000

Docs_trt 0.049063 0.008654 5.67 0.000

PCDIRawT -0.4568 0.2053 -2.23 0.027

PINTOTT1 -0.5992 0.1227 -4.88 0.000

S = 19.81 R-Sq = 94.0% R-Sq(adj) = 93.9%

SUBSEQUENT HLM RESULTS

STANDARD APPROX.

Fixed Effect Coefficient Error T-ratio d.f. P-value

For INTRCPT1, B0
323.879222 2.135376 151.673 10 0.000
INTRCPT2, G00

INTENSE, G01 -0.700477 0.244573 -2.864 10 0.017

For PCDIRAWT slope, B1

-0.498898 0.239024 -2.087 11 0.061
INTRCPT2, G10
For DCOS_TRT slope, B2
0.104552 0.023560 4.438 11 0.001
INTRCPT2, G20
For DOCS_AGE slope, B3
0.039043 0.014241 4.095 11 0.000
INTRCPT2, G30

INTENSE, G31 0.058312 0.006904 5.655 10 0.048

For PSUMOT_F slope, B4

0.468363 0.053556 8.745 10 0.000
INTRCPT2, G40

INTENSE, G41 0.012518 0.004306 2.907 10 0.016

Interpretation: An initial OLS analysis shows program intensity to be a potential significant (negative)
predictor of DOCS posttest motivation scores. A more accurate HLM analysis indeed confirmed the
existence of effects due to program intensity. Specifically, after accounting for the effects of age and prior
differences in DOCS motivation scores, program intensity, parental stress – dysfunctional interaction were
found to be negatively related to, and length of treatment was found to be positively related to DOCS
motivation posttest scores. Further, effects of interaction between program intensity and age and between
program intensity and DOCS motivation pretest scores were also found. Each 1.4 point drop in program
intensity scores is associated a 1 point gain in DOCS motivation posttest scores. Each 2 point drop in

137
parental stress – dysfunctional interaction scores is associated with a 1 point gain in DOCS motivation
posttest scores. Each additional 10 days of treatment is associated with a 1 point gain in DOCS motivation
posttest scores. Further, DOCS motivation posttest scores were related to age. However, the magnitude
of that relationship is minimized for older children in highly intensive program. The relationship is
maximized when the children are very young and are in program with low intensity. Pretest motivation
scores were also found to be related to motivation posttest scores. Again, there is an interaction with
program intensity and the magnitude of that relationship depends on program intensity. For children with
low pretest scores and low program intensity, the relationship between pretest scores and posttest scores
was minimized. The relationship is maximized for children with high pretest scores and high program
intensity.

Analysis C12: BSSI posttest (at end of ECI) classroom behavior scores as child outcome measure - Sample size: 79 cases

OLS RESULTS

Predictor Coef SE Coef T P

Constant 30.609 3.761 8.14 0.000

PKB_delay -20.077 4.381 -4.58 0.000

CONTOTT1 0.35619 0.09696 3.67 0.000

S = 10.99 R-Sq = 28.9% R-Sq(adj) = 27.0%

SUBSEQUENT HLM RESULTS

STANDARD APPROX.

Fixed Effect Coefficient Error T-ratio d.f. P-value

For INTRCPT1, B0
45.108386 1.461605 30.862 6 0.000
INTRCPT2, G00
For PKBDELAY slope, B1
-1.927275 4.254158 -0.453 5 0.669
INTRCPT2, G10

P_CONSUL, G11 -3.444751 0.509932 -6.755 5 0.000

Interpretation: An initial OLS analysis shows consultation to be a potential significant (negative) predictor
of BSSI-classroom behavior posttest scores. A more accurate HLM analysis indeed confirmed the
existence of effects due to consultation. Specifically, an interaction effect between whether a child was
classified as having a behavioral problem based on PKB-behavior pretest scores and the amount of con-
sultation to the program was found. A child without behavior problems can be expected to have a score
of 45 on the BSSI-classroom behavior posttest regardless of whether the child’s program has received any
consultation at all. A child with behavioral problems but whose program has not received any consulta-
tion at all is expected to score around 43 on the BSSI-classroom behavior posttest. A child with behavioral
problems and whose program has received consultation can be expected to score lower than 43 on the
BSSI-classroom behavior posttest. For every point increase in consultation score for the program, the
BSSI-classroom behavior posttest score of the child can be expected to drop another 3.4 point below 43.

138
 A P P E N D I X C
SAMPLES OF THE DEPENDENT MEASURES IN THE
SPECS EVALUATION RESEARCH BATTERY

139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
 A P P E N D I X D

SAMPLE QUALITY MENTORING PLAN FOR

ECI COLLABORATIVE PROGRAM CONSULTATION

COLLABORATIVE CONSULTATION AND QUALITY

MENTORING-MONITORING: A CASE STUDY

In its position statement on early childhood professional development (NAEYC 1994) NAEYC has identified
the professional development of teachers, providers and directors as a key strand to promoting high-
quality early childhood programs. Facilitating the professional development of individuals working with
young children birth through age 8 can be accomplished in many ways. Initial experimentation in
mentoring programs for early care and education teachers and providers began in the late 1980s, in
response to the identified staffing crisis and its impact on the quality of services (Whitebook, Howes, &
Phillips, 1990).
Mentoring and consultation for front-line and supervisory staff in the ECI project is a vital part of technical
assistance. It is a belief of ECI mentors that early childhood staff need to be supported and educated.
Consistent with the National Center for the Early Childhood Work Force definition of mentoring (Whitebook
and Bellm, 1996), mentors in the ECI project are former teachers and center directors who have worked in
the field for a significant time and who have received education and training in child development, early
childhood education and the teaching of other adults. They also have a significant grasp of best practice
standards and resources.
The ECI style of mentoring is one of a balanced partnership, reflecting collaborative consultation and
problem-solving. Semi-annual environmental and staff appraisals are conducted, using the ECERS, ITERS,
FDCRS, and Arnett Scale for Adult Sensitivity and Responsiveness. In content, the ECI technical assistance
team works closely with directors, quality monitors, and center- and home-based staff to determine
classroom goals, objectives, and activities. Through the development of quality enhancement and
technical assistance tools and their monitoring of quality enhancement progress, the ECI technical
assistance team further works to help staff set timelines for tasks and to assist in developing tools for
monitoring and reporting. The extent to which the ECI technical assistance team mentors is seen in:
• the development and monitoring of quality enhancement plans following the environmental and
staff interactions assessments
• monitoring of reporting
• in the visiting of classrooms
• and in committee work.
A unique aspect of the ECI mentoring approach is the use of child developmental progress information to
guide the teaching and care of teachers and providers. ECI mentors demonstrate how teachers can use
child performance data on the DOCS to plan group and individual learning activities in natural everyday
settings and routines and to communicate with parents so that they both strive for a seamless focus on
early school success “building blocks” in both the preschool and home.

155
The following case study will provide an example of how the technical assistance typically provides
mentoring:

QUALITY ENHANCEMENT PLAN

STRENGTHS TO BUILD ON:

Staff provides age appropriate
equipment and materials for
active physical play.

CLASSROOM: Infant FIRST DRAFT DATE: 2/15/02

TEAM: ECI Project Director, ECI Quality Assessment Coordinator, Center Quality Monitor
AREA OF FOCUS: Program Structure and Schedule Outdoor Free Play
FINAL DRAFT DATE:______TEAM:____________________________
OBJECTIVE: Plan and implement daily outdoor play period
DATE ACHIEVED:________

ACTIVITIES TIMELINE/FREQUENCY RESPONSIBLE STAFF DATE COMPLETED

1. Review daily schedule and

Classroom
identify appropriate outdoor 2/18/02
Teaching Team
activity period.

2. Meet weekly to plan outdoor 3/8/02 Classroom

activity for the following week. On-going Teaching Team

3. Submit weekly lesson plan Classroom

2/18/02
including outddor activity to the Teaching Team/
On-going
supervisor. Supervisor

4. Review curriculum resources

2/18/02 Classroom
for gross motor activities for
On-going Teaching Team
outdoor learning experiences.

September 20,2001/Quality Enhancement PLan/VO/Revised Big 9 9/13/01

156
TECHNICAL ASSISTANCE PLAN

CLASSROOM: Infant
AREA OF FOCUS: Program Structure and Schedule of Outdoor Free Play
PLAN DEVELOPED BY: ECI Project Director, ECI Quality Assessment Coordinator, Center Quality Monitor
DATE DEVELOPED: 2/15/02
OBJECTIVE: Plan and implement daily outdoor play period
DATE ACHIEVED:________

ACTIVITIES TIMELINE/FREQUENCY RESPONSIBLE STAFF DATE COMPLETED

ECI Project Director

1. Assist staff in establishing an ECI Quality Assessment
2/18/02
outdoor activity period. Coordinator, Center
Quality Monitor

ECI Project Director

2. Meet weekly to plan outdoor 3/8/02 ECI Quality Assessment
activity for the following week. On-going Coordinator, Center
Quality Monitor

ECI Project Director

3. Submit weekly lesson plan
2/18/02 ECI Quality Assessment
including outddor activity to the
On-going Coordinator, Center
supervisor.
Quality Monitor

ECI Project Director

4. Review curriculum resources
2/18/02 ECI Quality Assessment
for gross motor activities for
On-going Coordinator, Center
outdoor learning experiences.
Quality Monitor

* Activities may include: observation/feedback/demonstration-modeling/consultation/training

September 20, 2001/VO Revised Big 9 9/13/01

157
QUALITY ENHANCEMENT PROGRESS

CLASSROOM: Infant
WEEK OF: 2/18/02 TO:
AREA OF FOCUS: Program Structure and Schedule of Outdoor Free Play

OBJECTIVE: Plan and implement a

BRIEF DESCRIPTION TIMELINE IMPROVEMENT RATING
daily outdoor play period ACTIVITIES

Develop and submit a lesson

In process 1 2 3
plan incorporating music,
1. Use “bye-bye” buggy walks
movement and gross motor
Meet guidelines Why: Weather permitted
activities into the daily schedule

* Improvement Rating Key: 1 = We’re Starting / 2 = We’re Improving / 3 = We’re There

September 20, 2001/VO Revised Big 9 9/13/01

158
 A P P E N D I X E :

LETTERS OF CORRESPONDENCE AMONG ECIM, ECI

NEIGHBORHOOD LEADERSHIP COUNCILS AND SPECS

159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
 A P P E N D I X F

THE ORGANIZATIONAL CAPACITIES

OF THE SPECS EVALUATION TEAM

Early Childhood Partnerships (ECP) is a unique collaborative between Children’s Hospital of Pittsburgh, the
UCLID (University, Community, Leaders, and Individuals with Disabilities) Center at the University of
Pittsburgh and various local and statewide agencies devoted to addressing the multiple needs of all young
children and their families. Through three specific projects, Early Childhood Partnerships conducts
collaborative interagency consultation, training and technical assistance, program evaluation, and
research development to the early childhood and early intervention community of Allegheny and surround-
ing communities. These three projects serve young children, 0-8 years of age, in Allegheny County and
surrounding counties in Western Pennsylvania. These three projects, SPECS (Scaling Progress in Early
Childhood Settings) evaluation program, Healthy CHILD, and Head Start Inclusion Training Satellite (DSQIC)
engage in specific activities to address the needs of young children, these include:
• Site based in service training for program staff and administrators about simple practical
methods to link authentic assessment, curriculum planning and intervention, and program
efficacy evaluation in community programs for children with and without special needs.
• Applied program evaluation and outcomes research on child, family, program quality, early
literacy, community, health/mental health, and professional standards and competency.
• Program focused consultation on practical everyday issues pertaining to physical health, mental
health, and developmental learning needs of young children at developmental risk.
• Technical assistance and training to support inclusion of young children with special needs into
typical early childhood settings.
• Parent education and support for preventive developmental healthcare concerns
• Systems consultation and adminstrators about classroom and program organizational design
and transagency team work.
Through collaborative ventures with schools, private foundations, and state and federal government
departments Early Childhood Partnerships is able to pool community resources to develop creative
solutions to impact the efficacy of interagency child/ family services, professional development of
providers, and changes in public policy as well as advance the research base in the fields of early
childhood and early intervention.

The UCLID Center at the University of Pittsburgh is a regional center, established in 1995 through Maternal
and Child Health Bureau (MCHB) funding, whose primary purpose is to develop exemplary leaders in areas
of neurodevelopmental disabilities and related disorders. The Center was created to bring together
University faculty and students, Community service providers and health professionals, Leaders in
government, business, and philanthropy, and Individuals with Disabilities and their families, all united
toward a common vision. In all of its programs and activities, the UCLID Center stresses humanism,
egalitarianism, and inclusion. It values the participation of diverse constituencies and individuals who

174
differ in terms of their backgrounds, professions, circumstances, and cultures. The Center functions as an
interdisciplinary team in which each participant is included, respected, and appreciated as an equal.
The Office of the Provost at the University of Pittsburgh provides administrative oversight to the UCLID
Center. The Center has a highly qualified core faculty of health care professionals, representing 11 core
health care disciplines. The Center also has a broad-based Community Advisory Board (CAB) of 23
members, who represent many different agencies and interests in the community regarding disabilities
including advocacy groups, parents, and individuals with disabilities. The UCLID Center CAB members
represent government, education, social service, advocacy groups, parent groups, and other
organizations. One-third of the members are family members of children with disabilities or individuals with
disabilities. The CAB demonstrates the centrality of the UCLID Center’s philosophy of inclusion; members
of the community are invited to advise the Center’s projects on activities and approaches.
Each year since its establishment, the UCLID Center has provided interdisciplinary leadership training to 8
to 10 doctoral level students from various academic disciplines in clinical care, research, and community
consultation and technical assistance. These trainees participate in the various research and program
evaluation projects under the UCLID Center. The mission of the Center is to:
• provide clinical, research, policy and leadership training for health care and educational profes
sionals caring for individuals with neurodevelopmental disabilities or related disorders and their
families.
• promote innovative and community-based systems of health care, education, and social services
that are preventive, comprehensive, family-centered, and culturally-sensitive
• encourage university-community collaboration in achieving these training and service objectives.

Staff Qualifications and Commitment

STEPHEN J. BAGNATO, Ed.D., NCSP is a Developmental School Psychologist and Professor of Pediatrics
and Psychology at the University of Pittsburgh School of Medicine. He is Director of the SPECS Evaluation
Research Team. Dr. Bagnato is Faculty Director of the Developmental Psychology Interdisciplinary Training
Core for The UCLID Center at the University of Pittsburgh, a Maternal and Child Health Bureau leadership
education institute in neurodevelopmental disabilities. Within The UCLID Center and associated with
Children’s Hospital, Dr. Bagnato serves as Director of Early Childhood Partnerships (ECP), an innovative
University-Hospital-Community collaborative venture which is dedicated to community- and program-
centered consultation, professional development, technical assistance and applied program evaluation
research. EarlyCHILD advocates high quality service delivery, use of “best practices”, public policy
changes, and advancements in the research base regarding young children and families at developmental
risk and with neurodevelopmental disabilities and neurobehavioral disorders.
Dr. Bagnato specializes in authentic curriculum-based assessment strategies for early childhood
education and intervention. Dr. Bagnato has published over 90 applied research studies in early
intervention, early childhood education, school psychology, developmental disabilities, and developmental
neuropsychology. His latest publications include the third edition of the widely used resource text: LINKing
Assessment and Early Intervention: An Authentic Curriculum-based Approach (1997; Paul Brookes), and
the Temperament and Atypical Behavior Scale (TABS): Early Childhood Indicators of Developmental
Dysfunction (1999; Paul Brookes).

175
HOI SUEN, Ph.D., is a Professor of Educational Psychology at Penn State University who has extensive
expertise and consultant experience in program evaluation research, longitudinal research design
methodologies with large databases, psychometrics, and statistical techniques appropriate for progress
monitoring incorporating multidimensional and multisource data, especially Hierarchical Linear Modeling,
Multiple Regression techniques, and generalizability theory and analyses. Dr. Suen will work with the
research team to produce yearly statistical analyses and on-going data management support.

CAROL A. WHITACRE, is a Project Coordinator at the Early Childhood Partnerships at Children’s Hospital of
Pittsburgh. She is working with Beginning with Books to provide an evaluation of Project BEACON Plus,
an outreach caregiver training program with the focus on early literacy (children birth to 6 years of age).
She has extensive knowledge of child development and early literacy “best practices”. Ms. Whitacre
received her degree in Elementary Education and holds a Level I Teaching Certificate in Pennsylvania. She
has worked more than seven years in the field of educational research, evaluating the quality and needs
of early intervention and early childhood settings. She has worked with numerous early intervention
programs across Pennsylvania, assisting with procedures to fulfill state compliance and monitoring
requirements. Her experience also includes: coordination of various project activities, providing training
and consultation to providers, supervising graduate students, developing training manuals and materials,
conducting family and staff interviews, creating and maintaining database files.

ELEANA SHAIR, is the information systems coordinator for the Early Childhood Partnerships (Early CHILD),
a Hospital-University-Community collaborative based at Children’s Hospital of Pittsburgh and The UCLID
Center at the University of Pittsburgh which provides consultation, training, technical assistance, and
research services to community program partners. For 12 years, Ms. Shair has worked on the devel-
opment of computerized information system that captures programmatically relevant information that is
useful for research and that supports program planning and evaluation. Currently, Ms. Shair is
responsible for database planning, information management and analysis for various program evaluation
projects under Early CHILD: The Early Childhood Initiative (ECI) of Allegheny County, BEACON Plus early
literacy project, the Allegheny County early intervention program evaluation project. Ms. Shair’s disserta-
tion study involves management information systems in early intervention. Ms. Shair will devote 25%
in-kind (refer to budget justification) to coordinate overall data management procedures of the project.

JENNETTE COOK-KILROY is currently Coordinator of the SPECS evaluation effort of the Early Childhood
Initiative (ECI) and the Director of the Program Strand for the evaluation. Ms. Cook-Kilroy specializes in the
assessment of young children, specifically young children with social and emotional issues. Ms. Cook-
Kilroy has worked with over 30 programs to develop proactive behavioral interventions for preschool
students with social and emotional needs. As director of project Early CHILD (1995-1997), funded by the
Jewish Healthcare Foundation, Ms. Cook-Kilroy has provided local preschool and early intervention
programs in Pittsburgh and surrounding communities with staff training and technical assistance on
addressing the needs of young children with health care and mental health needs in early care settings.
At the University of Iowa, Ms. Cook-Kilroy coordinated a National Institute on Disability and Rehabilitation
Research (NIDRR) field-initiated grant which evaluated the use of functional assessment in preschool and
home environments. Ms. Cook-Kilroy was the primary author on a monograph summary titled Application

176
of functional analysis procedures to familiar settings for young children who engage in severe aberrant
behavior. Ms. Cook-Kilroy was the primary author on 3 papers and co-authored 6 papers presented at
national conventions. She has presented to over 20 community agencies serving infants and toddlers on
topics including functional assessment, behavioral intervention, classroom management, family-centered
approaches, and environmental adaptations for the special learner.

JANELL SMITH-JONES, is the Co-Director of the Child Research Strand of the SPECS Evaluation Effort of
the Early Childhood Initiative (ECI). Dr. Smith-Jones obtained her doctorate in1992 from the University of
Pittsburgh with specialty in developmental psychology and subspecialty in educational research. Dr.
Smith-Jones has participated in numerous research efforts spanning a variety of topics, including
neurodevelopmental outcomes of infants treated with ECMO, early patterns of motor development in very
low birthweight newborns, socioeconomic and psychoemotional privation and implications for school
failure in black children, decreasing morbidity through temporal patterning of care and contingent
responsivity in the neonatal intensive care nursery, and values-based parenting. Dr. Smith-Jones’
research and professional interest is in early childhood development, parent-child relationships and
developmental impacts, and early intervention.

177
 APPENDIX G

PROFESSIONAL PROFILES FOR THE UCLID FACULTY

ON THE SPECS EVALUATION RESEARCH TEAM

DIRECTOR, SPECS EVALUATION RESEARCH TEAM

STEPHEN J. BAGNATO, Ed.D., NCSP is a Developmental School Psychologist and Professor of Pediatrics
and Psychology at the University of Pittsburgh School of Medicine. He is Director of both the Early
Childhood Partnerships program at Children’s Hospital of Pittsburgh and Faculty Director of the
Developmental Psychology Interdisciplinary Training Core for The UCLID Center at the University of
Pittsburgh. Dr. Bagnato specializes in authentic early childhood assessment and program evaluation
research strategies for infants, toddlers, preschoolers, and families at developmental risk or with
neurodevelopmental disabilities and neurobehavioral disorders. Recipient of the Braintree Hospital
national brain injury research award in 1986, Dr. Bagnato has published over 100 applied research studies
in early childhood intervention, school psychology, neurodevelopmental disabilities, and developmental
neuropsychology.
Dr. Bagnato has been Principal Investigator for a school-linked research and model program development
grant from the US Department of Education, Office of Special Education and Rehabilitative Services
(OSERS), one of only four awarded nationally. HealthyCHILD: Collaborative Health Interventions for
Learners with Disabilities is a field-validated service delivery model involving a transagency, mobile
developmental healthcare team to meet the complex needs of preschoolers with acute and chronic
medical conditions and challenging behaviors in early childhood classrooms (e.g., Head Start, early
intervention, ECE). Also, Dr. Bagnato is the director of psychology interdisciplinary training for a Maternal
and Child Health Bureau Leadership Education Institute in Neurodevelopmental Disabilities at the
University of Pittsburgh and Children’s Hospital entitled, The UCLID Center: University, Community, Leaders
and Individuals with Disabilities.

Dr. Bagnato is Director of Early Childhood Partnerships (EarlyCHILD), a community-based consultation,

training, technical assistance, and research collaborative of Children’s Hospital of Pittsburgh and The
UCLID Center at the University of Pittsburgh. EarlyCHILD consists of three major subdivisions: (1) The
SPECS Evaluation Team (Scaling Progress in Early Childhood Setting): A longitudinal evaluation of The
Heinz Pennsylvania Early Childhood Initiative (ECI), the $1.5 million program evaluation/research
component of a comprehensive effort to upgrade early care and education programs (including Head Start)
to promote kindergarten success for 3,000 children and families in Pittsburgh and other PA sites. SPECS
also conducts assessment and program evaluation training for the Georgia State Department of Health and
Human Services, (2) The HealthyCHILD school-linked developmental healthcare partnership; and (3) The
Head Start Inclusion Training Satellite-DSQIC-Region III, emphasizing strategies for inclusion of young
children with special needs into Early Head Start and Head Start classrooms across Pennsylvania. Early
Childhood Partnerships is funded by government, foundation, and interagency grants and contracts.
Dr. Bagnato is a Fellow of the American Psychological Association (APA) in Division 16 and past or current
journal editorial board member for Journal of School Psychology, School Psychology Review, School

178
Psychology Quarterly, Journal of Early Intervention, Topics in Early Childhood Special Education, Infants
and Young Children, and Child Assessment News. Dr. Bagnato received the 1995-1996 Best Research
Article Award from Division 16 of APA for his national study on the social and treatment invalidity of
intelligence testing in early childhood and early intervention. He is co-author of the professional “best
practice” policy statements and standards on early childhood assessment, evaluation, and early interven-
tion for The National Association of School Psychologists, and the Division for Early Childhood of the
Council for Exceptional Children.
Dr. Bagnato and his team are in demand as a national consultant/trainer group for state government
departments, schools, and agencies on early intervention, early childhood “best practices”, challenging
and atypical behaviors, early childhood assessment, and authentic program outcomes evaluation
research. His published books and instruments include: The third edition of the widely used resource text,
LINKing Assessment and Early Intervention: An Authentic Curriculum-based Approach (1997; Paul
Brookes); Assessment for Early Intervention: Best Practices for Professionals (1991; Guilford); System to
Plan Early Childhood Services (SPECS) (1990; American Guidance Service); The Young Exceptional Child:
Early Development and Education (1987; Macmillan); and the Temperament and Atypical Behavior Scale
(TABS): Early Childhood Indicators of Developmental Dysfunction (1999; Paul Brookes).

179
 STEPHEN J. BAGNATO, ED.D., NCSP

Professor of Pediatrics & Psychology

Faculty Director, Developmental Psychology Interdisciplinary Training Core
Director, Early Childhood Partnerships
Children’s Hospital of Pittsburgh
The UCLID Center at the University of Pittsburgh

Professional Profile:
• Ed.D. from Pennsylvania State University in 1977
• Specialization in developmental school psychology and early childhood special education
• Research focus: authentic, developmentally-appropriate, curriculum-based assessment and program
evaluation outcome methods in early childhood settings for children at developmental risk or with
neurodevelopmental disabilities and neurobehavioral disorders
• Distinguished Fellow of the American Psychological Association, Division 16, 1989
• Braintree Hospital National Brain Injury Research Award, 1986
• 1996 Recipient, APA Division 16, “Best Research Article Award”, School Psychology Quarterly
• 2001 Chancellor’s Distinguished Public Service Award, The University of Pittsburgh
• Created Early Childhood Partnerships (1996), an innovative University-Hospital-Community collaborative
devoted to community-based interagency consultation, mentoring, technical assistance, service, and
applied research for children at developmental risk and serving as a catalyst to unify the early care
and education system
• PA Director for Head Start Inclusion Training Satellite-DSQIC (Disabilities Services Quality Improvement
Center)
• Published over 100 applied research studies, books, assessment/intervention products, chapters, and
monographs in early intervention, early childhood education, school psychology, developmental dis
abilities, and developmental neuropsychology

Selected Publications
Books & Chapters
Bagnato, SJ, Neisworth, JT, Munson, SM (1997). LINKing assessment and early intervention: An authen-
tic curriculum-based approach (3rd Edition). Baltimore, MD: Brookes
Bagnato, SJ, Neisworth, JT (1991). Assessment for early intervention: Best practices for professionals.
NY: Guilford.
Neisworth, JT, Bagnato, SJ (2001), Recommended practices in assessment (17-28). In S. Sandall, ME,
McLean, BJ Smith (Eds), DEC recommended practices in early intervention/early childhood special edu-
cation, Longmont, CO: Sopris West (Invited research chairs for DEC “best practices” in early childhood
assessment- 1995-2002)

180
Assessment Systems
Bagnato, SJ, Neisworth, JT, Salvia, J, Hunt, FS (1999). Temperament and Atypical Behavior Scales
(TABS): Early childhood indicators of developmental dysfunction. Baltimore, MD: Brookes.
Bagnato, SJ, Neisworth, JT (1990). System to Plan Early Childhood Services (SPECS). Circle Pines, MN:
American Guidance Service, Inc.

Refereed Articles
Bagnato,SJ, Mayes, SD (1986). Patterns of developmental and behavioral progress for young brain-
injured children during interdisciplinary intervention. Developmental Neuropsychology, 2(3), 213-244.
Bagnato, SJ, Kontos, S, Neisworth, JT (1987), Integrated daycare as special education: Profiles of pro-
grams and children, Topics in Early Childhood Special Education, 7(1), 28-47.
Bagnato, SJ, Neisworth, JT, Paget, KD, Kovaleski, J (1987). The developmental school psychologist:
Professional profile of an emerging early childhood specialist, Topics in Early Childhood Special
Education, 7(3), 75-89.
Bagnato, SJ, Neisworth, JT (1995). A national study of the social and treatment “invalidity” of intelli-
gence testing for early intervention, School Psychology Quarterly, 9(2), 81-102.
Bagnato, SJ, Neisworth, JT (1999). Collaboration and teamwork in assessment for early intervention,
Child and Adolescent Psychology Clinics of North America, 8(2), 1-17.
Bagnato, SJ, Neisworth, JT (2000), Normative detection of early regulatory disorders and Autism:
Empirical confirmation of DC:0-3, Infants and Young Children, 12(2), 98-109.
Bagnato, SJ, Suen, H, Brickley, D, Jones, J (in press). Child developmental impact of Pittsburgh’s Early
Childhood Initiative (ECI) in high-risk communities: First-phase authentic evaluation research. Early
Childhood Research Quarterly.

181
 GEORGE E. MCCLOMB, PHD

Associate Professor, School of social work

Associate Professor, Graduate School of Public and International Affairs
Director of Community Organization/Social Administration, School of Social Work
UCLID Core Faculty, Social Work
UCLID Project Co-Director

Professional Profile
BA, Sociology/Anthropology, Colgate University, Hamilton, NY, 1962
M.S.W. Community Organization, University of Pittsburgh, Pittsburgh, PA, 1964
MA, Political Science, University of Pittsburgh, Pittsburgh, PA, 1974
PhD, Political Science, University of Pittsburgh, Pittsburgh, PA, 1984

Interests
My interests are in politics, policy, and nonprofit organizations, particularly in the field of developmental
disabilities. Current research interests include studying the development and operation of community
coalitions and organizations intended to improve the lives of individuals with disabilities and children at risk
for disabilities. In this regard I served as the coordinator of the community strand of the SPECS evaluation
team for Pittsburgh’s Early Childhood Initiative. I was honored to receive the Chancellor’s Distinguished
Public Service Award for the year 2000.

Publications
Bagnato, S., Suen, Hoi, and McClomb, G.” Authentic Assessment and Program Evaluation in Early
Childhood Research: Profiles of Progress for the Early Childhood Initiative (ECI) Presented to the Western
Pennsylvania Evaluation Network, Pittsburgh, PA, in September 2000.
Shin, J. and McClomb, G. (1998), Top Executive Leadership and Organizational Innovation: An Empirical
Investigation of Nonprofit Human Service Organizations (HSOs). Administration in Social Work, 22(3), 1-21
McClomb, G.E (1987). Strategic Perspectives: A Study of Challenge and Opportunity. Urban League of
Pittsburgh, Pittsburgh, PA

Courses & Training

Directed the development of the newly implemented COSA (Community Organization / Social
Administration) Concentration in the School of Social Work.. Developed and taught :
• SWCOSA 2083 - Introduction to Community Practice as the new introductory theory
course for COSA students. This course was implemented in the Fall 2000-2001 Semester.
• SWWEL 2087 - Organizations and Public Policy as the second level, required policy
course for COSA students. The course was implemented in the Spring 2000-2001
Semester.

182
 • SWADM 2064 - Seminar in Developmental Disabilities as an advanced, interdisciplinary
course for students interested in management and policy issues in the disabilities field. A
first course of its type on campus, the seminar is presented by an interdisciplinary faculty
of scholars, agency professionals and parents, all at the cutting edge of matters in the field
of neurodevelopmental and related disabilities. The course was implemented in the 1994-
1995 Fall Semester.
• SWADM 2084 – Organizational Behavior and Organizational Analysis as the introductory
theory course for the prior Social Administration Concentration.

Collaborations, Consultations, and Service

Board of Directors
ARC (Association for Retarded Citizens of
Allegheny County) 2002 - Present
Board of Directors
The Hill House Association 2002 - Present
Board of Directors
School Readiness Group 2001 - Present
Board of Directors
The Emmaus Community of Pittsburgh 1996 - Present

183
 HEIDI M. FELDMAN MD, PHD

Ronald L. and Patricia M. Violi Professor of Pediatrics and Child Development,

School of Medicine, University of Pittsburgh
Secondary Appointment, Communication Disorders, School of Health
and Rehabilitation Sciences, University of Pittsburgh
UCLID Core Faculty, Pediatrics
UCLID Project Director

Professional Profile
BA. Summa Cum Laude, Psychology, University of Pennsylvania, Philadelphia, PA, 1970
PhD, Developmental Psychology, University of Pennsylvania, Philadelphia, PA, 1975
MD, University of California, San Diego, La Jolla, CA, 1979
Fellowship Training, Ambulatory Pediatrics, Child and Family Development,
The Children’s Hospital of Boston, 1983-1984,

Interests
I have had long-standing research interests in language development. As a developmental psychology
graduate student, I studied the creation of a manual language by deaf children of hearing parents who had
no exposure to either verbal or sign language. Since obtaining my MD, I have studied language abilities in
children developing typically and children with medical conditions, including neurological injury, autism,
and otitis media. More recently, I have become interested in how services are delivered to young children
with disabilities and interdisciplinary education for professionals working with them. I am appreciative of
several honors and recognitions: Membership in the Society for Pediatric Research and American
Pediatrics Society; the University of Pittsburgh Chancellor’s Distinguished Teaching Award; Down
Syndrome Center of Western PA Special Recognition Award; and the University of Pittsburgh School of
Medicine, Excellence in Education Award 2000.

Publications
Feldman HM, MacWhinney B, Sacco, K. Sentence Processing in Children with Early Unilateral Brain Injury.
Brain and Language, in press.
Paradise JL, Feldman HM, Campbell TF, Dollaghan CA, Colborn DK, Bernard BS, Rockette HE, Janosky JE,
Pitcairn DL, Sabo DL, Kurs-Lasky M, Smith CG. Early versus delayed tympanostomy-tube placement for
persistent otitis media: Developmental outcomes at age 3 years. New England Journal of Medicine, 2001,
344, 1179-1187.
Booth JR, MacWhinney B, Thulborn KR, Sacco K, Voyvodic J, Feldman HM. Developmental and
lesion effects in brain activation during sentence comprehension and mental rotation. Developmental
Neuropsychology. 2000, 18, 139-169.

184
Feldman HM, Dollaghan C, Campbell T, Kurs-Lasky M, Janosky JE, Paradise JL. Measurement properties
of the MacArthur Communicative Development Inventory at ages 1 and 2 years. Child Development
March/April 2000, 71:2, 310-322.
Feldman HM, Dollaghan CA, Campbell TF, Colborn DK, Kurs-Lasky M, Janosky JE, Paradise JL. Parent-
reported Language and Communication Skills at One and Two Years of Age in Relation to Otitis Media in the
First Two Years of Life. Pediatrics electronic pages
Feldman HM, Kolmen BK, Gonzaga A. Naltrexone and Communication Skills in Young Children with Autism.
Journal of the American Academy of Child and Adolescent Psychiatry, 1999, 38:5, 587-593.
Feldman HM, Ploof D, Cohen WI. Physician-Family Partnerships: The Adaptive Practice Model. Journal of
Developmental and Behavioral Pediatrics, 1999, 20:2, 37-42.
Feldman HM, Ploof DL, Hofkosh D, & Goehring EL. Developmental needs of infants and toddlers who
require lengthy hospitalization. American Journal of Diseases of Children, 1993, 147, 211-215.
Feldman HM, Holland AL, Kemp SS & Janosky JE. Language development after unilateral brain injury.
Brain and Language, 1992, 42, 89-102.
Goldin-Meadow S & Feldman HM. The development of language-like communication without a language
models. Science, 1977, 197(4301), 401-403.

Courses & Training

I am Faculty Co-Coordinator of the Area of Concentration in Disabilities Medicine at the University of
Pittsburgh School of Medicine. I supervise medical students, residents, and fellows in general pediatrics
and developmental-behavioral pediatrics. I also participate in the Developmental and Reproductive
Biology Course for second year medical students.

Collaborations, Consultations, and Service

• SubBoard on Developmental-Behavioral Pediatrics, American Board of Pediatrics,
• Advisory Group, PA Chapter of the American Academy of Pediatrics, Early Care
and Education Linkage System
• PA Department of Health Family Health Advisory Council
• Advisor, Homeless Children and Family Emergency Fund, Pittsburgh
• Professional Advisory Board, Childhood Apraxia of Speech Association.
Member, Disabilities Agenda 2000 Implementation Task Force

185
 KENNETH J. JAROS, PH.D., MSW

Dr. Jaros is presently an Assistant Professor of Behavioral and Community Health Sciences in the Graduate
School of Public Health at the University of Pittsburgh and an Assistant Professor in the School of Social
Work. He is Director of the Public Health Social Work Leadership Training Program in Maternal and Child
Health funded by the U.S. Public Health Service in the Department of Health and Human Services. He is also
a core faculty member of the federally funded UCLID Center Neuro-developmental Disabilities Training
Program at the University of Pittsburgh. The MCH Leadership Training Program has been operating under his
direction for over 10 years and has consistently been successful in graduating large proportions of minority
students at both the masters and doctoral level.
Dr. Jaros also directs a Maternal and Child Health Distance Learning Continuing Education Grant funded
through the Maternal and Child Health Bureau. This grant focuses directly on preparing public health systems
to address issues of health disparities in their service areas. Other continuing education responsibilities
included the coordination of the annual Pittsburgh Public Health Social Work Continuing Education Institute.
He has primarily been involved in program development, evaluation, and operations research activities in
maternal and child health, health and human services delivery, and other public management areas. Besides
managing the two federal MCH Training grant, he has directed a variety needs assessment and evaluation
projects relating to Children with Special Health Care Needs, substance abusing women, state-wide MCH
planning, peer education, and violence prevention. Areas of interest include children and youth services,
community organizing, and addressing issues of health disparities.
Dr. Jaros teaches courses in Maternal and Child Health Services, Health Program Evaluation, Social Work in
Public Health, and Health Services Research, as well as assisting in several seminar series. He has also
taught courses in the Heinz School of Urban and Public Affairs at Carnegie Mellon University. He consistent-
ly served as a faculty mentor in University and UPMC sponsored summer internship programs for minority
students. Within the Graduate School of Public Health, he chairs the Faculty Diversity Committee, and is the
current president of the Faculty Senate. In addition he chairs the Departmental Admissions and Student
Performance Committee, and is on the School of Social Work Doctoral Committee.
In addition to the above management, research, and academic responsibilities, Dr. Jaros has recently provid-
ed technical assistance and consultation to several community-based health and social service agencies
including the Allegheny County Health Department, the Greater Pittsburgh Community Food Bank, Lutheran
Service Society; Women’s Center and Shelter, and the Program for Women Offenders. He is presently the
Vice Chair of the Pennsylvania State Child Health Insurance Program (SCHIP) Advisory Council, has served
several years on the Pennsylvania Maternal and Child Health Advisory Council, and locally has served on
advisory boards for the following organizations: Gateway Health Plan; South Hills Health System Home Health
Agency; Western PA March of Dimes; and, St. Francis Mental Health & Mental Retardation Program. Dr.
Jaros is a former Chair of the Social Work Section of the American Public Health Association.
Before coming to the University of Pittsburgh in 1985, Dr. Jaros held administrative, planning, and direct serv-
ice positions with several publicly funded social service and mental health organizations in the Pittsburgh
area. He is also a former Peace Corps volunteer. He recently (1999) co-edited the book, Health and Welfare
for Families in the 21st Century, which received an annual Best Book Award from both the American College
of Nurse Midwives, and the American Nursing Association.

186
SPECS Evaluation Team
Children’s Hospital of Pittsburgh
The UCLID Center at the University of Pittsburgh
3705 Fifth Avenue
Pittsburgh, PA 15213
(412) 692-6300
Email: uclid@pitt.edu
www.pitt.edu/~uclid

The work of the SPECS research team is funded by grants from The
Howard and Vira I. Heinz Endowments, and supported by a leader-
ship education grant to faculty of The UCLID Center at the University
of Pittsburgh from the US Department of Health and Human
Services, Maternal and Child Health Bureau.

Dr. Bagnato and his Early Childhood Partnerships program were

awarded the 2001 University of Pittsburgh Chancellor’s
Distinguished Public Service Award for their work with ECI and
other community joint ventures aimed at improving the quality of
programs which support families and young children at develop-
mental risk and with developmental disabilities. The award nomina-
tions came from diverse community partners.

© 2002 Early Childhood Partnerships, SPECS Evaluation Team

The material in this piece is intellectual property of the SPECS Evaluation
Team of the Early Childhood Partnerships program of Children’s Hospital of
Pittsburgh and the UCLID Center at the University of Pittsburgh. Any quotation
or other dissemination of this material without the express written consent of
the SPECS Evaluation Team is prohibited.

Design: Reinnov8