School
Indicators & Profiles SIG
A service to members of the American
Educational Research Association
Measurement of School and Classroom Effectiveness:
A Hierarchical Approach
Eugene P. Adcock, Prince George's County Public
Schools, MD
Dawn E. Sipes, Prince George's County Public Schools,
MD
Gary W. Phillips, U.S. Department of Education
Objective or Purposes
The results of the third phase of a three-phase study of school/classroom
effectiveness will be presented. The paper will be organized into four
sections:
-
Hierarchical Linear Modeling theory
-
Phases I and II --- School Effectiveness
-
Detailed results of Phase III --- School and Classroom Effectiveness
-
Implications and Future Plans
A short history of the design and findings of the first two phases of this
study will be presented as background information. Each phase analyzed
data from mandatory state testing which was administered in a different
school year. Both of these phases utilized a two-level hierarchical model
(i.e., school and student levels). The overwhelming effects of student
poverty were isolated from other school effects and each school's Value-added
Index (a poverty-free measure) was computed. In addition, the effect sizes
of statistically significant variables were reported (e.g., "if a school's
average teacher college training increase by X amount, that school's student
test scores increase by Y amount").
The results of Phase III will comprise the bulk of the paper. This phase
introduced a third, intermediate level to the hierarchical model: the classroom.
The previous phase of this study was a two-level analysis (school and
student) of school effectiveness. It indicated that teacher college training
had a significant positive effect on student test scores, and that smaller
class size improved test scores. Among the non-significant factors were
teacher years of experience, teacher salary, student instructional cost,
and teacher absenteeism. As a two-level analysis, Phase-II evaluated these
variables on a school-wide basis. Thus, for example, teacher experience
was measured as the average of all teachers within a school. This design
constrained the level of detail available for assessment, but was an accepted
limitation of this intermediate project phase.
Phase III, as the end-goal of the three-phase project, assesses these
variables at their optimal level: the classroom. This three-level analysis
will provide a closer look at the effects of teacher variables (e.g., college
training, years of experience, salary, absenteeism), class size, and student
instructional cost.
Policy implications and plans for future incarnations of this research
will be discussed.
Perspective(s) or Theoretical Framework
Maryland's statewide school assessment measure, the School Performance
Index (SPI), is computed from a combination of test scores and attendance
data. The authors demonstrate that this statistic is highly correlated
with student poverty. The confounding of SPI with poverty limits the usefulness
of SPI as a prescriptive device for school improvement.
This research effort yields a poverty-free school score, the Value-Added
Index (VAI). The VAI indicates the degree to which the students within
a school achieve more (or less) than other schools, given that school's
environment. With this "leveled playing field", school comparisons are
more meaningful.
With the introduction of the classroom as an additional level of analysis
in Phase III, a more precise estimate of the effect sizes of factors such
as teacher college training, teacher absenteeism, and class size can be
obtained. This analysis can serve as a launching pad for the quantitative
assessment of the effectiveness of numerous teaching practices in future
studies.
Methods, Techniques or Modes of Inquiry
A brief review of the merits of Hierarchical Linear Modeling (HLM) will
establish the underlying design of the project. Beginning with simple pairwise
correlations, the reader will progress through a series of user-friendly
descriptions of increasingly complex and sophisticated analytic techniques.
By demonstrating with a simple hypothetical example, the advantages of
HLM over other statistical techniques will be posited.
Data Sources or Evidence
References will be provided which document the design and structure
of the Research, Evaluation and Assimilation Database (READ) warehousing
system. READ is a controlled relational database warehouse system which
contains the school district's historical legacy data. The design of READ
permits the linking of assessment outcome data to student characteristics,
teachers, and school characteristics and provides the ability to extract
data records in a way that reflects the actual hierarchical nature of schooling
(i.e., where students are assigned to classes and take courses from teachers
within schools, and schools are assigned to programs, such as Magnet programs).
This sophisticated data warehouse supports the extraction of a statistically
sufficient data structure for complex analyses such as HLM. HLM, in turn,
is a powerful method for evaluating how student, teacher and school characteristics
play a part in observable educational outcomes (e.g., mathematics test
scores, reading test scores) and, in particular, how the learning environment
of the school and teachers contribute to that outcome.
The combination of the READ data warehousing system and the HLM statistical
design provide a foundation for assessing school and classroom effectiveness.
Two successful previous phases of this study lend credibility to this approach.
Results and/or Conclusions/Point of View
Results will include a list of the variables that proved statistically
significant at the classroom and school levels, as well as a list of those
variables that were assessed but were not found to have a significant impact
on student achievement. In addition, the effect size of each significant
variable will be reported using a meaningful scale: test scale score points.
By reporting the effect size in this easy-to-understand metric, the practical
significance of the findings can be easily assessed.
Educational or Scientific Importance of the Study
The reader who is unfamiliar with HLM will receive a primer on the technique's
usefulness and underlying assumptions. Evaluators and others interested
in the measurement of school effectiveness will see a method to tunnel
beyond the overpowering effects of demographic variables, such as student
poverty, to more truly determine whether a given school is adding to, or
detracting from, the academic outcomes of its student population. The relative
effect size of variables under the school's control or under the district
office's control will be of interest to policy makers.
This statistical approach to isolating the effects of classroom variables
has enormous potential for evaluating teaching practices, classroom instructional
programs, and educational policy.
Eugene P. Adcock,
Prince George's County Public Schools
Sasscer Administration Building
14201 School Lane, Room 138
Upper Marlboro, MD 20772
< |