SIP:SIG Home

Officers

Members

Annual Meetings

Membership

Newsletter

Resources

AERA Home


 
 
 
 

School Indicators & Profiles SIG

A service to members of the American
Educational Research Association

Measurement of School and Classroom Effectiveness:
A Hierarchical Approach


Eugene P. Adcock, Prince George's County Public Schools, MD
Dawn E. Sipes, Prince George's County Public Schools, MD
Gary W. Phillips, U.S. Department of Education

Objective or Purposes

The results of the third phase of a three-phase study of school/classroom effectiveness will be presented. The paper will be organized into four sections:

  1. Hierarchical Linear Modeling theory
  2. Phases I and II --- School Effectiveness
  3. Detailed results of Phase III --- School and Classroom Effectiveness
  4. Implications and Future Plans
A short history of the design and findings of the first two phases of this study will be presented as background information. Each phase analyzed data from mandatory state testing which was administered in a different school year. Both of these phases utilized a two-level hierarchical model (i.e., school and student levels). The overwhelming effects of student poverty were isolated from other school effects and each school's Value-added Index (a poverty-free measure) was computed. In addition, the effect sizes of statistically significant variables were reported (e.g., "if a school's average teacher college training increase by X amount, that school's student test scores increase by Y amount").

The results of Phase III will comprise the bulk of the paper. This phase introduced a third, intermediate level to the hierarchical model: the classroom.

The previous phase of this study was a two-level analysis (school and student) of school effectiveness. It indicated that teacher college training had a significant positive effect on student test scores, and that smaller class size improved test scores. Among the non-significant factors were teacher years of experience, teacher salary, student instructional cost, and teacher absenteeism. As a two-level analysis, Phase-II evaluated these variables on a school-wide basis. Thus, for example, teacher experience was measured as the average of all teachers within a school. This design constrained the level of detail available for assessment, but was an accepted limitation of this intermediate project phase.

Phase III, as the end-goal of the three-phase project, assesses these variables at their optimal level: the classroom. This three-level analysis will provide a closer look at the effects of teacher variables (e.g., college training, years of experience, salary, absenteeism), class size, and student instructional cost.

Policy implications and plans for future incarnations of this research will be discussed.
 

Perspective(s) or Theoretical Framework

Maryland's statewide school assessment measure, the School Performance Index (SPI), is computed from a combination of test scores and attendance data. The authors demonstrate that this statistic is highly correlated with student poverty. The confounding of SPI with poverty limits the usefulness of SPI as a prescriptive device for school improvement.

This research effort yields a poverty-free school score, the Value-Added Index (VAI). The VAI indicates the degree to which the students within a school achieve more (or less) than other schools, given that school's environment. With this "leveled playing field", school comparisons are more meaningful.

With the introduction of the classroom as an additional level of analysis in Phase III, a more precise estimate of the effect sizes of factors such as teacher college training, teacher absenteeism, and class size can be obtained. This analysis can serve as a launching pad for the quantitative assessment of the effectiveness of numerous teaching practices in future studies.
 

Methods, Techniques or Modes of Inquiry

A brief review of the merits of Hierarchical Linear Modeling (HLM) will establish the underlying design of the project. Beginning with simple pairwise correlations, the reader will progress through a series of user-friendly descriptions of increasingly complex and sophisticated analytic techniques. By demonstrating with a simple hypothetical example, the advantages of HLM over other statistical techniques will be posited.
 

Data Sources or Evidence

References will be provided which document the design and structure of the Research, Evaluation and Assimilation Database (READ) warehousing system. READ is a controlled relational database warehouse system which contains the school district's historical legacy data. The design of READ permits the linking of assessment outcome data to student characteristics, teachers, and school characteristics and provides the ability to extract data records in a way that reflects the actual hierarchical nature of schooling (i.e., where students are assigned to classes and take courses from teachers within schools, and schools are assigned to programs, such as Magnet programs).

This sophisticated data warehouse supports the extraction of a statistically sufficient data structure for complex analyses such as HLM. HLM, in turn, is a powerful method for evaluating how student, teacher and school characteristics play a part in observable educational outcomes (e.g., mathematics test scores, reading test scores) and, in particular, how the learning environment of the school and teachers contribute to that outcome.

The combination of the READ data warehousing system and the HLM statistical design provide a foundation for assessing school and classroom effectiveness. Two successful previous phases of this study lend credibility to this approach.
 

Results and/or Conclusions/Point of View

Results will include a list of the variables that proved statistically significant at the classroom and school levels, as well as a list of those variables that were assessed but were not found to have a significant impact on student achievement. In addition, the effect size of each significant variable will be reported using a meaningful scale: test scale score points. By reporting the effect size in this easy-to-understand metric, the practical significance of the findings can be easily assessed.
 

Educational or Scientific Importance of the Study

The reader who is unfamiliar with HLM will receive a primer on the technique's usefulness and underlying assumptions. Evaluators and others interested in the measurement of school effectiveness will see a method to tunnel beyond the overpowering effects of demographic variables, such as student poverty, to more truly determine whether a given school is adding to, or detracting from, the academic outcomes of its student population. The relative effect size of variables under the school's control or under the district office's control will be of interest to policy makers.

This statistical approach to isolating the effects of classroom variables has enormous potential for evaluating teaching practices, classroom instructional programs, and educational policy.
 
 

Eugene P. Adcock,
Prince George's County Public Schools
Sasscer Administration Building
14201 School Lane, Room 138
Upper Marlboro, MD 20772 <