SIP:SIG Home

Officers

Members

Annual Meetings

Membership

Newsletter

Resources

AERA Home


 
 
 
 

School Indicators & Profiles SIG

A service to members of the American
Educational Research Association

Intervention or Reinvention?
An Analysis and Delineation of Issues of Construct Validity
in One State's Accountability Legislation


Jerry G. Mathews, Auburn University
E. Raymond Hackett, Auburn University
James R. Pennell, Auburn University

A.  Purposes

The purpose of this study was to examine the construct validity of accountability measures and performance indicators implemented for one state system of public instruction. Establishing construct validity is a necessary step in determining the appropriateness of measures for systemic decision making and public reporting. In this study accountability constructs were defined in terms of the intent written in accountability legislation, state board of education rulings, state department of education regulations, and adopted state accreditation measures. Construct validity was explored examining the performance measures defined in the State Superintendents Report Card for local education agencies. This paper serves as a challenge to the research community to take an active role in the study of measures defined as instruments of public education policy.
 

B.  Perspectives

State defined performance indicators for education are rapidly becoming the hallmark of the 1990's. As of 1996, 52 of 54 state education agencies had at least one annual accountability or indicator report. Thirty-two states now have some form of accountability legislation enacted that require the reporting of school and school system performance indicators. Most of the analysis and literature on the development of state-defined performance indicators describe a pattern of implementation with little prior conceptual development. Inter-school or school district ratings and comparisons are central to much of this legislation and focus, most often, on student performance measures and school finance data. These type of indicators provide more information on school and school system need than they do on the quality of education that is provided. The challenge in developing accountability legislation is to define assessment strategies and performance indicators that support reforming school practice and assure access to educational outcomes. Are these measures to be used for reinvention and renewal, or intervention?

Student and school indicators have become additional points of focus in accountability efforts and legislative mandates. The Council of Chief State School Officers (CCSSO) State Education Policies Reports have attempted to provide state-by state information on policies relating to time attendance, content standards, high school graduation requirements, and teacher licensure. In addition, the CCSSO has assisted states in developing performance indicators at the district and school levels. Research on issues related to state-mandated accountability efforts is increasing. A key focus for research should be on the construct validity of accountability measures. Are the performance indicators invoked by a state for systemic decision making, public reporting and accreditation related to the constructs circumscribed in law or administrative ruling? This paper was an attempt to begin research on construct validity in one state.
 

C.  Methodology

The methodology for this study included three phases: (a) a study of the constructs that define educational accountability for Alabama school districts, (b) an analysis of the relationship between report card indicators and the constructs defined in the accountability and accreditation legislation and rulings, (c) an analysis of local education agency characteristics and success at meeting state defined performance indicators.

In the first phase of the study, constructs that warranted measure were defined in terms of the intent written in accountability legislation, state board of education rulings, state department of education regulations, and adopted state accreditation measures. To analyze the accountability legislation and board of education or state education agency written policy documents, an approach informed by qualitative grounded methods was adopted. The texts were treated as field data or interview transcripts where the meanings are viewed as self-evident. The investigators used an iterative process of open coding, categorizing, recoding, and axial coding or recategorizing of relevant passages of laws and regulations.

In the second phase of the study the accountability measures defined in the State Superintendents Report Card for local education agencies in the Alabama were matched with the constructs defined in phase one through a meta analysis of the literature related to accountability legislation and rulings, and performance indicators. The meta analysis also served to explore issues related to accountability measures and construct validity in other states.

In phase three of the study school district performance on the fourteen report card measures were examined in the context of individual local education agency financial, demographic, student and academic characteristics using descriptive and multivariate statistics.
 

D.  Data Sources

The currently applicable accountability legislation, state board of education rulings, state department of education regulations, and adopted state accreditation measures were verified and original copy obtained from The Code of Alabama, Minutes of the State Board of Education, and Regulations of the Alabama State Department of Education. The State of Alabama State Superintendent's Report Card FY 1995-96 provides data on 14 performance indicators that define some measure of accountability for the 128 local education agencies (LEAs) in the Alabama system. Data from each of the LEAS for each of the indicators for the school year 1995-96 was collected from this report. Additional information on LEA characteristics was collected from the Annual Report of Statistical and Financial Data issued by the Alabama State Board of Education.
 

E.  Results or Conclusions

The results of the first phase of the study was a finite list of constructs defining local education agency accountability measures and accreditation measures in the state under study. These constructs were classified according to four domains:  minimum compliance with regulations; academic performance; student behavior; and administrative or fiscal management.

The results of the second phase of the study found that a meta analysis of the related literature warranted classifying this state's accountability measures along three dimensions: (1) is used as a measure for related constructs and is reported as useful for systemic decision making, public reporting or accreditation; (2) is used as a measure for related constructs and is reported as not useful for systemic decision making, public reporting or accreditation; (3) is not used as a measure for any of the constructs defined.

The results of the third phase of the study indicate that state-established performance indicators can be analyzed as to their appropriateness for systemic decision making and public reporting. However, the measures defined in Alabama State Superintendent's Report Card provide more information on school and school system need than they do on the quality of education that is provided.
 

F.  Educational Importance of the Study

The implication in this study for educational researchers is that state established accountability measures and performance indicators should undergo rigorous analysis in terms of the validity and reliability related to the constructs that they are purported to measure. This paper serves as a challenge to the educational research community to take an active role in the study of measures defined as instruments of public education policy.
 

Jerry G. Mathews
>4036 Haley Center
Auburn University, AL 36849


Hosted by:
Learning Point Associates