Introduction

What constitutes an adequate education? And what should this cost? Several states are exploring these questions as they attempt to revise their school funding formulas. Some are doing so in response to court orders that require them to eliminate current funding inequities (e.g., New Jersey, Ohio, Wyoming), while others are trying to introduce more equity and accountability into their school funding formulas (e.g., Illinois, Minnesota, Mississippi, Oregon). Maine is doing so in response to the desire on the part of policymakers to change Maine's school funding from an expenditure-driven formula to a results-driven formula. This paper will describe Maine's efforts to date and the impact of this work on perceptions and policymaking.
 

Background

The purpose of our contribution is to explore the effect of ignoring one or more levels of variation in hierarchical regression analysis. In a first analysis four hierarchical levels will be considered: the individual pupil, the class group, the teacher and the school. We will investigate the impact of ignoring the highest level (the school level), the highest two levels (the school and the teacher level) and ignoring one or two intermediate levels (the teacher and/or class level) on the attribution of the variance to the levels taken into consideration .
 

Unlike many states in which the state constitution requires the state to provide a thorough and efficient education for all students, Maine's constitution merely states that "...the legislature are authorized, and it shall be their duty to require the several towns to make suitable provision, at their own expense (emphasis added), for the support and maintenance of public schools...". However, since the early 1830's, the state of Maine has played an ever increasing role in the financial support of schools by providing state tax money to the towns in support of education. And until recently, Maine's formula has been an expenditure-driven one. Beginning in 1995 Maine adopted a foundation program (except for transportation, special education , debt service, and vocational education) designed to ensure that the state paid for at least 57% of education costs. However, this funding law has come under criticism for three major reasons: (1) it allows inequities to continue because of disparities in the capacity of different communities to raise substantially different amounts of local school aid; (2) it permits the state in times of limited resources to reduce its share of costs and thereby increase local costs; and (3) it provides little accountability for how the funds are spent at the local level.

From 1995-1997, the Maine Legislature took two key steps to change the way schools are to be funded and held accountable. First, the legislature passed the Learning Results, a set of learning standards in eight disciplines that schools must ensure that all students achieve before they graduate from high school. Second, the legislature directed the State Board of Education to develop a method of funding the programs and services necessary for students to achieve the statewide learning standards. More specifically, Maine Public Law 1997, Chapter 24 charges the State Board of Education with the development of:

...an implementation plan for funding essential programs and essential services. The plan must be based on the criteria for student learning developed by the Task Force on Learning Results and established in Public Law 995, Chapter 649 and in rules adopted by the Board and the Department of Education. The plan must include establishment of a system to measure and ensure that schools are held accountable for student learning results.

Thus, the state board was charged with identifying the necessary inputs (programs and services) required to produce the desired outputs (high student achievement) and to develop a plan for holding schools accountable for student achievement. From the perspective of policymakers (and business leaders), this charge seems very rational, reasonable, appropriate and appealing. But as researchers and educators know all too well, the devil is in the details. Some of these details and the approach taken in addressing them are described in the next section of this paper. A subsequent section discusses the impact of this approach on perceptions and policymaking.
 

The Maine Strategy

In response to the legislation, the state board in late spring 1997 established the Essential Programs and Services (EPS) Committee and contracted with the Maine Education Policy Research Institute (MEPRI) for research and technical assistance in determining the cost of a Maine core education. The committee consisted of members representing various stakeholder groups (e.g., teachers, administrators, school committees, State Board of Education, Department of Education, and businesses). The research institute is a non-partisan institute jointly funded by the state legislature and the Maine university system, and charged by law with tracking K-12 developments and reform in Maine and with conducting targeted research at the request of the legislature. The University of Southern Maine office of the research institute has been working with the EPS Committee in developing the Maine core education model since summer 1997.

The first step the committee faced in developing the Maine EPS model was defining what constituted an essential program or service. After considerable deliberation, the committee concluded that their work should be guided by one fundamental premise: The purpose for developing a Maine EPS model is to ensure that all schools have the necessary and essential programs and services so that all students have equitable educational opportunities to achieve the Learning Results. Based on this premise, the committee has developed a working definition of essential programs and services, a definition that will be refined as work on the EPS model continues throughout this year. This working definition is as follows:

Essential Programsare those programs and courses schools need to offer all students, including special education and applied technology students, so that students may meet the Learning Results standards in the eight Learning Results program areas of:
 

• Career Preparation
• English Language Arts
• Health and Physical Education
• Mathematics
• Modern and Classical Languages
• Science and Technology
• Social Studies
• Visual and Performing Arts
  

Essential Services are those services which are necessary to ensure that each Maine student is offered an equal opportunity to achieve standards within the essential programs described above. These include:

• school unit personnel
• guidance services
• library services
• health services
• administrative services
• early childhood services
• social and family services
• special education services
• Limited English Proficiency (LEP services)
• disadvantaged youth services
• co-curricular student activities
• applied technology
• professional development
• student assessment
• equipment and supplies
• technology resources
• facilities
• transportation
• debt services
• others (to be determined)

Having established this purpose and these definitions, the committee came face to face with several "devil in the detail" questions: (1) What standards of achievement constitute sufficient achievement (how good is good)? (2) What level of each of these programs and services is necessary and what should these cost? and (3) How do we know that certain levels of programs and services will bring about the desired achievement levels? Early reviews of what other states are doing in defining core education also led the committee to one inescapable conclusion: no magic bullet exists. Maine would be charting new waters in developing a results-driven model for funding its K-12 education, a model that tied essential inputs to explicit outputs (student achievement).

After reviewing the literature on the education production model, the committee still found some merit in exploring the connection between inputs and outputs in Maine schools. That is to say, the committee reviewed the research conducted by Hanusheck (1996, 1997), Hedges, Laine & Greenwald (1994) and others and recognized that the production model findings are, at best, mixed. However, they did conclude that some of the most recent research held promise. Thus, recognizing the central need to provide convincing arguments for the way in which the committee would ultimately define essential program and services and their costs in Maine, the committee concluded it was important to examine empirical evidence from Maine schools. Accordingly, the committee requested that the research institute conduct a study of Maine performance and the relationships, if any, between performance and school characteristics.
 

High and Low Performing Maine Schools

The first step in exploring the Maine evidence entailed defining student performance and a standard of student performance to use in examining Maine schools. The Maine legislation establishing the statewide learning standards also mandated that the statewide achievement test, the Maine Educational Assessments (MEA), be used to assess achievement of the learning standards. The MEA is in the process of being revised to measure the eight disciplines in the Learning Results, and the first testing is scheduled for 1999. Thus, it will be several years before the state will have adequate, reliable and defensible evidence on student achievement of the Learning Results. At present, the existing MEA provides the only statewide data on student performance. Because the current MEA is the only available data on performance, and because the current MEA does assess content similar to that found in the new Learning Results, MEA data for the last three years was used as a surrogate for future Learning Results performance.

The MEA measures 4^th, 8^th, and 11^th grade student achievement in six core areas: reading, writing, mathematics, science, social studies, and arts and humanities. All of these areas are included to varying degrees in the new statewide learning standards, but the current MEA will still need substantial realignment for use in the future. At present MEA achievement is reported in six norm-referenced scale scores and three criterion based proficiency level scores. Student level scale scores and proficiency levels are reported for reading, writing and mathematics.

Three criteria were selected for defining a high or low performing Maine school. These were the following:

1. A school level composite scale score average of one-half standard deviation above or below the state average.
2. 75% or more (or less) of the students in the school scoring at the Basic or above proficiency levels in reading, writing and mathematics.
3. A school level composite scale score average 0.50 residual z-score above the predicted school score.

To determine the first criteria, the six subtest scores over the last three years were averaged. The average could range from 100-400 points with a standard deviation of 50 points. In the case of the second criteria, student performance in reading, writing, and mathematics is also reported based on four criterion-referenced proficiency levels: Novice, Basic, Advanced and Distinguished. The average percent of students reaching Basic or above in the three content areas for the last three years was used in identifying schools.

Determining the third criteria, the value-added criteria, was more complex. Based on the work of Phillips and Adcock (1996) and Hofmann (1997), multiple regression was used to examine the relationship between school level school scale performance (Criteria A) and six community and school demographic variables. These variables were (1) median household income; (2) percent below poverty level; (3) percent college degree; (4) percent in professional position; (5) percent free or reduced lunch students in the school; and (6) the midpoint of the school MEA comparison score band. The four community variables were from the 1990 census. The free and reduced lunch percentages were provided by the Maine Department of Education, and the comparison band information was provided by the private testing agency responsible for developing and administering the MEA. Three demographic factors are used by the agency in calculating school comparison score bands: (1) percent free and reduced lunch; (2) percent of parents in white collar positions; and (3) presence of computer/encyclopedia in home (elementary level) or highest degree level achieved by one parent (middle and high school level). Table 1 reports the simple correlation matrix for the six demographic variables and 11^th grade school level MEA performance. The information for the 4^th and 8^th grades appears in Appendix A.
 

. Table 1 Correlations Between Demographic Variables and 11^th Grade MEA Performance  

Based on this evidence, a stepwise multiple regression analysis was conducted for each of the three grade levels (4^th, 8^th and 11^th). This analysis yielded two variables in the case of 4^th grade (comparison score band midpoint and percent poverty), and only one variable in the 8^th and 11^th grade cases (comparison score band midpoint). These findings were expected given that the comparison band score is calculated for parents only and does not include non-parent community members (i.e., census data). The resulting R² were .26, .42 and .52 for the 4^th , 8^th and 11^th grades respectively. Based on this analysis, actual and predicted scores were generated for each school in Maine, and the three criteria described earlier were used in identifying high and low performing schools. This procedure resulted in the identification of 133 high performing and 120 low performing schools. The breakdown by school level appears in Table 2. Data from these sets of schools were examined using analysis of variance procedures to determine if significant differences were present between the schools on a variety of input and resource variables. Variables examined in this analysis were those found in other research studies, and variables that are being used by the Maine research institute in developing a statewide indicator system. 

Table 2 Breakdown of High and Low Performing Schools  

Table 3 reports the findings from these analyses. This empirical information, along with evidence collected from other states and task forces which are attempting to develop core education models, is being used by the committee in developing the Maine model.

The committee has chosen to use prototypical schools (like New Jersey and Wyoming) as a means for describing the Maine model and components. Examples of three prototypical schools, and the work to date, appear in Appendix B. Work on the model is far from being completed and will continue throughout this calendar year and into the next legislative session.

Some of the additional tasks the committee will need to complete include: (1) developing an accountability system including consideration of rewards and sanctions; (2) determining the local and state shares of funding the EPS model; and (3) determining what portion of the funds distributed will be as block and/or targeted grants.
 
  Table 3 Variable Examined in High and Low Performing Schools

*Statistically significant differences (p<.05) between high and low performing schools;

NS = no statistically significant differences; NA = not applicable
 

Impact on Perceptions and Policymaking

Considerable work is still to be completed on the Maine model, and undoubtedly the model will undergo many changes as a result of further discussion of the committee, public hearings and legislative review. Still the impact of using school level achievement data in building the Maine model is already evident in perceptions and policymaking.

First, and foremost, the empirical data has helped inform committee discussions and decisions. One example is the evidence on teacher-student ratios. The assumption held by most educators, and many parents, is that class size is a critical variable in achievement. Nationally the evidence is mixed, but the data from Maine schools indicated that teacher-student ratios were not a distinguishing variable in high and low performing schools. That is to say, with the one exception of schools which housed only part or all of grades 5-8, no significant differences were found for teacher- student ratios in high and low performing schools. Subsequent analysis of these middle schools indicated the differences found between the high and low performing schools were related to differences in school size. The committee has used this finding in establishing ratios in the EPS model.

Another example of the impact of the empirical evidence in the model building process is in the area of teacher educational background. The evidence from the Maine schools indicated that a higher percentage of teachers in high performing schools held masters degrees in comparison to their colleagues in low performing schools. The committee has recognized that degree attainment may be a powerful marker in itself, but also a marker for a broader indicator of professional development. Thus, the committee is exploring ways to build funding into the Maine EPS model for both further formal education (i.e., degree programs) and professional development.

A second impact of the work to date on the EPS model is a key change in perceptions held by many educators and policymakers throughout the state. Folk wisdom is that high achieving schools are located in rich, high SES communities. In fact, the lack of these community resources is often used as a justification for low performance - both on the part of educators and legislators alike. The evidence from this study of Maine schools suggests this is not always the case. The chart on the next page identifies the high (+) and low (-) performing high schools by county. This chart (and a similar one for elementary schools) has served as an eye opener and a springboard for many lively discussions throughout the state. Individuals are realizing that some schools across the state, from both rich and poor communities, are being successful in helping a majority of their students achieve a statewide standard of performance. They are realizing that what goes on inside the school can make a difference.

A third impact has been in the area of information development. Like most states, much of the education data collected in Maine is of a financial nature and used for regulatory and compliance purposes only. Fortunately, Maine has collected some recent school resource survey data, but still the state is data poor for purposes of policy research. Much of the data is incomplete, unverified, and at the district level. This study by the EPS Committee has significantly increased the awareness on the part of policymakers that if they intend to make more data-informed decisions, and to hold schools accountable for their use of resources and results, more precise information is needed from schools and school systems. Accordingly, policymakers have instituted a review of all educational data collected by the state in anticipation of developing a new data collection system, one which will provide information useful in an accountability system and in future policy research studies.

Fourth, this study is having a positive impact on the future of policy research development in the state. The research described in this paper is less detailed, comprehensive and rigorous than one would expect in an in-depth study of the production model. Nevertheless, it has served as a good example of how research may inform the policymaking process. Consequently, in the case of Maine, the research has created a move for improving the quality and level of data collected in the state. In addition, the study of high and low performing schools and the use of this information in developing new statewide policy such as the EPS model has created a renewed (or created a new) interest and appreciation on the part of the Education Committee of the legislature for the value and usefulness of policy research. As a result, the committee is turning more and more to the research institute for information and analysis, and has recommended doubling the institute budget this year (as an emergency measure).
 

Summary

Using school level achievement data as a strategy in defining a core education is having a substantial impact on perceptions and policymaking in Maine. The data and its use in exploring the empirical evidence about Maine schools has proved useful to the committee charged with defining the essential programs and services that schools should have available to them in providing equity of educational opportunities for all students. The committee used the study findings to inform and guide their deliberations. Additionally, the study has proved beneficial in other areas. It has been useful in calling into question Maine folk wisdom about the power of community factors in setting student achievement and the potential power of effective schools in overcoming these community factors. The study has also served as a catalyst for improving the education data collection system in Maine and for improving both the stature and funding of the state education policy research institute. Only time will tell if the findings from this analysis have a lasting impact on how Maine defines and funds a core education, but the prospects are good that the analysis will improve Maine education policy development in the future.
 
 
 
 

REFERENCES

Ferguson, R. & Ladd, H. (1996). How and Why Money Matters: An Analysis of Alabama Schools. In Ladd, H. (1996). Holding Schools Accountable. Washington, D.C.:  The Brookings Institution.

Hanushek, E. (1996). School Resources and Student Performance. In Burtless, G.(1996). Does Money Matter? Washington, D.C.: The Brookings Institution.
 
Hanusheck, E. (1997). Assessing the Effects of School Resources on Student Performance: An Update. Educational Evaluation and Policy Analysis 19 (2): 141-164.
 
Hedges, L., Laine, R. & Greenwald, R. (1994). Does Money Matter? A Meta-Analysis of Studies of the Effects of Differential School Inputs on Student Outcomes. Educational Researcher. 23 (3): 5-14.
 
Hofmann, R. (March 1997). School district effectiveness, difficulty of the educational task, and Ohio ninth-grade proficiency exams: A seven year study. Paper presented at the annual meeting of the American Educational Research Association.
 
Phillips, G. & Adcock, E. (April 1996). Practical applications of hierarchical linear models to district evaluations. Paper presented at the annual meeting of the American Educational Research Association.
 
 
 
 
  APPENDIX A

Table 4  Correlations Between Demographic Variables and 4^th Grade MEA Performance  

  Table 5  Correlations Between Demographic Variables and 8^th Grade MEA Performance

David L. Silvernail
Director
Center for Educational Policy,
Applied Research and Evaluation
University of Southern Maine
Gorham, Maine 04038
Voice: (207) 780-5297
Fax: (207) 780-5315
E-Mail: davids@usm.maine.edu