SIP:SIG Home

Officers

Members

Annual Meetings

Membership

Newsletter

Resources

AERA Home


 
 
 
 

School Indicators & Profiles SIG

A service to members of the American
Educational Research Association

Are Teachers' Classroom Assessment Practices Useful Indicators of Education Reform


Jeffrey S. Beaudry, University of Southern Maine


Introduction, Goals and Objectives

Over the past ten years, the Maine Educational Assessment (MEA) has focused on creating a statewide assessment system that provides a positive link with teaching and learning. In addition to the on-demand assessment of student achievement, the MEA provides teachers and administrators with Performance Level Guides containing student work and holistic rubrics or scoring guides for all content areas and for all performance levels. Teachers are encouraged to orient students to the types of questions to be asked and the proper format for responses to the questions. In the 1994-1995 MEA survey over 30 percent of teachers in high school English/Language indicated that they not only used these materials for orienting and preparing students for the MEA, but also incorporated the MEA materials as models for their own classroom assessments (Beaudry, 1997). While teachers may have reported that these materials have been useful, questions remain as to whether the use of rubrics and scoring guides is related to the improved students' performance.

A number of questions surround the MEA approaches to changing classroom assessment. For example, what is the reported level of familiarity by Maine's teachers with MEA resources and materials in reading and writing? Do the teachers provide the opportunity for student orientation to the materials? What is the impact of the Performance Level Guides in reading and writing on teachers' classroom assessment practices? Finally, what is the prevalence of students' use of scoring guides to evaluate their own work or the work of others?

Theoretical Framework

Cognitive classroom perspectives (Glaser, 1976; Bransford & Vye, 1989; Resnick & Resnick, 1992) indicate that students' perform at a higher level when they understand the components of quality for specific achievement goals. Recommendations by classroom assessment practitioners like Marzano, Pickering, & McTighe (1993), and Stiggins (1997) argue strongly for the central role that students can perform in classroom assessment. Students can be effective participants in processes to develop definitions of quality work in scoring guides and rubrics and apply these guides in self-assessment and assessment of other students' work. While there have been arguments favoring analytic scoring guides over holistic scoring guides (Marzano & Kendall, 1997; Stiggins, 1997), teachers can begin to understand appropriate applications with either approach to classroom assessment. This study examines the relationship of the MEA approach to improving classroom assessment with students' performance on reading and writing achievement.
 

Methods & Data Source

Since 1992 the Maine Educational Assessment shifted from selected response, multiple choice questions to open-response, student constructed response questions. The MEA is an annual assessment for all students in grades 4, 8, and 11. Teachers receive a packet of materials before administering this examination, including Performance Level Guides which show examples of student work corresponding to the levels of performance in the scoring guide or rubric. The information for this analysis comes from a survey of teachers in grades 4 (n=380), 8 (n=229), and 11 (n= 122). The survey examines teachers' background, classroom practices, and attitudes towards curriculum and instructional improvement.

To examine the effects of using holistic rubrics in classrooms, Maine high schools were divided into high- and low-achieving groups according to whether their 3-year average scaled scores were in the highest and lowest quartile of MEA reading and writing results. Based on the comparisons of low- and high-achieving schools three questions were examined: 1) is there a higher frequency of use of holistic rubrics by teachers to evaluate students' work, 2) is there a higher frequency of use of holistic rubrics by students to evaluate other students' work, 3) and what is the proportion of students at higher performance levels in reading and writing in high-achieving schools in comparison with low-achieving schools? In addition, correlational analysis and path analysis will be used to demonstrate the interaction of classroom assessment variables with resources like teachers' professional background and student achievement.
 

Results and Discussion

According to the 1994-1995 survey of teachers, approximately 78 percent of high school English/Language Arts teachers reported that they had used the MEA materials to familiarize students with the test format before administering the test. Just over 30 percent of this group indicated that they incorporated the MEA materials as models for their own classroom assessments. Approximately 9 percent of all teachers reported that the MEA materials were unavailable to them. These findings were consistent across grade levels.

A follow up question on the survey asked whether the teachers themselves used the MEA scoring guides teaching to evaluate students' responses. Roughly 20 percent of all high school English/Language Arts teachers reported that they never used holistic rubrics at all. Fifty-four percent (54 percent) of high school teachers indicated that they used holistic rubrics once or twice a month and an additional 20 percent used them to evaluate students' responses in English/Language Arts on a weekly basis.

How involved are students in the use of holistic rubrics to evaluate their own or other students' responses? Fifty-five percent (55 percent) of Maine's teachers provided opportunities for students to use the MEA holistic rubrics or similar scoring guides to evaluate other students' written work on a monthly basis. One third of the teachers never used this form of evaluation in their classroom.

Based on the 1994-1995 survey it is apparent that 80 percent of Maine's teachers are familiar with and use the MEA reading and writing resources like holistic rubrics and scoring guides. This indicates that a great majority of the English/Language Arts teachers in Maine high schools take the time to orient students to the MEA. Almost one third of the surveyed teachers have found the MEA materials useful enough to be incorporated in their regular instruction. Still, a surprisingly large proportion of English/Language Arts teachers, approximately 22 percent, said they made little or no connection to the MEA for their students the classroom for their students. Similarly, 80 percent of teachers reported using rubrics to evaluate student work. A much smaller proportion of teachers, roughly 45 percent, directly involve their students in using rubrics to evaluate their own work. Overall, a majority of high school teachers found the MEA materials accessible and employed strategies for assessment using the holistic rubrics and scoring guides in their classrooms. The assessment practices modeled by the MEA in reading and writing have apparently been useful for teachers as well as for Maine's students.

In the 1994-1995 MEA survey over 85 percent of high school teachers in high-achieving schools reported that they used holistic rubrics to evaluate student work as compared with 65 percent of teachers in low-achieving schools. Teachers in low-achieving schools tended to use holistic rubrics less frequently, with 14 percent of  high school teachers using them on at least a weekly basis as compared with 31 percent of teachers in high achieving schools. The disparity between high- and low-achieving high schools is accentuated when examining the use of holistic rubrics by students. Almost 60 percent of teachers in low-achieving schools never engaged students in classroom assessment by using holistic rubrics to evaluate their own work or the work of other students. In high-achieving schools over 77 percent of the teachers reported that students engaged in the use of holistic rubrics, and 20 percent of the teachers used this approach at least once a week.

What are the performance level profiles of high- and low-achieving high schools? The results in Table 1 show the frequency distribution of students across four performance levels in reading and writing-novice, basic, advanced, and distinguished--is remarkably consistent. Overall, approximately 47 percent of students in both school groups achieved the basic performance level in reading and writing. However, the high-achieving high schools have fewer students in the novice level, approximately 16 percent of their students as compared with 34-37 percent of students in low-achieving schools. Also, high-achieving high schools had twice the proportion of students at the advanced level and four times the proportion of students at the distinguished level in both reading and writing.
 

Table 1: Percentage of students at each performance level
 
Novice Basic Advanced Distinguished
Level 1 Leve1 2 Leve1 3 Leve1 4
Reading Writing Reading Writing Reading Writing Reading Writing
Low-achieving
high schools
36.7 34.1 47.3 46.6 15.0 16.2 1.0 1.0
High-achieving
high schools
15.2 16.6 48.7 48.1 32.0 31.2 4.1 4.1
 

These findings suggest that there may be a positive relationship between the use of rubrics and student performance on the MEA. In the two questions involving classroom assessment, teachers and students in high-achieving schools were more involved in using assessment methods for reading and writing modeled by the MEA. As suggested by Stiggins (1997) students involved in their own assessment may be able to make a clearer connection between classroom instruction and achievement expectations. Analyses of the use of holistic rubrics for reading and writing results showed virtually the same results. The shift in achievement is most noticeable at the novice and advanced levels, an encouraging development for motivating low-achieving students. By having students as key users of assessment the MEA may be attaining their goal of fostering higher student achievement through connections with classroom assessment practices.
 

Educational Significance

The MEA has provided an important view of student achievement by targeting changes in teachers' classroom assessment practices, which will be an important component in the development of credible, local accountability systems. This study of the use of holistic rubrics and students' work provides an opportunity to examine the strength of this connection, and serves as a foundation to plan for future research examine a wider range of teacher assessment practices.
 

Jeffrey S. Beaudry
Bailey 221, University of Southern Maine
Gorham, ME 04038