School Indicators & Profiles SIG
A service to members of the American
Are Teachers' Classroom Assessment Practices Useful Indicators of Education Reform
Jeffrey S. Beaudry, University of Southern Maine
Introduction, Goals and ObjectivesOver the past ten years, the Maine Educational Assessment (MEA) has focused on creating a statewide assessment system that provides a positive link with teaching and learning. In addition to the on-demand assessment of student achievement, the MEA provides teachers and administrators with Performance Level Guides containing student work and holistic rubrics or scoring guides for all content areas and for all performance levels. Teachers are encouraged to orient students to the types of questions to be asked and the proper format for responses to the questions. In the 1994-1995 MEA survey over 30 percent of teachers in high school English/Language indicated that they not only used these materials for orienting and preparing students for the MEA, but also incorporated the MEA materials as models for their own classroom assessments (Beaudry, 1997). While teachers may have reported that these materials have been useful, questions remain as to whether the use of rubrics and scoring guides is related to the improved students' performance.
A number of questions surround the MEA approaches to changing classroom assessment. For example, what is the reported level of familiarity by Maine's teachers with MEA resources and materials in reading and writing? Do the teachers provide the opportunity for student orientation to the materials? What is the impact of the Performance Level Guides in reading and writing on teachers' classroom assessment practices? Finally, what is the prevalence of students' use of scoring guides to evaluate their own work or the work of others?
Cognitive classroom perspectives (Glaser, 1976; Bransford & Vye,
1989; Resnick & Resnick, 1992) indicate that students' perform at a
higher level when they understand the components of quality for specific
achievement goals. Recommendations by classroom assessment practitioners
like Marzano, Pickering, & McTighe (1993), and Stiggins (1997) argue
strongly for the central role that students can perform in classroom assessment.
Students can be effective participants in processes to develop definitions
of quality work in scoring guides and rubrics and apply these guides in
self-assessment and assessment of other students' work. While there have
been arguments favoring analytic scoring guides over holistic scoring guides
(Marzano & Kendall, 1997; Stiggins, 1997), teachers can begin to understand
appropriate applications with either approach to classroom assessment.
This study examines the relationship of the MEA approach to improving classroom
assessment with students' performance on reading and writing achievement.
Methods & Data Source
Since 1992 the Maine Educational Assessment shifted from selected response, multiple choice questions to open-response, student constructed response questions. The MEA is an annual assessment for all students in grades 4, 8, and 11. Teachers receive a packet of materials before administering this examination, including Performance Level Guides which show examples of student work corresponding to the levels of performance in the scoring guide or rubric. The information for this analysis comes from a survey of teachers in grades 4 (n=380), 8 (n=229), and 11 (n= 122). The survey examines teachers' background, classroom practices, and attitudes towards curriculum and instructional improvement.
To examine the effects of using holistic rubrics in classrooms, Maine
high schools were divided into high- and low-achieving groups according
to whether their 3-year average scaled scores were in the highest and lowest
quartile of MEA reading and writing results. Based on the comparisons of
low- and high-achieving schools three questions were examined: 1) is there
a higher frequency of use of holistic rubrics by teachers to evaluate students'
work, 2) is there a higher frequency of use of holistic rubrics by students
to evaluate other students' work, 3) and what is the proportion of students
at higher performance levels in reading and writing in high-achieving schools
in comparison with low-achieving schools? In addition, correlational analysis
and path analysis will be used to demonstrate the interaction of classroom
assessment variables with resources like teachers' professional background
and student achievement.
Results and Discussion
According to the 1994-1995 survey of teachers, approximately 78 percent of high school English/Language Arts teachers reported that they had used the MEA materials to familiarize students with the test format before administering the test. Just over 30 percent of this group indicated that they incorporated the MEA materials as models for their own classroom assessments. Approximately 9 percent of all teachers reported that the MEA materials were unavailable to them. These findings were consistent across grade levels.
A follow up question on the survey asked whether the teachers themselves used the MEA scoring guides teaching to evaluate students' responses. Roughly 20 percent of all high school English/Language Arts teachers reported that they never used holistic rubrics at all. Fifty-four percent (54 percent) of high school teachers indicated that they used holistic rubrics once or twice a month and an additional 20 percent used them to evaluate students' responses in English/Language Arts on a weekly basis.
How involved are students in the use of holistic rubrics to evaluate their own or other students' responses? Fifty-five percent (55 percent) of Maine's teachers provided opportunities for students to use the MEA holistic rubrics or similar scoring guides to evaluate other students' written work on a monthly basis. One third of the teachers never used this form of evaluation in their classroom.
Based on the 1994-1995 survey it is apparent that 80 percent of Maine's teachers are familiar with and use the MEA reading and writing resources like holistic rubrics and scoring guides. This indicates that a great majority of the English/Language Arts teachers in Maine high schools take the time to orient students to the MEA. Almost one third of the surveyed teachers have found the MEA materials useful enough to be incorporated in their regular instruction. Still, a surprisingly large proportion of English/Language Arts teachers, approximately 22 percent, said they made little or no connection to the MEA for their students the classroom for their students. Similarly, 80 percent of teachers reported using rubrics to evaluate student work. A much smaller proportion of teachers, roughly 45 percent, directly involve their students in using rubrics to evaluate their own work. Overall, a majority of high school teachers found the MEA materials accessible and employed strategies for assessment using the holistic rubrics and scoring guides in their classrooms. The assessment practices modeled by the MEA in reading and writing have apparently been useful for teachers as well as for Maine's students.
In the 1994-1995 MEA survey over 85 percent of high school teachers in high-achieving schools reported that they used holistic rubrics to evaluate student work as compared with 65 percent of teachers in low-achieving schools. Teachers in low-achieving schools tended to use holistic rubrics less frequently, with 14 percent of high school teachers using them on at least a weekly basis as compared with 31 percent of teachers in high achieving schools. The disparity between high- and low-achieving high schools is accentuated when examining the use of holistic rubrics by students. Almost 60 percent of teachers in low-achieving schools never engaged students in classroom assessment by using holistic rubrics to evaluate their own work or the work of other students. In high-achieving schools over 77 percent of the teachers reported that students engaged in the use of holistic rubrics, and 20 percent of the teachers used this approach at least once a week.
What are the performance level profiles of high- and low-achieving high
schools? The results in Table 1 show the frequency distribution of students
across four performance levels in reading and writing-novice, basic, advanced,
and distinguished--is remarkably consistent. Overall, approximately 47 percent
of students in both school groups achieved the basic performance level
in reading and writing. However, the high-achieving high schools have fewer
students in the novice level, approximately 16 percent of their students as compared
with 34-37 percent of students in low-achieving schools. Also, high-achieving
high schools had twice the proportion of students at the advanced level
and four times the proportion of students at the distinguished level in
both reading and writing.
These findings suggest that there may be a positive relationship between
the use of rubrics and student performance on the MEA. In the two questions
involving classroom assessment, teachers and students in high-achieving
schools were more involved in using assessment methods for reading and
writing modeled by the MEA. As suggested by Stiggins (1997) students involved
in their own assessment may be able to make a clearer connection between
classroom instruction and achievement expectations. Analyses of the use
of holistic rubrics for reading and writing results showed virtually the
same results. The shift in achievement is most noticeable at the novice
and advanced levels, an encouraging development for motivating low-achieving
students. By having students as key users of assessment the MEA may be
attaining their goal of fostering higher student achievement through connections
with classroom assessment practices.
The MEA has provided an important view of student achievement by targeting
changes in teachers' classroom assessment practices, which will be an important
component in the development of credible, local accountability systems.
This study of the use of holistic rubrics and students' work provides an
opportunity to examine the strength of this connection, and serves as a
foundation to plan for future research examine a wider range of teacher
Jeffrey S. Beaudry