21st CCLC: State Education Agency

Tips From Your Peers: Michigan

  1. How do you come up with performance standards for your state? How do you communicate these standards to grantees?

    Michigan's initial focus was on assisting grantees to implement programs successfully, meet requirements such as licensing, and provide support in best practices for youth programs and academic enrichment. At the fall 2005 kickoff, programs were notified of the federal performance targets and provided with feedback as to their performance in relation to those targets. This year, the state evaluators are assessing the performance of participating students on improvements in grades, state test scores, local test scores, and classroom behavior as well as identifying reasonable state performance standards based on the existing data and local context. These standards will be communicated through e-mail and site visit contacts, at state meetings with grantees, on the Michigan 21st Century Community Learning Center website, and in comparison with individualized data in the annual report form.

  2. What are the specific instruments you use?

    We use a logic-model-type report format online.

  3. How do grantees know how they are performing?

    Grantees receive performance feedback in multiple ways:

    • State education agency (SEA) staff make two site visits to grantees during their five-year grant period. Each site visit is documented and recommendations are identified and discussed with the grantee, who also receives a copy of the site visit report.
    • Each grantee is required to conduct a local evaluation and receives feedback through their local evaluation process.
    • The state evaluators coordinate collection of all enrollment, attendance, survey, and school outcome data. These data are analyzed and populated into the Annual Report Form so grantees receive standard individualized information at the grant and site levels. After reviewing the findings for their program, grantees provide feedback and interpretation of the findings that contribute to state evaluation data and use the information for program improvement and revision of objectives.
    • A fall kickoff is held at the beginning of each school year. During the kickoff, the state evaluators present data to show how Michigan grantees are performing compared to the federal (and in the future, state) targets and the degree to which grantees vary in meeting performance targets across the state.
    • Beginning this year, sites will be required to conduct a self-assessment of their program quality using the Youth Program Quality Assessment (YPQA).
  4. What methods and instruments do you use to evaluate your state program?
    • A Web-based enrollment and attendance data collection system (EZReports), which collects information at the individual level on activity types, time conducted, associated staff and service provider characteristics, student and parent demographics, and site and grantee federal-reporting information
    • End-of-year surveys (developed for the state evaluation) of students and parents that assess program quality and immediate outcomes, such as school engagement, learning associated with the program, and program involvement
    • Federal teacher surveys
    • School outcomes data at the individual level, including grades, state standardized test scores, local standardized test scores, school attendance, and suspensions and expulsions
    • YPQA, an activity- and site-level instrument that assesses supports and opportunities provided to participants conducted by site personnel, objective observers, and/or SEA staff
    • The Michigan 21st CCLC Annual Report Form, in which grantees document program characteristics and issues faced as well as interpret data specific to their program that is provided from the state evaluation
    • A substudy assessing links and the degree of alignment between the 21st CCLC program and school-day academic instruction; includes interviews with site personnel, school administrators, and teachers as well as observations of instructional practices in the afterschool program and in school classes

    Please identify your key evaluation questions.

    The Michigan 21st CCLC evaluation is guided by a logic model whose key evaluation questions are the following:

    • Are high-quality programs more successful in attracting and keeping students than low-quality programs?
    • Do students who receive a higher dosage of 21st CCLC programming, particularly high-quality programming, show improved short-term outcomes (e.g., school engagement, homework completion, classroom behavior, perceived learning) than students with a lower dosage?
    • Do students who show improved shortterm outcomes demonstrate longerterm improvement in academic and behavioral outcomes?
  5. How do you use data for program improvement?
    • At the state level, SEA staff use the data from the Annual Report Form to identify training needs, challenges faced by multiple grantees, and successful strategies that may be recommended to other grantees as well as to provide individual guidance to grantees. SEA staff rate the YPQA as part of their site visits and use the self-assessments from grantees to give detailed recommendations for improving programming quality. SEA staff monitor site enrollment, attendance, and retention through their access to the Web-based reporting system, which provides reports of site statistics in real time. All data have been used by SEA staff to update and revise the grant-application process to ensure that grantees are informed and educated about programming needs prior to initiating programming.
    • At the grantee and site levels, grantees receive all data they collect for state evaluation purposes in standardized raw form to enable their local evaluators to analyze the data for local questions and according to local needs. Because relationships and contracts between local evaluators and grantees vary, the Annual Report Form provides standardized data analyses to the grantees so they can use this information in program improvement.
  6. How widely do you disseminate your evaluation results and/or report?

    Reports are disseminated through the state department of education, the legislature, and the governor; placed on the 21st CCLC website; presented to grantees; and presented at national and international conferences on adolescence, child development, evaluation, and youth development.