Evaluating the Content and Quality of Next Generation Assessments

This study's purpose was to evaluate the quality of four standardized assessments—three new, multi-state assessments (ACT Aspire, PARCC, and Smarter Balanced) and a well-regarded existing state assessment (Massachusetts' state exam [MCAS])—to determine whether they met new criteria developed by the Council of Chief State School Officers (CCSSO) for test quality. Three research questions answered in the study were:

  1. Do the assessments reflect strong content?
  2. Are they rigorous?
  3. What are their strengths and areas for improvement?

To answer these questions, the study employed a new methodology developed by the National Center for the Improvement of Educational Assessment (NCIEA) based on the CCSSO’s 2014 Criteria for Procuring and Evaluating High-Quality Assessments (see Related Resources below). This methodology evaluated the degree to which test items and supporting program documentation (e.g., test blueprints and documents describing the item creation process) measured the critical competencies reflected in college and career ready standards, namely the Common Core. Review panels comprised of vetted practitioners, content experts, and assessment specialists who evaluated grades 5 and 8 ELA/Literacy and math assessments based criteria corresponding to the research questions above. Results from this two-year study show that, overall, PARCC and Smarter Balanced assessments had the strongest matches to the CCSSO criteria, and ACT Aspire and MCAS matched well with regard to quality of their items and the depth of knowledge they assessed. The review panel also provided descriptions of strengths and recommendations for improvement for all four assessments.

 

Content Comments 

"All four tests we evaluated...are, in fact, the kind of tests that many teachers have asked state officials to build for years. Now they have them" (p. 9). This strong statement is supported by the rigorous work from the authors, the principal investigators, and the dozens of reviewers who analyzed PARCC, Smarter Balanced, ACT Aspire, and MCAS assessments for grades 5 and 8. The most important finding is that these new assessments are major improvements over the previous generation of state tests. The study's process is described in detail in this 100+ page report. Despite its length, this report is highly readable. However, readers can also choose to read the Foreward or the Executive Summary for high-level summaries of the study. In the full report, particularly useful to policymakers and test developers (and those who support them) is the Recommendations Section, especially given that the majority of states are planning to use either a customized state assessment or a vendor-developed option. This report is one of two studies published on the four assessments analyzed. The other report, by HumRRO, examines the high school tests (see Related Resources below).