REPORT SEES DISCONNECT BETWEEN NAEP, COMMON STANDARDS + Report

By Erik Robelen | eDwEEK eDUCATION mATTERS BLOG| http://bit.ly/fNfcU9

• [Correction: This blog post should have said that the NAEP math items examined in the Brookings Institution study were, on average, two to three years below the 8th grade math recommended by the common-core standards. The algebra items were at about the 6th grade level. The items from the "numbers" strand were at about the 5th grade level.]

January 11, 2011 8:37 AM | A new analysis from the Brookings Institution raises questions anew about what the advent of common standards—and the development of common assessments to complement them—means for the future of NAEP, often called "America's report card." Overall, the report suggests that the coming common exams mean "a new era is dawning for NAEP," though what that future will look like remains murky.

The analysis by Brookings senior fellow Tom Loveless seeks to match up the common standards in mathematics with publicly released test items from the National Assessment of Educational Progress. In algebra, it found that the NAEP items were, on average, two to three years below the 8th grade math recommended by the common-core standards. The study examines NAEP math items from the 8th grade assessment, coding all publicly released items from the algebra and number strands based on the grade at which the common standards recommends teaching the math assessed by the item. In all, 171 items were available.

In an interview, Loveless told me: "You'll have a state like California participating in the common core, and it will have a state score and another based on NAEP. And because these tests are designed so differently and cover different material, they may or may not say the same thing."

He added: "There's the potential that we're going to confuse a lot of people."

(For an earlier Education Week look at what the common standards could mean for NAEP's future, check out this story by my colleague Sean Cavanagh.)

The report from the Brown Center on Education Policy at Brookings notes that the discrepancy between NAEP and the type of testing expected for the common standards has to do with the varying definitions of an 8th grade math test (or whichever exams for particular subjects and grade levels might be compared). NAEP assesses all of the math that young people have learned through 8th grade; that is, a lot of the content comes from material presumably learned in earlier grades. By contrast, the tests developed to match the common standards are expected to gauge the knowledge and skills learned specifically by the end of each grade level. (Of course, another difference is that NAEP tests only a sample of students, while the common assessments will test virtually all students.)

The report suggests that one way to help minimize the two tests sending contradictory signals about student performance would be to ratchet up the difficulty of NAEP items, bringing the test in closer harmony with the common standards. In the interview, Loveless acknowledged that some NAEP backers may not like this idea, fearing it would disrupt the longitudinal value of NAEP in assessing student achievement over time. However, he said it's not unprecedented, noting that the National Assessment Governing Board essentially did that with the 12th grade NAEP in math for 2005. "With the 12th grade test, they actually broke the trend [line]," he said.

Another option would be to use adaptive testing to help bridge the gap between the two tests, he said.

The report cautions that there are still a lot of questions as the common core evolves into a set of assessments. (Actually, presumably two sets of assessments, given that two separate consortia of states are currently working to develop new exams.)

"Much work remains to bring the common-core standards to life in a real assessment," the report says. "Once that happens, an education process will be needed that informs the public and political leaders on what the NAEP and the common core measure, what they have in common, and what differentiates their results."

In the end, all of this suggests NAEP to be at something of a turning point, the report says.

"A new era is dawning for NAEP. The program has supplied the nation with progress reports on student learning since 1969," it says. "Now, common-core assessments are on the way. Whether the new assessments push NAEP aside, succeed in augmenting the information provided by NAEP, or force a redefinition of NAEP's role in monitoring student learning will be at the top of the NAEP policy agenda in the years ahead."

In the interview, Loveless elaborated: "NAEP was a huge step forward in 1969, and has had to constantly change and modify and reinvent itself along the way to maintain relevance. ... It's going to have to do that again. The rise of the common-core stands is going to force NAEP into additional changes."

Brown Center Report on American Education | Number 12

NAEP and the Common Core Standards

BY Tom Loveless, Senior Fellow, Governance Studies The Brookings Institution | http://bit.ly/hWtaqy

INTRODUCTION
The following is a special advanced release of a Brown Center report on NAEP and the Common Core standards. The full version of the 2010 Brown Center Report on American Education will be published at a later date.

<< DOWNLOAD FULL REPORT

January 11, 2011 — Unlike most countries, the United States does not have national education standards, no single set of expectations for what all American teachers should teach and all American students should learn. It never has. A question that the rest of the world considers foundational to its national school systems—deciding the content of the curriculum—sits in the hands of local authorities. That is because the United States has 50 state school systems. Heterogeneity extends to the deepest levels of schooling. Even students transferring from one teacher to another within the same school may, as a consequence, learn a different curriculum than their former classmates.

So it was an historical event when the Common Core State Standards in mathematics and reading were released in June 2010. Launched by the National Governors Association and the Council of Chief State School Officers, the Common Core Standards project brought together experts in both reading and math to develop a set of standards that would be, in what became a mantra, both “higher and fewer in number” than existing state standards.[1] The standards are voluntary—states choose whether to participate—but for the first time most American students will study a uniform curriculum through at least the eighth grade. A draft of the experts’ work circulated for several months, and, based on input from other experts and the general public, the standards were finalized.[2] In September 2010, two consortia were awarded federal grants totaling $330 million to develop annual assessments aligned with the Common Core standards, and as of December 2010, 43 states and the District of Columbia have signed on to those efforts.[3] The tests are due to be given for the first time in the 2014–2015 school year.[4]

The nation currently monitors the math achievement of fourth, eighth, and twelfth graders on the National Assessment of Educational Progress (NAEP).[5] Since 1990, the main NAEP has assessed mathematics proficiency in five content strands—number properties and operations, algebra, geometry, measurement, and data analysis/statistics/probability.[6] How well does NAEP match up with the Common Core standards in mathematics?

We tackled this question by analyzing NAEP items from the eighth-grade assess­ment. NAEP items are periodically released to the public to give an idea of the content of the test. For the current study, we coded all public release items from the algebra and number strands[7] based on the grade at which the Common Core recommends teaching the mathematics assessed by the item. The 2009 NAEP Framework in Math­ematics calls for number and algebra items to comprise half of the eighth-grade assess­ment.[8] A total of 171 items were available, 98 from the number strand and 73 from algebra.[9] We were unable to code four items (two from each strand) because they assess skills not found in the Common Core.

A precursor to this study can be found in the 2004 Brown Center Report.[10] In that study, we coded the grade level of public release items labeled as “problem solving,” one of NAEP’s process strands (different from the content strands). Only problems involving the application of arithmetic were analyzed. At what grade level are students taught the arithmetic required to answer NAEP problem-solving items? We discov­ered that the mean fourth-grade NAEP item registered at 3.2 and the mean eighth-grade item at 3.7, suggesting that the typical item could be answered using arithmetic taught by the end of third grade. Primarily, this finding stems from NAEP’s reliance on whole number arithmetic in word problems. We found that approximately 70 percent of the eighth-grade items focused on whole numbers. Problems with fractions, decimals, or percents—forms of rational numbers taught after third grade—are not common on NAEP.[11]

The 2004 study used the Singapore Math program as a rubric to code the grade level of items, assigning a value according to the grade and semester in which the arithme­tic of the item was taught. By using the Com­mon Core and evaluating the entire context of items, the current study’s rubric produces higher grade-level estimates for items. Prob­lems involving only simple arithmetic are classified at a higher grade level if they are posed in the context of more sophisticated topics that are taught at a later grade (e.g., co­ordinate plane, equations with two variables). Selected NAEP items are shown [in the full report available for download on this page].

---

[1] “Common Core State Standards Development Work and Feedback Group Announced,” News Release (Washington: National Governors Association Center for Best Practices and the Council of Chief State School Officers, July 1, 2009).
[2] See the Common Core State Standards Initiative web­site on About the Standards, http://www.corestandards.org/about-the-standards .
[3] Catherine Gewertz, “Common-Standards Watch: South Dakota Makes 44,” Curriculum Matters, Education Week, November 29, 2010.
[4] “Beyond the Bubble Tests: The Next Generation of Assessments,” Prepared Remarks of U.S. Secretary of Education Arne Duncan to State Leaders at Achieve’s American Diploma Project (ADP) Leadership Team Meeting, Alexandria, VA, September 2, 2010.
[5] The long-term trend NAEP test assesses students at ages 9, 13, and 17.
[6] See the NCES website on the NAEP Mathematics Framework, http://nces.ed.gov/nationsreportcard/mathematics/whatmeasure.asp .
[7] The number strand refers to the number sense, prop­erties, and operations strand for the 1990–2003 NAEP mathematics framework and the number properties and operations strand in the current mathematics framework.
[8] National Assessment Governing Board, U.S. Department of Education, Mathematics Frameworks for the 2009 National Assessment of Educational Progress (Washington: 2008).
[9] See the NAEP Questions Tool, http://nces.ed.gov/nationsreportcard/itmrlsx/landing.aspx .
[10] Tom Loveless, The 2004 Brown Center Report on American Education: How Well Are American Students Learning? (Washington: The Brookings Institution, 2004), pp. 5–17.
[11] Theresa Smith Neidorf and others, Comparing Math­ematics Content in the National Assessment of Educational Progress (NAEP), Trends in International Mathematics and Science Study (TIMSS), and Program for International Student Assessment (PISA) 2003 Assessments (NCES 2006–029). (U.S. Department of Education, 2006).