L.A. UNIFIED CONTRACTS WITH OUTSIDE CONSULTANT TO ISSUE ITS OWN ‘VALUE-ADDED’ RATINGS OF DISTRICT SCHOOLS

Louis Freedberg  - California Watch Blog | http://bit.ly/ct4RvV

Photo by Louis Freedberg - Supplies in a California kindergarten class 

October 15, 2010 | For the past year, the Los Angeles Unified School District has contracted with an outside consulting firm that has been doing its own "value-added" analysis of how effective schools are contributing to student outcomes.

The researchers with the firm, Education Strategy Consulting, said the results they have come up with are substantially different from those published for individual schools by the Los Angeles Times in its pathbreaking but controversial series of reports "Grading the Teachers."

Most of the attention generated by the L.A. Times series has focused on the paper's evaluations of individual teachers and its decision to publish their names alongside their "effectiveness" rating.

The Times also rated entire schools by effectiveness and included a list of the "most effective" schools in the district.  California Watch was unable to tell how different the district's ratings are, however, because the consultants declined to disclose specific ratings until the district releases the results of their value-added study later this fall.

Education Strategy Consulting, based in Charlottesville, Va., did not attempt to rate teachers, but focused entirely on schools. According to Matthias Hild, who did the data analysis, the firm was able to include information about individual students that was absent from the L.A. Times series, including students' racial or ethnic background, whether they were in the district's special education program, and so on. (Hild is an assistant professor at the Darden School of Business at the University of Virginia and a former senior research fellow at NASA's Jet Propulsion Laboratory and the California Institute of Technology.)

Hild said he used a different statistical approach from the one used by the Rand Corporation researcher Richard Buddin, with whom the L.A. Times contracted to do the complex data analysis [PDF] that formed the basis of the Times' series. They used a hierarchical linear model, compared to Buddin's "fixed effects" approach [PDF], which are different statistical approaches that can be used to calculate "value-added" effects of teachers on student outcomes.

Jason Felch, a principal author of the L.A. Times series, told me that he and his colleagues looked closely at which models to use and settled on the "fixed effects" approach as the best one.

"Neither one is right or wrong, and they do involve different assumptions, and we should not be surprised if they came up with somewhat different results," he said. "That said, if researchers are using the same baseline data, whatever model is used should produce somewhat comparable results."

He said that what is most important is that when LAUSD releases its value-added report, it provides all the assumptions and variables included in its analysis so the results can be accurately and fairly compared.

The researchers at Education Strategy Consulting told me they have been working on this project for the past year, but they wanted to be sure they had resolved all the issues before releasing the results. "We wanted to make sure were were getting it right, because this is high-stakes stuff," said Ben Sayeski, a senior partner in the firm. 

They also wanted to deal with other possible problems such as ensuring students' scores were attributed to the right teachers. Some students, for example, may have spent more time with other instructors in English language-learning courses or special education classes than with their home room teacher, so that the latter could not be held responsible for all the gains the student may or may not have made. In other cases, students are in classes taught by a team of teachers. "We had to establish at the end of the day who 'owns' the scores of those students," Sayeski.

Felch agreed the issue of which scores are attributed to which teacher is a "real issue that is something that is tricky to resolve." That is why, he said, the Times averaged teachers' effectiveness over a period of six years. In addition, teachers were given an opportunity to respond to their value-added ratings before they were published, and to identify any factors that may have affected their scores in a particular year, such as whether they were team-teaching or on maternity leave.

Sayeski said that the differences between his firm's approach and that taken by the L.A. Times are fundamental. The debate underscores the challenge for the layperson, or educators who are not statisticians for that matter, to make sense of highly complex statistical techniques which lie at the heart of a major policy debate – which in turn has the potential to directly affect the lives of teachers and their students.

__________________

Louis Freedberg - Senior Reporter and Adviser to California Watch  Priorities: Future of journalism; K-12 and higher education; criminal justice; immigration;  state political reform. Working On: California's unsuccessful efforts to obtain education stimulus funds in competitive grants.

• smf notes: Education Strategy Consulting’s website is not terribly up to date – the most recent newsletter is dated July 2007/biographies of the principals have not been updated since 2009 - and contains no information about clients, partners or sponsors.