A Times analysis, using data largely ignored by LAUSD, looks at which educators help students learn, and which hold them back.
By Jason Felch, Jason Song and Doug Smith, Los Angeles Times
Over seven years, John Smith's fifth-graders have started out slightly ahead of those just down the hall but by year's end have been far behind. (Irfan Khan, Los Angeles Times)
How teachers can comment to The Times
Chart: Comparing teachers
Khang Nguyen, Los Angeles Times / August 13, 2010
Chart: Assessing teacher performance
Les Dunseith, Los Angeles Times / August 13, 2010
August 14, 2010|12:34 p.m. -- The fifth-graders at Broadous Elementary School come from the same world — the poorest corner of the San Fernando Valley, a Pacoima neighborhood framed by two freeways where some have lost friends to the stray bullets of rival gangs.
Many are the sons and daughters of Latino immigrants who never finished high school, hard-working parents who keep a respectful distance and trust educators to do what's best.
The students study the same lessons. They are often on the same chapter of the same book.
Yet year after year, one fifth-grade class learns far more than the other down the hall. The difference has almost nothing to do with the size of the class, the students or their parents.
It's their teachers.
With Miguel Aguilar, students consistently have made striking gains on state standardized tests, many of them vaulting from the bottom third of students in Los Angeles schools to well above average, according to a Times analysis. John Smith's pupils next door have started out slightly ahead of Aguilar's but by the end of the year have been far behind.
In Los Angeles and across the country, education officials have long known of the often huge disparities among teachers. They've seen the indelible effects, for good and ill, on children. But rather than analyze and address these disparities, they have opted mostly to ignore them.
Most districts act as though one teacher is about as good as another. As a result, the most effective teachers often go unrecognized, the keys to their success rarely studied. Ineffective teachers often face no consequences and get no extra help.
Which teacher a child gets is usually an accident of fate, in which the progress of some students is hindered while others just steps away thrive.
Though the government spends billions of dollars every year on education, relatively little of the money has gone to figuring out which teachers are effective and why.
If you have taught third through fifth grades in the Los Angeles Unified School District, you may be in our database, and we invite you to comment on your value-added score. To get started, please type your first and last names in the box below.
Seeking to shed light on the problem, The Times obtained seven years of math and English test scores from the Los Angeles Unified School District and used the information to estimate the effectiveness of L.A. teachers — something the district could do but has not.
The Times used a statistical approach known as value-added analysis, which rates teachers based on their students' progress on standardized tests from year to year. Each student's performance is compared with his or her own in past years, which largely controls for outside influences often blamed for academic failure: poverty, prior learning and other factors.
Though controversial among teachers and others, the method has been increasingly embraced by education leaders and policymakers across the country, including the Obama administration.
In coming months, The Times will publish a series of articles and a database analyzing individual teachers' effectiveness in the nation's second-largest school district — the first time, experts say, such information has been made public anywhere in the country.
This article examines the performance of more than 6,000 third- through fifth-grade teachers for whom reliable data were available.
» Discuss
Do you think the value-added approach should be used to evaluate teachers? Why or why not?
Among the findings:
• Highly effective teachers routinely propel students from below grade level to advanced in a single year. There is a substantial gap at year's end between students whose teachers were in the top 10% in effectiveness and the bottom 10%. The fortunate students ranked 17 percentile points higher in English and 25 points higher in math.
• Some students landed in the classrooms of the poorest-performing instructors year after year — a potentially devastating setback that the district could have avoided. Over the period analyzed, more than 8,000 students got such a math or English teacher at least twice in a row.
• Contrary to popular belief, the best teachers were not concentrated in schools in the most affluent neighborhoods, nor were the weakest instructors bunched in poor areas. Rather, these teachers were scattered throughout the district. The quality of instruction typically varied far more within a school than between schools.
• Although many parents fixate on picking the right school for their child, it matters far more which teacher the child gets. Teachers had three times as much influence on students' academic development as the school they attend. Yet parents have no access to objective information about individual instructors, and they often have little say in which teacher their child gets.
• Many of the factors commonly assumed to be important to teachers' effectiveness were not. Although teachers are paid more for experience, education and training, none of this had much bearing on whether they improved their students' performance.
Other studies of the district have found that students' race, wealth, English proficiency or previous achievement level played little role in whether their teacher was effective.
"In the past, too often we've just gone with gut instinct and haven't been careful about whether those things are important," said Richard Buddin, a senior economist and education researcher at Rand Corp., who conducted the statistical analysis as an independent consultant for The Times.
Many teachers and union leaders are skeptical of the value-added approach, saying standardized tests are flawed and do not capture the more intangible benefits of good instruction. Some also fear teachers will be fired based on the arcane calculations of statisticians who have never worked in a classroom.
The respected National Academy of Sciences weighed in last October, saying the approach was promising but should not be used in "high stakes" decisions — firing teachers, for instance — without more study.
No one suggests using value-added analysis as the sole measure of a teacher. Many experts recommend that it count for half or less of a teacher's overall evaluation.
And in Los Angeles, the method can be used for only a portion of the district's roughly 14,000 elementary school instructors: California students don't take the test until second grade and teachers must have had enough students for the results to be reliable.
Nevertheless, value-added analysis offers the closest thing available to an objective assessment of teachers. And it might help in resolving the greater mystery of what makes for effective teaching, and whether such skills can be taught.
On visits to the classrooms of more than 50 elementary school teachers in Los Angeles, Times reporters found that the most effective instructors differed widely in style and personality. Perhaps not surprisingly, they shared a tendency to be strict, maintain high standards and encourage critical thinking.
But the surest sign of a teacher's effectiveness was the engagement of his or her students — something that often was obvious from the expressions on their faces.
Study in contrasts
On a spring day at Broadous, all eyes in Room 26 were on the white board.
Miguel Aguilar had brought his fifth-graders to the edge of their seats — with a math problem.
Aguilar, a stocky 33-year-old who grew up in the area, is no showman. Soft-spoken and often stern, he doles out praise sparingly. It only seems to make his students try harder.
"Once in a while we joke around, but they know what my expectations are," he said. "When we open a book, we're focused."
It seems to work: On average, his students started the year in the 34th percentile in math compared with all other district fifth-graders. They finished in the 61st. Those gains, along with strong results in English, made him one of the most effective elementary school teachers in the district.
On this day, Aguilar had invited a student to the board to divide two fractions — a topic on the upcoming state exam. As his classmates compared notes in whispers, the boy wrote out his answer. Aguilar turned to the class.
"Do you agree?" he asked, without hinting at the correct response.
"Yes!" they called back in unison.
"Good," he said softly, allowing a faint smile. "You know this."
John Smith's students in Room 25 were studying fractions too.
Speaking in a slow cadence, he led his class in reciting a problem aloud twice. He then called on a student slouched in the back. The boy got the answer wrong.
"Not so much," Smith said dryly, moving on to another pupil without explanation.
It was only 11a.m., and already it had been a tough day: Three of Smith's students were sitting in the principal's office because of disruptive behavior. All were later transferred permanently to other classrooms.
In an interview days later, Smith acknowledged that he had struggled at times to control his class.
"Not every teacher works with every kid," said Smith, 63, who started teaching in 1996. "Sometimes there are personality conflicts."
On average, Smith's students slide under his instruction, losing 14 percentile points in math during the school year relative to their peers districtwide, The Times found. Overall, he ranked among the least effective of the district's elementary school teachers.
Told of The Times' findings, Smith expressed mild surprise.
"Obviously what I need to do is to look at what I'm doing and take some steps to make sure something changes," he said.
Jobs with security
Public school students are graded and tested all the time. Schools are scored too — California rates them in an annual index.
Not so with teachers.
Nationally, the vast majority who seek tenure get it after a few years on the job, practically ensuring a position for life. After that, pay and job protections depend mostly on seniority, not performance.
Teachers have long been evaluated based on brief, pre-announced visits by principals who offer a confidential and subjective assessment of their skills. How much students are learning is rarely taken into account, and more than 90% of educators receive a passing grade, according to a survey of 12 districts in four states by the New Teacher Project, a New York-based nonprofit.
Almost all sides in the debate over public education agree that the evaluation system is broken. The dispute centers on how to fix it.
Value-added analysis offers a rigorous approach. In essence, a student's past performance on tests is used to project his or her future results. The difference between the prediction and the student's actual performance after a year is the "value" that the teacher added or subtracted.
For example, if a third-grade student ranked in the 60th percentile among all district third-graders, he would be expected to rank similarly in fourth grade. If he fell to the 40th percentile, it would suggest that his teacher had not been very effective, at least for him. If he sprang into the 80th percentile, his teacher would appear to have been highly effective.
Any single student's performance in a given year could be due to other factors — a child's attention could suffer during a divorce, for example. But when the performance of dozens of a teacher's students is averaged — often over several years — the value-added score becomes more reliable, statisticians say.
The approach, pioneered by economists in the 1970s, has only recently gained traction in education.
A small number of states and districts already use value-added scores to determine which teachers should be rewarded and which need help. This summer, one district took a harder line: Washington, D.C., schools Chancellor Michelle Rhee fired 26 teachers based in significant part on their poor value-added scores.
Prompted by federal education grants, California and several other states are now proposing to make value-added a significant component of teacher evaluations. If the money comes through, Los Angeles schools will have to rely on the data for at least 30% of a teacher's evaluation by 2013.
The Times found that the district could have acted far earlier. In the last decade, district researchers have sporadically used value-added analysis to evaluate charter schools and study after-school programs. Administrators balked at using the data to study individual teachers, however, despite encouragement from the district's own experts.
In a 2006 report, for instance, L.A. Unified researchers concluded that the approach was "feasible and valid" and held "great promise" for improving instruction. But district officials did not take action, fearful of picking a fight with the teachers union in the midst of contract negotiations, according to former district officials.
In an interview last week, A.J. Duffy, president of United Teachers Los Angeles, was adamant that value-added should not be used to evaluate teachers, citing concerns about its reliance on test scores and its tendency to encourage "teaching to the test." But Duffy said the data could provide useful feedback.
"I'm not opposed to standardized tests as one means to helping teachers look at what's happening in their classrooms," he said.
Supt. Ramon C. Cortines, who was appointed in late 2008 and plans to retire in the spring, acknowledged that the district could have done more with value-added analysis but was focused on other ways of improving instruction, as well as with staying solvent.
"We have better data than anyone else in the nation — we just don't use it well," he said. "I think it's the next step. It has to be done."
A task force created by the Los Angeles school board to promote teacher effectiveness raised the issue in April, urging the use of value-added scores as one measure of performance.
The task force chairman, Ted Mitchell, said the changes were long overdue.
"I think it's simply a failure of will," said Mitchell, who also heads the State Board of Education.
'Don't be a robot!'
Even at Third Street Elementary in Hancock Park, one of the most well-regarded schools in the district, Karen Caruso stands out for her dedication and professional accomplishments.
A teacher since 1984, she was one of the first in the district to be certified by the National Board for Professional Teaching Standards. In her spare time, she attends professional development workshops and teaches future teachers at UCLA.
She leads her school's teacher reading circle. In her purse last spring, she carried a book called "Strategies for Effective Teaching."
Third Street Principal Suzie Oh described Caruso as one of her most effective teachers.
But seven years of student test scores suggest otherwise.
In the Times analysis, Caruso, who teaches third grade, ranked among the bottom 10% of elementary school teachers in boosting students' test scores. On average, her students started the year at a high level — above the 80th percentile — but by the end had sunk 11 percentile points in math and 5 points in English.
Caruso said she was surprised and disappointed by her results, adding that her students did well on periodic assessments and that parents seemed well-satisfied.
"Ms. Caruso was an amazing teacher," said Rita Gasparetti, whose daughter was in Caruso's class a few years ago. "She really worked with Clara, socially and academically."
Still, Caruso said the numbers were important and, like several other teachers interviewed, wondered why she hadn't been shown such data before by anyone in the district.
"For better or worse," she said, "testing and teacher effectiveness are going to be linked.… If my student test scores show I'm an ineffective teacher, I'd like to know what contributes to it. What do I need to do to bring my average up?"
During recent classes observed by a reporter, Caruso set clear expectations for her students but seemed reluctant to challenge them. In reviewing new vocabulary, for instance, Caruso asked her third-graders to find the sentence where the word "route" appeared in a story.
"Copy it just like it's written," she instructed the class, most of whom started the year advanced for their grade.
"Some teachers have kids use new words in their own sentences," Caruso explained. "I think that's too difficult."
She dismissed the weekly vocabulary quizzes that other teachers give as "old school."
Down the hall from Caruso, fourth-grade teacher Nancy Polacheck was grilling her students on vocabulary, urging them to think hard about what the words meant.
"Don't be a robot!" she said.
Polacheck is another teacher whom Oh identified as one of her top performers. And the Times analysis suggests that the principal is right: Polacheck's students gained 5 percentile points in math after a year in her class, and 4 points in English. That put her in the top 5% of elementary school teachers.
An animated woman with a blond ponytail flowing from the top of her head into her bespectacled eyes, Polacheck has been teaching for 38 years. The desks in her classroom are often set up like seats around a stage, with Polacheck, a self-described "drama queen," in the center.
Her teaching style is a rat-a-tat-tat of questions, the most common of which is "why?"
Polacheck said her colleagues at Third Street think her expectations are too high. She was reluctant to be singled out in any way, repeatedly asking a reporter why she was being interviewed.
"In the past, if I were recognized, I would become an outcast," said Polacheck, who eats her lunch alone in her classroom. "They'd say, 'She's trying to show off.' "
A long process
As the district was appointing the task force and seeking federal dollars, some enterprising principals in L.A. schools began making back-of-the-envelope assessments of teachers using raw test scores.
One clear lesson so far: Finding the least effective teachers is only the first step in a long process.
In 2008, officials at Sunrise Elementary in Boyle Heights, one of 15 campuses under Mayor Antonio Villaraigosa's control, identified several teachers whose student scores were sliding.
The mayor's Partnership for Los Angeles Schools gave them math and literacy coaches, more feedback and the opportunity to watch their more effective peers, said Angela Bass, the group's former superintendent of instruction.
Of three Sunrise teachers who ranked in the bottom 10% districtwide, just one has dramatically improved, according to the Times analysis.
Bass acknowledged that it could take years for foundering instructors to improve, if they do at all. In the meantime, about 20 students a year will continue to sit through their classes.
"It's tragic," Bass said. "It means we've failed them."
Miko Dixon, the principal at Topeka Drive Elementary in Northridge, took a tougher approach. Upon starting the job in 2009, she said, she identified four highly ineffective teachers. One had decided, with the approval of the previous principal, to keep his second-graders as they moved to third grade, ensuring two years of poor teaching in a row.
"It's criminal," Dixon said. "If you get a bad teacher in second and third grade, you're doomed."
Dixon has begun trying to remove the four teachers, a painfully slow process in California. It's far more likely that they will feel the pressure and transfer to another school, she said.
"That's not right," Dixon said, "but it's reality."
Parental trust
For now, parents remain mostly in the dark.
Even the most involved mothers and fathers have little means of judging instructors other than through classroom visits and parking lot chatter. Others don't even have time for that.
Without reliable information, it comes down to trust. Which instructor a child gets is usually decided behind closed doors by principals and teachers, whose criteria vary widely.
"Mi niño, all his teachers are good," said Maura Merino, whose son Valentin Cruz was in the fifth-grade class of John Smith, the low-performing Broadous teacher, last school year. "He never had a problem. Everything is OK."
Merino said it's hard for her to tell the difference between teachers because she doesn't speak English. If she knew her son was assigned to a struggling teacher, "I wouldn't know what to do," she said, speaking in Spanish. "But I would try to get him to the best."
In a conversation after school one day, several Broadous teachers, including Aguilar and Smith, said parents should have the chance to see how teachers measure up.
They "might be more empowered to demand a good teacher," said teacher Eidy Hemmati. And it might keep teachers "on their toes a little bit more," Smith said.
But many others say it would be impossible to accommodate every parent's desire for the best teacher, and publicizing disparities would only turn one educator against the other.
Broadous Principal Stannis Steinbeck refused even to discuss the differences among her instructors, hinting at the tensions that might arise on staff.
"Our teachers think they're all effective," she said.
Times data analyst Sandra Poindexter contributed to this report.
No comments:
Post a Comment