Merit Pay for Students Fails to Raise Scores, Study Finds
Eboni McCallum, a 6th grader at Shaw at Garnet-Patterson Middle School in the District of Columbia, reacts to her $34 from the Capital Gains program, which gives rewards for academics and behavior, on October 17, 2008. —Bill O'Leary/The Washington Post/Getty Images-File
By Dakarai I. Aarons | Ed Week | Vol. 29, Issue 29
Published Online: April 16, 2010 -- A multicity experiment to test the effect of paying students for performance succeeded in increasing achievement when the payments were tied to specific behaviors related to learning, such as reading books, but not when the awards depended directly on test scores, new findings show.
“Providing incentives for achievement-test scores has no effect on any form of achievement we can measure,” wrote Harvard University economist Roland G. Fryer in a working paper published last week by the National Bureau of Economic Research.
Still, he concluded, if cash rewards are linked to behavior that can contribute to better student outcomes, “incentives can be a cost-effective strategy to raise achievement even among the poorest students in the lowest-performing schools.”
Mr. Fryer, who is the chief executive officer of Harvard’s Education Innovation Laboratory, conducted the experiment during the 2007-08 and 2008-09 school years in more than 250 urban schools in Chicago, Dallas, the District of Columbia, and New York City. In all, roughly 38,000 students collectively received a total of $6.3 million. The experiment was paid for with both public and private funding.
While the experiment shows providing monetary rewards for good practices can boost achievement in a more cost-effective way than other education reforms, such as lowering class size, Mr. Fryer concluded, they are not a cure-all to education’s ills.
“Incentives alone, like these other reforms, are not powerful enough to close the achievement gap,” he says in the working paper.
Brian A. Jacob, a professor of education policy and economics at the University of Michigan in Ann Arbor, said Mr. Fryer’s experiment is especially valuable for educators and policymakers because most studies looking at extrinsic motivation have been done in the context of a psychology lab, not in schools.
“I think what this does is provide a baseline for other researchers and practitioners to experiment with other financial incentives,” Mr. Jacob said. “What I would take away from this if I were a district superintendent or a policymaker is that it is certainly worth continuing to look at this, although I wouldn’t change a statewide policy on the basis of this study alone.”
The experiments were all school-based randomized trials and they varied from city to city in what students were rewarded for, how often, and how much.
New York City 4th and 7th graders were paid based on their performance on a series of 10 interim exams given to all students in district schools. Students could earn as much as $500 over the course of the school year. On average, 4th graders earned $139.43, and 7th graders collected $231.55.
Ninth graders in the Windy City were paid every five weeks based on grades in five core courses. Those students could earn up to nearly $2,000 per year. Chicago students on average earned $695.61, and the highest achiever collected $1,875.
Chicago schools discontinued the program this year because of a lack of funding for a new cohort of students, but will finish making payments to those who were taking part in the program, said spokeswoman Monique D. Bond.
Middle schoolers in the District of Columbia were paid monetary awards based on measures that included attendance and behavior. The maximum amount was based on the size of the school. On average, students earned $532.85 for the year, with the highest-earning student receiving $1,322.
Dallas 2nd graders were paid $2 for every book up to 20 per semester that they read, as long as they also could pass a quiz testing their comprehension of the book. On average, students received $13.81 out of a maximum of $80.
Representatives for Mr. Fryer said this week that his busy schedule precluded an interview.
Mixed Results
There were minimal effects on test scores for the students paid in New York City, and minimal increases in grade points averages for Chicago students. But there was no overall increase in achievement for students in either city.
The Dallas 2nd graders saw statistically significant increases in reading comprehension, vocabulary, and language, posting test-score gains that continued even after the experiment was over. D.C. students saw “moderate gains” in their reading and math scores after middle schoolers were paid for factors including attendance, behavior, turning in homework, and wearing school uniforms.
The Chicago and Dallas results were both surprising, Mr. Jacob said: High school students generally know what it takes to produce better grades in their classes, and yet the money did not seem to motivate the Chicago 9th graders to do better. The relatively limited scope of the reading intervention in Dallas produced significant gains, just by getting students to read more books, Mr. Jacob said.
The study’s mixed results, Mr. Fryer writes, suggest that students lack the know-how to translate excitement about receiving the financial awards into actions that would boost their achievement.
“Financial Incentives and Student Achievement: Evidence from Randomized Trials” by Roland G. Fryer, Jr. –107 pages/free download
Thursday, Apr. 08, 2010
Should Schools Bribe Kids?
By Amanda Ripley | Time Magazine
In junior high school, one of my classmates had a TV addiction — back before it was normal. This boy — we'll call him Ethan — was an encyclopedia of vacuous content, from The A-Team to Who's the Boss?
Then one day Ethan's mother made him a bold offer. If he could go a full month without watching any TV, she would give him $200. None of us thought he could do it. But Ethan quit TV, just like that. His friends offered to let him cheat at their houses on Friday nights (Miami Vice nights!). Ethan said no.
One month later, Ethan's mom paid him $200. He went out and bought a TV, the biggest one he could find.
Since there have been children, there have been adults trying to get them to cooperate. The Bible repeatedly commands children to heed their parents and proposes that disobedient children be stoned to death or at least have their eyes picked out by ravens. Over the centuries, the stick (or paddle or switch) has lost favor, in most cases, to the carrot. Today the petty bribes — a sticker for using the toilet or a cookie for sitting still in church — start before kids can speak in full sentences.
In recent years, hundreds of schools have made these transactions more businesslike, experimenting with paying kids with cold, hard cash for showing up or getting good grades or, in at least one case, going another day without getting pregnant. (See pictures of kids comparing their paychecks at school.)
I have not met a child who does not admire this trend. But it makes adults profoundly uncomfortable. Teachers complain that we are rewarding kids for doing what they should be doing of their own volition. Psychologists warn that money can actually make kids perform worse by cheapening the act of learning. Parents predict widespread slacking after the incentives go away. And at least one think-tank scholar has denounced the strategy as racist. The debate has become a proxy battle for the larger war over why our kids are not learning at the rate they should be despite decades of reforms and budget increases.
But all this time, there has been only one real question, particularly in America's lowest-performing schools: Does it work?
To find out, a Harvard economist named Roland Fryer Jr. did something education researchers almost never do: he ran a randomized experiment in hundreds of classrooms in multiple cities. He used mostly private money to pay 18,000 kids a total of $6.3 million and brought in a team of researchers to help him analyze the effects. He got death threats, but he carried on. The results, which he shared exclusively with TIME, represent the largest study of financial incentives in the classroom — and one of the more rigorous studies ever on anything in education policy. (See Roland Fryer Jr. in the 2009 TIME 100.)
The experiment ran in four cities: Chicago, Dallas, Washington and New York. Each city had its own unique model of incentives, to see which would work best. Some kids were paid for good test scores, others for not fighting with one another. The results are fascinating and surprising. They remind us that kids, like grownups, are not puppets. They don't always respond the way we expect.
In the city where Fryer expected the most success, the experiment had no effect at all — "as zero as zero gets," as he puts it. In two other cities, the results were promising but in totally different ways. In the last city, something remarkable happened. Kids who got paid all year under a very elegant scheme performed significantly better on their standardized reading tests at the end of the year. Statistically speaking, it was as if those kids had spent three extra months in school, compared with their peers who did not get paid.
"These are substantial effects, as large as many other interventions that people have thought to be successful," says Brian Jacob, a University of Michigan public-policy and economics professor who has studied incentives and who reviewed Fryer's study at TIME's request. If incentives are designed wisely, it appears, payments can indeed boost kids' performance as much as or more than many other reforms you've heard about before — and for a fraction of the cost.
Money is not enough. (It never is.) But for some kids, it may be part of the solution. In the end, we all want our children to grow into self-motivated adults. The question is, How do we help them get there? And is it possible that at least for some kids, the road is paved not with stickers but with $20 bills?
Fryer runs an education-innovation laboratory that has a staff of 17 and an annual budget of about $6 million. His goal is to use the scientific method to figure out how to close the learning gap between America's white and minority kids by the year 2025. When I visit Fryer at his Harvard lab this spring, he hands me an agenda for the day and proudly introduces me to his team. For the next three hours, as we talk about the experiment, Fryer is charming and intense, occasionally lapsing into economist speak and then apologizing for being a "nerd."
But Fryer's fascination with the lives and choices of kids is not entirely academic. He grew up poor in Texas, where he lived with his dad, a copier salesman. When Fryer was 16, his dad was arrested for sexual assault and Fryer had to bail him out of jail.
Meanwhile, Fryer raised himself, and not very well. He got a job at McDonald's and stole from the cash register. He sold marijuana and carried a .357 Magnum for a while. But he was fiercely competitive on the basketball court and the football field, and that's where he excelled, earning a basketball scholarship to the University of Texas at Arlington.
In his first semester of college, Fryer took a calculus class. On his initial exam, he scored 45 out of 100. "My friends started calling me Colt 45," he remembers. The failure enraged him, and his pride kicked in. "I didn't want to be like everyone else from my neighborhood," he says.
Fryer started working hard in school for the first time. He graduated in two and a half years with an economics degree. Then he got his Ph.D. at Penn State University, where he began to use the tools of economics to study the problems of inequality. He joined Harvard's faculty at age 26, a case study in the power of shifting motivations.
At Harvard, Fryer heard about a school in New York City that was trying to incentivize kids on a small scale. The idea appealed to him because, unlike reforms focused on the teacher or the curriculum, it treated kids not as inanimate objects but as human beings who behave in interesting ways. But he had no idea if it would work.
In 2005 he persuaded Gavin Samms, a friend and Harvard colleague, to go to New York City with him to try to sign up some schools for a pilot program. "We didn't know anything about what we were doing," Fryer says. They couldn't afford to stay in New York, so they stayed at a hotel in the Meadowlands — a grim tract of wetlands in New Jersey. Then they drove around to pitch the idea to principals.
One day while they were visiting a school, they got a call from the school system's headquarters, which had originally approved their project. "They said, 'You gotta leave now,' " Samms remembers. " 'You gotta leave the schools.' " Fryer protested, but he lost. "It was just too political," he says. "It was an election year. They'd already gotten letters saying, 'You can't be paying kids.' "
See TIME's Pictures of the Week.
New York City schools chancellor Joel Klein doesn't remember kicking Fryer out, but he concedes that the program was contentious. "When people want to try new and different things in education," Klein says, "it will always stir up controversy."
In January 2007, after the mayoral election had come and gone, Fryer returned to New York — this time with a more audacious plan. He wanted to create a treatment group and a control group, just like a real scientist. And he had a $2 million grant from the Broad Foundation, which had taken an interest in Fryer because of the scientific rigor of his approach.
This time, Fryer wanted to get a random sample of city schools to participate. Which is not as easy as it sounds. At some schools, the principal and teachers opened their arms wide and said, "Sure. We're struggling here. We'll try anything." At others, Fryer had to spend hours pleading with staff who felt kids should learn for the love of learning — not for the cash. "To this day, I can't tell you what will predict one or the other," he says. "I could walk into a completely failing school, with crack vials on the ground outside, and say, 'Hey, I went to a school like this, and I want to help.' And people would just browbeat me about 'the love of learning,' and I would be like, 'But I just stepped on crack vials out there! There are fights in the hallways! We're beyond that.' "
Eventually, Fryer and his team got 143 schools to sign up. About half would be randomly selected as a control group, meaning the kids would not be paid. In the other half, students would earn money for their performance on 10 routine tests given throughout the year.
The summer before the experiment began, a New York Daily News reporter heard about the plan. The story, headlined "It's a Cash Course," quoted an antitesting activist who called the plan "horrendous." One of Fryer's other funders pulled half a million dollars. Fryer got kicked out of the schools again, he says. This time, Klein took him to a Yankees game. A few days later, Fryer was allowed back in the schools. But he started waking up at 3 a.m. to check the newspapers.
The anger was not something Fryer had anticipated. "I totally underestimated how pissed off people would be because of this," he says. "This is exactly the kind of R&D education needs. I never said it was going to solve all education problems. I just thought it deserved to be tested." (See the 10 best college presidents.)
The most damning criticism of Fryer came from psychologists like the University of Rochester's Edward Deci, who has spent his career studying motivation. Deci has found that money — like other tangible rewards — does not work very well to motivate people over the long term, particularly for tasks that involve creativity. In fact, there is a lot of evidence that rewards can have the perverse effect of making people perform worse.
A classic experiment in support of this hypothesis took place at a nursery school at Stanford University in the early 1970s. There, researchers divided 51 toddlers into groups. All the kids were asked to draw a picture with markers. But one group was told in advance that they would get a special reward — a certificate with a gold star and a red ribbon — in exchange for their work. The kids did the drawings, and the ones in the treatment group got their certificates.
A few weeks later, the researchers observed the children through a one-way mirror on a normal school day. They found that the kids who had received the award spent half as much time drawing for fun as those who had not been rewarded. The reward, it seemed, diminished the act of drawing. So instead of giving kids gold stars, Deci says, we should teach them to derive intrinsic pleasure from the task itself. "What we really want is for people to value the activity of learning," he says. People of all ages perform better and work harder if they are actually enjoying the work — not just the reward that comes later.
In principle, Fryer agrees. "Kids should learn for the love of learning," he says. "But they're not. So what shall we do?" Most teenagers do not look at their math homework the way toddlers look at a blank piece of paper. It would be wonderful if they did. Maybe one day we will all approach our jobs that way. But until then, most adults work primarily for money, and in a curious way, we seem to be holding kids to a higher standard than we hold ourselves.
In the fall of 2007, the New York City experiment began. Fourth-graders could earn a maximum of $25 per test, and seventh-graders could earn up to $50 per test. To participate, kids had to get their parents' permission — and 82% of them did. Most of them also opened savings accounts so the money could be directly deposited into them. Meanwhile, Fryer and his team found other testing grounds. In Chicago, Fryer worked with schools chief Arne Duncan, now President Obama's Education Secretary, to design a program to reward ninth-graders for good grades. Over beer and pizza in a South Side bowling alley, they sketched out a plan to pay kids $50 for each A, $35 for a B and $20 for a C, up to $2,000 a year. But half of their earnings would be set aside in an account, to be redeemed only upon high school graduation.
In Washington, middle schoolers would be paid for a portfolio of five different metrics, including attendance and good behavior. If they hit perfect marks in every category, they could make $100 every two weeks. Schools in Dallas got the simplest scheme and the one targeting the youngest children: every time second-graders read a book and successfully completed a computerized quiz about it, they earned $2. Straightforward — and cheap. The average earning would turn out to be about $14 (for seven books read) per year.
See the best pictures of 2009.
See 25 people who mattered in 2009.
The early feedback was promising. Principals were lobbying to get their schools switched out of the control group and into the treatment group. Parents began using the paychecks as progress reports, contacting teachers to find out why their kids' checks had gone up or down. In Chicago, Duncan discovered that the program affected kids in ways he'd never expected. "I remember going to schools and seeing how excited the kids were when they got their checks. They were like pep rallies — but around academic success!" he says. Fryer appeared on The Colbert Report and CNN to talk about the experiment, and that's about when the death threats started. All the while, Fryer refused to speculate about what the data would reveal. He was not all that interested in whether the kids raised their grades or turned in their homework. Grades are subjective. The more objective measure would come at the end of the year, when the students took their standardized tests. Would they improve more than the kids who were not getting paid? Or would they, as the psychologists predicted, actually do worse? "If it doesn't work, we're going to stop and start doing something else," says Washington schools chancellor Michelle Rhee. "But if it does work, it should drive where we put our money." (Watch Michelle Rhee talk about D.C. schools.)
The results began to trickle into the lab last summer. In New York City, the $1.5 million paid to 8,320 kids for good test scores did not work — at least not in any way that's easy to measure. In Chicago, under a different model, the kids who earned money for grades attended class more often and got better grades, two major accomplishments. Those students did not, however, do better on their standardized tests at the end of the year.
In Washington, the kids did better on standardized reading tests. Getting paid on a routine basis for a series of small accomplishments, including attendance and behavior, seemed to lead to more learning for those kids. And in Dallas, the experiment produced the most dramatic gains of all. Paying second-graders to read books significantly boosted their reading-comprehension scores on standardized tests at the end of the year — and those kids seemed to continue to do better the next year, even after the rewards stopped.
The kids had much in common. In all four cities, a majority were African American or Hispanic and from low-income families. So why did the results vary so dramatically from city to city?
One clue came out of the interviews Fryer's team conducted with students in New York City. The students were universally excited about the money, and they wanted to earn more. They just didn't seem to know how. When researchers asked them how they could raise their scores, the kids mentioned test-taking strategies like reading the questions more carefully. But they didn't talk about the substantive work that leads to learning. "No one said they were going to stay after class and talk to the teacher," Fryer says. "Not one."
We tend to assume that kids (and adults) know how to achieve success. If they don't get there, it's for lack of effort — or talent. Sometimes that's true. But a lot of the time, people are just flying blind. John List, an economist at the University of Chicago, has noticed the disconnect in his own education experiments. He explains the problem to me this way: "I could ask you to solve a third-order linear partial differential equation," he says. "A what?" I ask. "A third-order linear partial differential equation," he says. "I could offer you a million dollars to solve it. And you can't do it." (He's right. I can't.) For some kids, doing better on a geometry test is like solving a third-order linear partial differential equation, no matter the incentive.
Similarly, in Chicago, kids were paid for grades — a result they could not always control. There, the findings were mixed. Kids who got paid did indeed get better grades, and they also attended class more — a week and a half more over the school year. That is a big deal, since nearly half of Chicago's high school kids drop out before they graduate and the kids who skip school and fail courses as freshmen tend to be the ones who drop out. We won't know until 2012 if the experiment lowered the dropout rate, but we do know that the rewards did not raise standardized-test scores.
So what happens if we pay kids to do tasks they know how to do? In Dallas, paying kids to read books — something almost all of them can do — made a big difference. In fact, the experiment had as big or bigger an effect on learning as many other reforms that have been tested, like lowering class size or enrolling kids in Head Start early-education programs (both of which cost thousands of dollars more per student). And the experiment also boosted kids' grades. "If you pay a kid to read books, their grades go up higher than if you actually pay a kid for grades, like we did in Chicago," Fryer says. "Isn't that cool?"
It may also help that the kids in Dallas were the youngest in the experiment, making them more receptive to reforms. It's hard to know for sure. Another caveat is that the Dallas model worked differently on different kids. Most (including Hispanic kids and poor kids) did better when they were being paid. But the ones who spoke very little English and took their standardized tests in Spanish did not benefit from the incentives, a mystery that Fryer addresses at some length in his study but cannot entirely explain. (See pictures of Detroit schoolkids sharing their dreams for the future.)
Meanwhile, in Washington, each school got to choose three of the payment metrics, and some of the elements ended up being outcomes like test scores. But the students were also paid on the basis of attendance and behavior — two actions that are under their direct control. Under this hybrid model, the kids who got paid did better on their standardized reading tests. Because of the small size of the school system, the Washington sample was less well balanced than those in the other cities. But its results contain one remarkable finding: the kids who were helped the most by the experiment were the ones who are normally among the hardest to reach. "The typical reform helps girls more than it helps boys," Fryer says. "[This] is the opposite. In D.C., all the results are being driven by the boys. That's fascinating."
When I talked with Washington students, teachers and principals about the experiment, they appeared to have very low expectations for its long-term impact. Many of them, speaking from experience, seemed to think that nothing as simple as money could reach a certain hard core of kids. "The children we had challenges with before, we still have challenges with," says Vealetta Moore-Parker, a guidance counselor who runs the incentives program at Burroughs Education Campus.
Nevertheless, according to Fryer's results, kids with a history of serious behavioral problems saw the biggest gains in test scores overall. Their reading scores shot up 0.4 standard deviations, which is roughly the equivalent of five additional months of schooling.
Kids may respond better to rewards for specific actions because there is less risk of failure. They can control their attendance; they cannot necessarily control their test scores. The key, then, may be to teach kids to control more overall — to encourage them to act as if they can indeed control everything, and reward that effort above and beyond the actual outcome.
The Knowledge Is Power Program (KIPP), one of the most successful charter-school networks in the U.S., has been doling out financial incentives for 15 years, using a model that happens to align perfectly with the results of Fryer's study. KIPP students get paid for actions they can control — getting to school on time, participating in class and having a positive attitude — with "money" they can redeem for supplies at the school store. Over the years, KIPP leaders, who now run 82 schools nationwide, have learned a lot about which rewards work and which do not. They have found that speed matters, for example. Recognition, like punishment, works best if it happens quickly. So KIPP schools pay their kids every week. (Interestingly, the two places Fryer's experiment worked best were the ones where kids got feedback fast — through biweekly paychecks in Washington and through passing computerized quizzes in Dallas.)
Just like grownups, kids need different kinds of incentives to get through the day, some highbrow and some low, some short-term, some longer-term. And money and other external rewards can be a gateway to more substantive motivators. KIPP fifth-graders get a lot of prizes like pencils; high school kids can earn freedoms — like the privilege of listening to their iPods at lunch. "Our ultimate goal is to get kids to be intrinsically motivated," says Joshua Zoia, who founded the KIPP Academy in Lynn, Mass. "But we have to get kids hooked in. We have to meet them where they are."
When Fryer briefed Rhee, the Washington schools chancellor, about the results, she was shocked — happily so. "It is just so hard to show impact in education," Rhee says, citing past experiments that showed no payoff despite enormous effort. "We don't see results like this for a lot of other things we're doing," she says. So she went to the Washington city council to ask for more money to keep paying kids — and to keep studying what happens. "If next year's data show something different, so be it," Rhee says. "We'll take it year by year." The program has wound down in Chicago, Dallas and New York City, although schools in all three places continue to experiment with incentives.
Fryer believes there's more good research to be done on incentives. But he doesn't think incentives alone can fix our schools; he is increasingly convinced that the answer will involve a combination of reforms and that the interaction among those reforms will matter more than any single change in isolation. And whatever we do, he says, we have to test it first — and fearlessly. "One thing we cannot do is, we cannot restrict ourselves to a set of solutions that make adults comfortable."
Chyna is an eighth-grader at the Takoma Education Campus in Washington. Chyna likes to refer to herself in the third person, and when you ask her a personal question, she looks you dead in the eye, asks, "Honestly?" and waits for you to reply before giving you her answer.
Chyna wants to be a lawyer or a radio personality when she grows up. But last year she had a hard time. She and a friend got into a fight with another girl at school. "We basically jumped her," Chyna admits. The police came, and Chyna found herself in a juvenile-detention center, waiting for her mom to pick her up.
This year is going better. When I meet her, she has just received her regular paycheck. She earned $95, her highest check yet. She squeals with happiness and hugs her girlfriends. When I ask her how she did it, she says, "I tried my hardest." She adds, "I tried to wear my uniform, because I knew I wanted some money because my birthday is next week." She has saved her past four paychecks for this reason. The money, she says, gives her just enough incentive to hold her tongue.
"For the most part, I'm still Chyna," she says. "But once in a while I just snatch it back, 'cause I know that paycheck is coming." Then I ask her about the psychologists' argument that she should work hard for the love of learning, not for short-term rewards. "Honestly?" she asks. "Yes, honestly," I say. She looks me dead in the eye. "We're kids. Let's be realistic."
No comments:
Post a Comment