For organizations seeking government and philanthropic funding, reliable proof of progress is an increasingly important currency. “Programs that had good evidence would get the bulk of the money,” says Gordon Berlin, President of MDRC, a nonprofit research organization focusing on education and social policy. In this segment from the Business of Giving, Mr. Berlin discusses about the role of rigorous research evidence in informing how government and philanthropy invest in education and social programs.
The following is a conversation between Gordon Berlin, President of MDRC and Denver Frederick, host of The Business of Giving on AM 970 The Answer in New York City. This interview has been edited for clarity.
Denver: After this country’s “war on poverty” in the 1960s, there was a growing concern that the programs that were being funded were ineffective and selected simply because they sounded good. So, enter MDRC, an organization founded in 1974 to rigorously test and evaluate programs before they became public policy. And here to join us this evening is the President of MDRC, Gordon Berlin.
Good evening, Gordon, and welcome to The Business of Giving.
Gordon: Well, thank you for having me. It’s a pleasure to be here.
Denver: Notwithstanding that brief introduction, tell us in some detail what MDRC does. What are the mission and purpose of the organization?
Gordon: Sure, I’d be happy to. So, MDRC, as you said, was founded in the mid-1970s. Put yourself there; Nixon was President. There was broad disaffection with social programs. We’re beginning to wonder about the “war on poverty” and how effective it really was. And there was a growing sense that what we needed to do was to develop new ideas and test them more rigorously before we made them public policy. That was really the spirit with which MDRC was founded.
“You take a group of people that are eligible. You flip a coin essentially in a lottery-like process. Everybody who gets a “Heads” ends up in a program. Everybody who gets a “Tails” does not. You follow both groups over time, and the only thing that could explain any difference you find is that one group participated in the program.”
Denver: The gold standard of evaluation designs is the use of randomized control trials, a term which is almost synonymous with MDRC. What are randomized control trials? And why are they so useful in determining the effectiveness of a program?
Gordon: Well, one of the biggest problems in evaluating a program is trying to figure out what difference the program made… above and beyond what might have happened to anyone. So take a job training program. Let’s say the program has a 60% placement rate, and people stay on the job for 90 days at that 60% rate or so.
Well, in the middle of the great recession, what do you think of that rate? Is it a good rate or a bad rate? Now, go back to the roaring 1990s, and ask yourself the same question–when the unemployment rate nationally was only 4%. So, the idea here is that you need a counterfactual–a control group, a comparison. That is a group of people that are just like the people that you’re serving in this program– to tell you, over time, what would have happened. The problem in most studies– when they try to identify that group–is that there are a huge amount of statistical assumptions involved. And there are always arguments about it.
And the most elegant, simple, believable approach is random assignment. You take a group of people that are eligible. You flip a coin essentially in a lottery-like process. Everybody who gets a “Heads” ends up in a program. Everybody who gets a “Tails” does not. You follow both groups over time, and the only thing that could explain any difference you find is that one group participated in the program.
Denver: That makes an awful lot of sense. Let’s talk about a few of these programs. You’re working on so many at any one time–it could be as many as a hundred. One I find of real interest though is the earned income tax credit. Now, in the United States, it does seem that we’re focused on poverty as a child problem. And the earned income tax credit– which addresses this– is pretty much targeted at adults with children, particularly single mothers. But there are some ideas about expanding this program to include more single people, and especially low-income men. Tell us about this. Why do you think it might be effective?
Gordon: Sure! So, you’re right. The earned income tax credit is the nation’s largest and, arguably, its most effective anti-poverty program. But almost all of the resources–well over 90% of the resources– go to single parents, usually with children. So, if you think about that for a second, it means that if you and I are working at the same job, side by side, and we’re both making about $15,000 a year, but you have two children and I have none, you’re going to get an extra $6,500 or so a year, and I’m going to get nothing.
If the two of us decided we wanted to get married, then actually what will happen is you would lose some of your benefit. Because now, our income would be $30,000 a year. So, what we really ought to be doing is thinking about a system that overcomes this disincentive to work, to marry, to have children. We want the low-income families to have the same incentive structure that the rest of us have. And the right way to do that would be to make work pay for everybody, including singles, and not to penalize people at the low end when they marry. And that’s essentially what we’ve begun to realize– really where the next frontier is in any poverty policy.
Denver: Where does that stand right now? I know there’s a lot of bipartisan support for it. But it also costs a lot of money.
Gordon: Right! So, estimates run to as much as $30 billion… even more than that. That’s a very big dollar amount. So, what we’ve done is put together a demonstration project–that’s what MDRC is about– it’s called a research and demonstration intermediary organization. We combine in one organization both research capacity and program operations capacity. And what we’re trying to do here is to design and rigorously test new approaches. So we’ve designed a single ZITC that pays up to $2,000 a year to low-income working singles, and we’re testing it– at scale– right here in New York City, with about 6,000 people.
And we’re about to start a test of the same model in Atlanta because we don’t want it to rise or fall just on the New York City labor market. We want to get a more diverse set of experiences there. So, because it’s so expensive, we want to understand: Are there any unintended consequences? You know, the Nixon problem in 1974. And we also want to understand what the response is. Do we get an increase in employment? Do we get an increase in earnings? As incomes rise, are people really better off? What kind of effects does it have on a range of secondary issues– like the likelihood that you might get involved in a criminal activity; the likelihood that you’ll marry; the likelihood that you’ll pay child support and support your children? So, once we understand more about that, we’ll be able to understand what the benefits are of a program like that.
“We want the low-income families to have the same incentive structure that the rest of us have. And the right way to do that would be to make work pay for everybody, including singles, and not to penalize people at the low end when they marry. And that’s essentially what we’ve begun to realize– really where the next frontier is in any poverty policy.”
Denver: Sticking with poverty for a minute, the numbers in this country are not particularly good. We have about 45 million Americans who live in poverty, and about one-third of those are children. What do you see as the key drivers of poverty?
Gordon: Well, there are essentially two key drivers. One of them is low earnings. We have an awful lot of low-wage work. And one of the big problems has been that despite all the economic growth we’ve experienced over the last 40 years or so, earnings in general have been stagnant… or even falling… especially for those with limited education. So one big problem is low income, low earnings–the fact that wages just have not been growing.
A second big problem is family structure. It’s a lot harder for a single-parent family– even one that’s working– to get by than it is for a family with two earners. And there’s also the problem of sharing the responsibilities and demands of taking care of your children.
So, it’s those two features, working together, that are really the big problem. If you think about this problem of low earnings for everybody– that’s been going on for the last 40 years– how did families cope? Well, they essentially did three things: They had fewer children; they postponed marriage; and they sent the second adult in the household into the labor market. Those three strategies helped low income, modern income families keep their income up, even though wages weren’t growing.
But those strategies are all used up now, and I think that’s part of the reason that we’re feeling and sensing and– in this election– seeing so much angst in the broader population. Working people are having a hard time getting by.
Denver: Interesting! Let’s move on to education for a minute. Community colleges enroll about half the undergraduates in this country. They are incredibly important institutions. But the graduation rates are disappointingly low–only about 20% of these students graduate with a degree within three years. So, in 2007, City University of New York launched a program called ASAP (Accelerated Study and Associate Programs). What is that program? And is it working?
Gordon: That’s a great example of a social program in which we’ve begun to build really reliable evidence about its effectiveness. As you said, graduation rates are really low. We’ve tried a lot of things, and they all made small, modest, positive differences. We’ve dealt with providing incentives and financial aid. We’ve dealt with student support services. We’ve done things to try to group students together so they’ll have more support on campus. All those things made small, modest differences, but there were no quantum leaps.
What the City University of New York did was grab hold of those pieces and put them together as a package– in conjunction with the requirement that students attend full time and take the remedial ed. These were mostly students who weren’t quite ready for college; they needed to brush up on their Math and English; they needed to take those courses first. And the result, after 3 years, was a near doubling of graduation rates. No one has produced results quite like that. It’s really very exciting. The Mayor, to his credit, ran with those results and has substantially increased the funding of the City University of New York– in order to expand that program quite substantially. But, of course, the question for the country is, does this thing have legs elsewhere?
Denver: Yes, as a national model.
Gordon: Exactly! So, we’ve been working with three colleges in Ohio, and that test is under way. It’s looking very promising. We believe so far that we’re on track to get similar results. If that does turn out to be the case, then I think we’ve begun to really show the way for community colleges– for those students who can go full time.
There’s some flexibility in how you think about full-time attendance. You can use the summer; you can use the inter-sessions to try to get your hours up. And if you think about it, it makes sense. Because the students are getting the supports they need, and they’re really seeing progress. It’s one thing when you’re taking one course at a time…it’s a long way to graduation. When you’re taking four courses at a time, it starts to add up pretty quickly. At the end of the year, you can see light at the end of the tunnel…
Denver: Sure! You can sense the finish line.
Gordon: Exactly! And that’s what ASAP is doing.
Denver: Let me stick with education for a minute. There’s been a question as to whether small high schools–maybe enrolling about 100 children per grade–are more effective than larger high schools, as evidenced by the number of those students that enroll in college. What does the data show in that regard?
Gordon: So, there was a lot of controversy about this. There’s a certain sense that everybody has right from the start–that of course, if it’s smaller, it will be better. But, it’s also likely to be a little bit more expensive because you won’t have the economies of scale. The Gates Foundation funded New York City and many other cities to experiment with smaller high schools. New York’s approach was especially ambitious and rigorous. They had a very involved process that included the unions, included the Board of Education, included community groups.
The process for creating the schools was a bottom-up process, where they really encouraged local interested groups and teachers and principals that had ideas for small schools, to propose one. They had several criteria. One was that it had to have an academically rigorous curriculum. Another was that it should have a partner. So for example, the School for Law and Social Justice, here in New York, has the largest law firm in New York as its partner.
Those children spent time at that law firm on a court case that develops over several years. We were able to follow students who got into those small schools because they were oversubscribed, as well as kids who didn’t get in because there just weren’t enough seats. And it turned out that attending one of these small, rigorous high schools substantially increased the graduation rates. And almost all of that increase was accounted for by Regents diplomas. The students were much more likely to pass the English Regents Exam– with a score high enough that they could enter the City University of New York without taking remedial education.
Denver: It is so great to hear about things that are working.
Gordon: Yeah. One of the really exciting parts about it, all of the young people that were involved in these high schools were in some of the poorest communities in the Bronx and Brooklyn. Seventy-five percent of them were behind either in Math or English, or both, when they enrolled in the ninth grade. And yet the program still had a nine percentage point increase in graduation rates. And a nearly eight percentage point increase in the likelihood that they would go to college. So for African-American males–very poor, and behind when they started–an increase of 31% in college going. It is really quite dramatic for a high school reform.
Denver: Yeah, that is super!
Gordon: The other thing about those reforms that’s so exciting? It’s in 123 high schools, serving some 40,000 students. This is not a little boutique program. This is a major initiative…
Denver: It’s got some scale–
Gordon: …and it’s really contributed to increasing graduation rates in the city of New York.
Denver: What’s your opinion, Gordon, on the role of financial incentives in public policy? I know there’s some question about the intrinsic versus extrinsic motivation around these incentive structures, but they’re all around us! I just did my taxes, and I was glad to get my mortgage interest written off. I’m sure if we went downstairs and bought some cigarettes, there would be a disincentive around the cost of that pack of cigarettes. Where are financial incentives effective, and what makes them effective?
Gordon: It’s interesting! We’ve generally found that incentive programs are more likely to work than other kinds of programs… like human service programs, in part, because I think they are easy to administer, and the messaging about them is a lot clearer. We ran a project called “Jobs Plus” that provided employment assistance to people living in public housing. It also kept their rents flat when they went to work, so their rents didn’t go up. And we saw a pretty impressive increase in employment and earnings– to the point where we concluded that public housing could be a platform for encouraging self-sufficiency.
We’ve already talked about ASAP; it included some incentive payments. They got a free MetroCard, as long as they showed up for counseling and kept current in everything. And then there are programs like the “Make Work Pay” programs that we tested at MDRC for single moms on welfare. All of them consistently had increases in employment and earnings and income, as well as gains for the children.
We did test a model called the “conditional cash transfer program” that incentivized a wide range of things that are similar to a model in Mexico. It said to low income families: If your children are in school, if they get health check ups, if they do well in school, if you participate in job training, we’ll pay you incentives for each one of those accomplishments that you achieve. Against the control group– not to make a really substantial difference– except there were some positive findings for a group of students who were ready…they were proficient when they entered high school…those students did quite well. There was some decrease in emergency room use, and there was an increase in getting dental exams. But generally, the big behavioral changes we were counting on didn’t really occur, in part because it turns out that in low-income households, their elementary school-aged children attend at a pretty high rate. There wasn’t much room to make a difference there. Low-income families keep up their Medicaid Health Insurance…there wasn’t a lot of room there to make a difference either.
Denver: There are some programs and interventions that you just know are going to work. You run them by some of your buddies, and they say, “Oh, what a great idea!” But when you test them, you find out you were just wrong. Sometimes, not only did they not work, they made the situation worse. Are there any of those that come to mind that you can share with us?
Gordon: Well, I don’t know about any that really made the situation worse, but it’s certainly true that if you look across 40 years of MDRC’s existence, most of the time we’ve learned that something didn’t work, or it only made a really small difference. And that’s one of the realities here. That’s why people became so disaffected with social programs. And I think the whole evidence-based movement is about trying to really turn that around. Let’s build better evidence about what works. Let’s use that evidence as a blueprint to strengthen these programs and move them forward.
Denver: Let me ask you about that. Several years ago, Peter Orszag who was a director of OMB (Office of Management and Budget), said that just about 1% of government spending was backed by any evidence whatsoever, as to whether a program actually worked or not. Now, I hear that there is a quiet revolution taking place in the Obama administration asking for evidence-based outcomes and having funding tied to those outcomes. Where do you stand on all this?
Gordon: Well, we think this is a really important development, and it’s one that Congress has participated in. Essentially what they’ve done is… when the Obama administration came in… and the secret that isn’t really talked about is actually, this began at the Office of Management and Budget during the Bush years…..
They were increasingly interested in trying to turn around the performance of government programs, and OMB began sending clear messages that they were going to put the government’s dollars behind the programs that were most effective. And they began creating these tiered evidence structures. Programs that had good evidence would get the bulk of the money. Programs that were innovative and worth trying would get smaller amounts of money, but only if they built better evidence about them. That has started slowly in something called the Social Innovation Fund. Another program called “Investing in Education” known as I-3, in which they literally ran competitions for these three different tiers–developing new ideas, a middle tier that was about to scale but had not scaled, and programs that had shown some good evidence at scale. They really rationed the dollars, so that those with the best evidence were moving forward.
Since then, Congress used this evidence structure in the Affordable Care Act. There’s $1.5 billion for home-visiting programs, and they’re only funding those programs that had good evidence. The recent passage of Labor Workforce Act also included a new set of evidence requirements, as did the recently passed education bill.
Denver: Is technology — smartphones, the internet of things, wearable tech — changing the nature of your work when you’re evaluating a program?
Gordon: It is. It’s slowly starting to make a difference. It has some amazing advantages if we can all figure out how to do it. It’s sort of a slow transition. In our world, the big question is… we have to go out later. If you’re studying a preschool program, for example, you need to find out how those children did in elementary school… and even beyond. If you’re studying a high school intervention, you need to find out whether they graduated. So we have to go find those students in both groups, find out what happened to them.
I think there’s an increasing sense that we can use smartphones. So for example, everybody has one. Maybe we can pay your bill over the next year or two, or a portion of your bill if you agree regularly to answer some questions for us. This could be especially helpful around very young children, or a child with asthma, where you’re hoping and wanting the parents to do more testing of the child, make sure they’re taking their medications correctly, or doing some reading, or using the black and whites if it’s really an infant. You could imagine the kind of reminder that you’d get every day. And that gets into how you’d use these things around what is now called Behavioral Economics.
There’s a growing field that we’re participating in around little nudges, encouragements… and you can get some pretty interesting changes in behavior when you do that.
Denver: I bet you can! Nothing like real-time data and real-time feedback. Let me get you out on this: in examining the collective body of evidence from all the studies that you have done in education, in poverty, in employment, are there trends that are starting to change our overall approach to tackling these complex social problems?
Gordon: Well, I think what’s happening is that we’re really beginning to learn that there is a real body of evidence, that there are things that work. We’ve gone through several of them here– earned income tax credit, this ASAP model for community colleges, the small high schools. And now the challenge, the next frontier, is how do we do these things at scale and keep those effects?
“Because everybody knows, as you take something to scale, you’re going to lose something. The McDonalds of the world don’t lose very much, when they do it. And maybe we can learn from them, but we need to do a better job of taking some of these things that work, but doing it in a way that allows them to adapt to their local context.”
Denver: You have an incredible website. All the information you’ve been talking about, and much, much more is on it. That website would be?
Denver: Well, Gordon Berlin, the President of MDRC, it was a pleasure having you on the program. Thanks for explaining the work that you do in such a clear and lucid fashion.
Gordon: Thank you! It’s been my pleasure.
*The Business of Giving can be heard every Sunday evening between 6:00 PM and 7:00 PM Eastern on AM 970 The Answer in New York and on iHeartRadio. You can follow us @BizofGiv on Twitter and at Facebook.com/BusinessOfGiving.