The promise of comprehensive support for student success

A model tutoring and mentoring program in Toronto

Dionissi Aliprantis

I'm wondering if you could talk a little bit about Pathways to Education intervention and some of your research on it.

Philip Oreopoulos

Pathways to Education is an interesting program that was started in the early 2000s as a grassroots effort in a very disadvantaged neighborhood in Toronto. It was the largest public housing project in Toronto, where you had 5,000, 6,000 households all paying rent geared to income. It was a very high-density public housing project.

And at that time, the high school dropout rate was something like 50 percent. And so more than half of the students going to high school from those projects were not even finishing high school. The community center that was set up in there was given some money from the province to try to do something. And they decided, after looking around, they would construct this program to try to reverse this type of trend and really support students at the high school level, thinking if we could break the cycle of these dropouts, that would potentially change the community.

I looked at the data and the dropout rate went from 50 percent to something like 25 percent or 30 percent. A 20 percent reduction in high school dropout is off the charts, very exciting, and worth doing more of.

Pathways to Education was an interesting program which involved many different angles, with four pillars.

One pillar was coaching. So a student would meet with a student support worker every two weeks, and sit down and discuss what's going on in their lives.

The second pillar was tutoring. Until their grades were sufficiently high, they would be required to attend tutoring—I think at least three or four times a week after school.

The third pillar was group activities. Every three or four weeks, they would be organized to do some fun activity together, with the idea of maybe keeping kids out of trouble and doing more interesting activities like going bowling or pizza night or even going to a baseball game, things like that.

The fourth pillar would be financial incentives to do all of this, so that if someone was going to attend the coaching or tutoring or going to the group activities, then they would get free public transportation. This appealed to a lot of people, that they wouldn't have to pay to go to school or get around. And also that they could accumulate a scholarship that would get up to $4,000 or $5,000 to allow them to go to college.

And then in the middle of all that, there was sort of some contract that both the parent and the child would sign, committing to saying, “If I do this program, I commit to taking it seriously and working with my coach to make everything happen.” This was for the entire years of high school. So, starting in grade nine and every year after that, they would get access to the program. So a lot of support.

Dionissi Aliprantis

I don't know if you have any hypotheses about those pillars.

Philip Oreopoulos

One reason why we were generally confident about the Pathways results, even though the results were a bit noisy, is that there is a very similar program to Pathways at the college level in the US called the ASAP program (Accelerated Study in Associate Programs). The ASAP program looks like Pathways in that it provides multiple layers of support for students entering two-year colleges, often students with disadvantaged backgrounds.

MDRC was the organization that ran these experiments, looking at the impact of the program, and they did it twice. In both experiments, once in New York and another in Ohio, when they offered college students coaching, tutoring, and financial incentives like public transportation and textbook money, they consistently found a very large improvement of graduation rates. And so that maps on with what we were finding in Pathways, which makes us sort of feel that there's a growing story here that these types of intensive multi-pronged approaches to help support students can be very effective.

Are all four pillars of the program necessary?

Philip Oreopoulos

I think some people feel you need a multi-pronged approach to stand a chance of reaching the most people, because in the same program, students and people are different, and where in one case, one person might be helped from coaching, the other person wouldn't, but would be helped with the tutoring. And so if you have this kind of "We're just going to try to support you every way we can," hopefully something's going to stick and then make the difference in the aggregate kind of way. I think some people feel that.

There's just one catch. It's super expensive.

You know, you might be able to justify their cost effectiveness because they have such long-run benefits, but tell that to the administrator who needs to come up with an extra $3,000 a year in order to support these. Right?

So that's a nice segue to this question around trying to think about which of the four pillars matter more. Can you just focus on one of them? Can you get the costs down? That is related to the direction I've been going now with tutoring, and other people doing coaching, to try to understand. Can you have as much impact or almost as much impact from focusing on these efforts almost one at a time, or do you need the collection of services together? I think those are really interesting questions.

Tutoring may be the most effective single pillar of the program

Dionissi Aliprantis

Can you talk to me about tutoring a little bit?

Philip Oreopoulos

Sure. So one of my roles as a co-chair in education at JPAL (The Abdul Latif Jameel Poverty Action Lab) is to put together these review articles of randomized control trials and try to gather conclusions from a collection of work rather than just one experiment at a time. And so a few years ago, my colleagues and I started to investigate—what does the research say about tutoring interventions in general—having come across a few studies in the Chicago Ed Lab and others that seemed to suggest offering more personalized learning through tutoring could be promising.

So we went through a long exercise of collecting the 100 or so randomized control trials in tutoring in the last 40 years or so. And when you put them all together, you find roughly 85 percent of them find significant effects. The median effect is improving test scores by around 40 percent of a standard deviation, which is the equivalent of something like eight months of learning. And that effect is—

Dionissi Aliprantis

Off the charts. I mean, it's massive.

Philip Oreopoulos

It's very large. And the fact that so many of these studies consistently find this impact, sometimes we describe it as, as close to consensus as we're ever going to get in education policy of a program that seems to be very effective. So I am really excited about this because of all the things that I explore in my research, and looking at other people's research, and in teaching education policy, often you get a little bit discouraged because the more convincing the study appears to be, the smaller the effect. Whereas in tutoring, it does seem to be that providing more individualized learning in a way that allows students to progress at their own pace, get immediate feedback, and work until they understand the material,  especially in reading and math, really seems to be something that we should be doing more of.

Tutoring is expensive; using near-peers is a strategy for lowering the cost

Philip Oreopoulos

The catch again is that tutoring is expensive, often costing $3,000 to $4,000 per student per year, in a way that makes it prohibitive.

Dionissi Aliprantis

Well, that brings up the next question I wanted to ask you. I'm curious if you could speak about Matt Kraft’s proposal for scaling up near-peer tutoring. I’ve worked in the field with some 501(c)(3)s, most recently with a program called The Math Movement, where we do a lot of work with near-peer mentoring with middle and high school students. And I've seen firsthand what I consider to be very effective strategies for improving academic outcomes.

I'm thinking about near-peer mentors for two reasons. Number one, it can address the issue of cost. Maybe they're not as effective, but maybe they are, because they're closer to the kids, and they're cool, and all that. And then the other thing is, I don't know if there's any evidence on this, but I actually think these kinds of relationships also have a really big effect on the tutors themselves. So if you're a high schooler, and you're encouraging some middle schooler to apply themselves, to work hard, that they can do it, it kind of changes your perspective when you look back at your own classes and your own educational path. So I'm curious how you think about this issue of scaling up tutors and individualized learning, some of that might be with technology, but have you given much thought to near-peer mentors?

Philip Oreopoulos

Yeah, all of this stuff is all I think about right now. I'm really interested in this stuff. There's surprisingly little research done on peer-to-peer, especially in an experimental setting. So I don't think we really have a good idea. But I think that the fair extrapolation from the tutoring results is that it has potential. We certainly found that the tutoring effects are larger when you have paraprofessionals, basically qualified teachers, who are providing the tutoring. But we also found notable and still significant and encouraging impacts from volunteers providing the tutoring as well. So there does seem to be potential there.

I also think that it is just as important thinking about how the tutoring is provided in a way that makes it easier for someone with less experience, or just an older grade, to offer that tutoring. So, if you just ask any grade 10 students to come in and help grade 6 students in math, it might not be enough setup to expect to have a large impact. But if you give them a set of problems that you want them to go through, or you asked to go through a computer exercise with the child where they get immediate feedback, then it makes it easier for the older student to help the younger student because there's more structure. So I think there is a lot of potential. And I agree that having older kids help younger kids, not only could be very cost effective, but it also could really help the older kid in making them feel more responsible and feel like they are a part of a community.

There are some hints that that would happen. There's this one study in Italy by Michela Carlana and Eliana La Ferrara, who set up a tutoring program during the pandemic where they managed to cooperate with the three large universities in Italy and send out requests for volunteers to give just two or three hours a week of their time to help struggling elementary and secondary students in Italy. The tutoring occurred either virtually or over the phone, whichever was convenient, and this was done as a randomized control trial. In some of the studies, they not only randomized the tutees who were selected to receive the tutoring, but they also randomized the tutors who got the program. And so we not only found that the tutees improved in the subjects that they were working on with the tutors, whether it be math or language, but also the experience and mental health of the tutors also improved from this relationship. And so I think that's really encouraging for this idea of a win-win. I think that there is a lot of exciting potential to develop a culture of tutoring through volunteering at colleges, and I think it could be set up in a way with better and better technology that doesn't involve a lot of time, where you give back an hour or two hours a week of your time, and are able to help students who really can benefit from that in return.

Dionissi Aliprantis

That's a very exciting idea.

Technology is another strategy for lowering the cost of tutoring

Philip Oreopoulos

In addition to using peer to peer as our volunteers to try and to solve the tutoring cost issue, I think technology can also play a role. I think what's really exciting about this area of research right now is, one, there's a lot of people recognizing that personalized learning is really needed right now, especially to address learning loss that's occurred from the pandemic. I think that there's a lot of interest in trying different things, and there's a need for that, as well.

I have been also working on my own version of what might be scalable, thinking about how education technology can help. And one of the projects on my to-do list for a long time has been trying to evaluate Khan Academy, which is this online platform that was one of the first to offer an easy way for any student to go online and get help with the subjects that they're trying to learn in school.

Over time, Khan Academy offers this wide breadth of subjects to take for free—even, like, entire curriculums, especially in math. So all the way going from grade three, up to college calculus. They have these roadmaps, where you can take very short incremental topics, watch a video for three or five minutes, and then take an exercise that's four or seven questions long. And the idea is that it simulates the tutoring experience, and that it provides anyone the opportunity to get a short lesson, and then try to test your knowledge on whether you understand it by doing these short questions. And if you get the question wrong, you get immediate feedback on why it was wrong and what the correct answer is, and then it allows you to kind of try again and again and again until you understand.

That kind of approach is very much in line of how educators think that we should be learning. And as long as you start at a level that's not too easy, not too hard, you can progress incrementally enough that there's no step that's too complicated that you can't get. And that approach avoids the classroom type of setting where whether you get or understand a topic or not, you always move on. So it has the potential, like tutoring does, to try to allow students to progress at their own pace, making sure that they don't lose gaps along the way, so they have the strong foundation as they build up and up.

So that's really exciting. The catch is that students don't just go on up on Khan Academy by themselves and start going through it. Who does this in their free time, right? And so, in order to facilitate more use of something like it, we can bring in partners like teachers themselves, to view something like Khan Academy as a better way to do homework, or something that could be a substitute for the time for something else that they're doing.

The program that we're trying to test within that structure, we call it Coaching with Khan Academy. And we're trying to coach not the students, but the teachers how to use Khan Academy more effectively in the classroom. The recipe that we're encouraging the teachers to follow is to, once a week, introduce a small assignment to the students in class, give them a chance to practice in class. The assignment may or may not line up with what they're doing, and also, if a student is struggling, they might get a different assignment. So not everyone may get the same assignment, depending on where they are and what they need to be working on the most.

And then ideally, the student has the rest of the week to work on the assignment, maybe sometime during school, but maybe as homework, with the important thing to keep working on it until they get all the questions correct. So we think that this approach lines up really well in simulating a tutoring experience, and we hope to even add on to that an additional component where for students, that's not enough for them to be making progress. We would offer them a volunteer tutor to work with them virtually at home on their homework.

So the teacher and the coach work to try to identify which students could use the most extra help, and then, getting back to what we were talking about earlier, taking advantage of volunteers or peer-to-peers and the structure. It's nice because everything is working together; they go home, and the parents are invited to find some time—one or two hours a week—to have a volunteer tutor connect virtually. And then once the virtual connection happens, all the student has to do is share their screen on Khan Academy with the assignment they're working on, so that the tutor has the structure to just help them look over their shoulder and see how they're doing. And that's linked back to what they're doing in the classroom, and so they could potentially go really well together. And know this is all for free or very low cost compared to the one-on-one tutoring model.

Hal Martin and Andrew Zajac contributed to this article.

The opinions expressed in this article are those of the participants and do not necessarily represent the views of the Federal Reserve Bank of Cleveland or the Board of Governors of the Federal Reserve System.