by Clive R. Belfield

Voucher programs are intended to raise the academic achievement of students, but, unfortunately, so far the evidence suggests that Cleveland’s voucher students perform no better than their counterparts in public schools. 

Vouchers for K–12 education continue to attract interest, offering the promise of greater parental choice, enhanced school efficiency, and improved educational outcomes for students. The first formal voucher program was established in Milwaukee in 1990, and its practical success was followed by programs in Cleveland, Florida, Colorado, and Washington, D.C. With these programs has come sustained inquiry into vouchers and their anticipated effects. Here we review the evidence that has accumulated so far about voucher programs in general, and then we take a closer look at the Cleveland Scholarship Program (CSP) in particular, including its effects on student test scores.

Ohio plans on introducing a CSP-like program in 2006 at the lowest-performing schools across the state. Evidence on the efficacy of the CSP program is therefore critical, both for the direct development of policy in Ohio and for voucher initiatives in other states.

Research to Date on Vouchers

Voucher programs are expected to raise students’ academic achievement, and the most high-profile research on vouchers has looked at whether they do. The evidence shows, at best, moderate advantages for voucher participants.

For the Milwaukee Program, Rouse (1998) found small but positive effectsize differences for math but no effect for reading. However, the data were from the first five years of the program; religious schools were not participating, and the voucher students were concentrated in a few schools. For the Florida program, Figlio and Rouse (2005) found modest results from data on over 180,000 students. Voucher users in initially low-performing schools do post higher test scores, but much of this is attributable either to student characteristics (that is, the users were self-selected, high-ability students) or teaching to the high-stakes test (suggesting that the achievement gains may not have been genuine cognitive gains).

Randomized field trials for vouchers in three cities found small test score gains after three years (Howell and Peterson, 2002). These treatment effects were primarily for African Americans in one setting, with no evidence of cumulative gains for those who used the voucher for the longest periods. Finally, it is worth noting that new evidence from expanded public school choice points to the same conclusion, with few achievement gains from placement in a choice school (Cullen et al., 2005).

Other studies have investigated broader questions, such as: How do parents choose schools when vouchers are available to them? In what ways are private schools accessed by voucher students better than public schools? How do vouchers influence public finances?

In looking at school-choice decisions by parents, it is clear that many affluent families already have choices; attention therefore focuses on how voucher programs might open up choices for those families that are constrained. Thus far, all voucher programs have been targeted to low-income families or to districts with low-performing schools. Clearly, vouchers expand the choices of low-income families.

However, several mediating factors mean that voucher programs are less effective in promoting choice among low-income families than is implicit in a simple reading of the program eligibility rules. First, religion pervades family preferences of schools (Campbell et al., 2005). Certainly, parents value high test scores, but preferences are varied, and many families choose their neighborhood school.

Second, race has a strong influence on people’s choice of school. The relationship is complicated by different patterns across Hispanic and African American children and by the fact that public schools show strong patterns of racial segregation. However, vouchers consistently lead to greater student segregation (even in a highly minority public school system such as Washington, D.C.).

Third, school choice is an action—it requires that parents actually make use of the voucher to change schools. Usage rates are much lower—perhaps by one third—than offer rates (Howell and Peterson, 2002, table 2-2). Moreover, even within low-income groups, the children most likely to use the voucher are those who are the most motivated. Also, a nontrivial proportion of those who receive vouchers are already in private school. Thus, it is an overstatementto declare that vouchers significantly help the disadvantaged.

Research has also investigated the supply of private schooling. Without supply, family preferences become meaningless—and if private schools do not operate in ways distinct from public schools, there will be no advantage to students from choosing them.

Again, several consistent findings emerge. Most participating schools are religious (across various faiths); secular schools have a small market share. The supply of new schools appears reasonably elastic: For example, almost one-half of schools participating in the Milwaukee program were founded after it was introduced. And voucher-student enrollees are increasingly a majority within their school: by 2001, 40 percent of the schools participating in the Milwaukee program had more than 80 percent of their students claiming vouchers. So voucher programs attract new private schools rather than extra places in existing private schools.

However, research on the inputs and technologies that private schools use (beyond selecting their students) and which are more efficient has yielded very little: Economists are still no further ahead in identifying the separate benefits of ownership, innovation, and technical efficiency; that is, they are no further ahead in knowing which inputs work best (Hanushek, 2004). After controlling for student characteristics, most research finds only very modest advantages to private school.

Finally, greater competition is likely to improve schools’ performance (Belfield and Levin, 2002), but the competitive pressures exerted by small-scale voucher programs are likely to be very modest.

Other research has examined how voucher programs might impinge on the existing public school system and its local financing. Because schooling is financed locally, individual school choices will feed back into house prices, district spending, and so school quality.

Thomas Nechyba found several novel results in a study he conducted in 2003, in which he modeled the introduction of large-scale voucher programs and examined the effects on public schooling and finance. First, competition for high-ability students would increase; these students would pay lower tuitions, both as a result of the voucher and as a consequence of schools’ greater eagerness to enroll them. Second, public schools would engage in more ability tracking to prevent high-ability students from switching to private school. Both factors suggest further educational inequalities, with greater rewards (and resources) for high-ability students. Third, public school quality is most likely to decline in middle or high income school districts.

Importantly, opinions about vouchers are likely to be driven by perceived effects on property values than on educational outcomes; homeowners may be wary of education reforms that may raise uncertainty as to the value of their home.

The Cleveland Scholarship Program

The Cleveland Scholarship Program ooperates in the Cleveland Municipal Schools District (CMSD), which has 75,000 students across 130 schools (2005–2006). Eligible schools are nonpublic chartered schools located within the CMSD and approved by the state superintendent. Surrounding public school districts are eligible to apply. Currently 5,734 students participate.

Initially, eligible children had to be entering a grade between kindergarten and eighth grade, reside within the CMSD, and require no segregated special education. Low-income families were given preference, with those below 200 percent of the poverty level provided with 90 percent of tuition or $2,250, whichever was lower. Families above 200 percent of the poverty level were provided with 75 percent of tuition or $1,875, whichever was lower. About one-quarter of students came from the latter group. In the first year of operation (1996), enrollment was 1,996, with total funding of $5 million (with transport paid by the CMSD).

In comparison, per-pupil expenditure in the CMSD in 1996 was $6,675 (excluding transport). In 2003–04, the scholarships were made available for ninth grade and beyond. Funding was increased to $3,000 for grades kindergarten through eight and set at $2,700 for higher grades. Where voucher applications exceeded available placements, a lottery system was used.

Despite its usefulness for informing future voucher reforms, the CSP has been the subject of little academic inquiry. The program is sufficiently large to allow for samples of students according to voucher status and religious schooling, and with recent data, it offers an up-to-date evaluation of vouchers in light of recent school-choice reforms. Given the duration of the program, it is possible to examine the question of whether persistence in the program yields higher rewards. Importantly, the CSP voucher is relatively ungenerous: If effects can be found for this program, it is likely that achievement gains would be even larger for programs which are more generous.

Students were classified according to voucher status so that effects could be identified: Users (offered a voucher and used it to attend private school); nonusers (offered a voucher but did not use it or stopped using it); applicant rejects (applied for a voucher but not offered one); eligibles (eligible to receive a voucher but did not apply for one); and ineligibles (did not apply and were not eligible according to the program rules).

When studying the achievement data in places where voucher programs are in place, various biases could make identifying the effects that are due just to vouchers more difficult. Three biases are particularly important, but their effects are (probably) offsetting for this program. Applicant bias occurs where only those who apply for the voucher are likely to benefit from it. This will bias gains toward users, because applicants are typically motivated families. Eligibility bias occurs where those who are eligible differ both from those who apply and (separately) from those who do not apply. This will bias gains away from users, because CSP eligibility is conditional on low family income. (It is possible with these data to control for eligibility). Usage bias occurs as those who use the voucher differ from those who do not use it, conditional on application and eligibility. This will bias gains in favor of users relative to nonusers because usage is positively correlated with ability and family resources. In addition, data collection inevitably generates some response bias as survey attrition rates are higher for nonusers.

Evaluating the CSP

Since 1996, the CSP has been evaluated by the Center for Education Evaluation at Indiana University (see Metcalf et al., 2003). The center began collecting data for students entering kindergarten in1997 and has continued until they were in the sixth grade in 2003. (No substantive changes in program design occurred during this period, but the legal status of the program was resolved only in 2002).

The dataset includes over 4,000 students who attend over 100 separate schools. In terms of voucher status, the dataset is composed of users (23 percent), nonusers (10 percent), applicant rejects (16 percent), and a public school comparison group (51 percent). The dataset has three advantages: It is longitudinal (including achievement measures); it includes students from multiple comparison groups; and it is a large sample spread across many different schools. However, because the data do not allow us to distinguish eligibles from ineligibles, both of these are included in the public school comparison group. Also, the sample sizes are considerably lower with attrition and missing responses.

Evidence on the CSP comports with extant research. Specifically, most students chose religious schools, and high (and growing) proportions of voucher applicants and users had previously been enrolled in private school. Student characteristics across voucher status are similar to those in other studies. African American students and low-income students are much less likely to use the voucher, conditional on being offered it.

To identify the achievement gains from the voucher program, we estimate a series of education production functions. Test scores in second and fourth grade are regressed against a set of student and school characteristics, including prior-year test scores. The Terra Nova test is applied: widely used across U.S. schools, it measures basic skills across a range of subjects. It is norm-referenced to allow comparison between students. The impacts of voucher status on achievement are reported in table 1. These impacts are effect-size gains relative to the public school comparison group in reading, math, and language. (“Effect sizes” are a statistical method used to assess the magnitude of an effect. In this and similar studies, an effect size is the gain in achievement divided by the standard deviation of achievement; a gain of 0.1 is therefore one-tenth of a standard deviation).

TABLE 1: EFFECT-SIZE TEST SCORE GAINS OVER PUBLIC SCHOOL STUDENTS
		Reading	Math	Language
Second grade
	Voucher user	-0.060	-0.092**	0.097**
	Nonuser	-0.019	-0.021	0.162*
	Rejected application	0.083	0.026	0.136**
Number of students		1,733	1,786	1,736
Fourth grade
	Voucher user	0.042	-0.113***	0.038
	Nonuser	-0.065	0.044	0.076
	Rejected applicant	-0.030	-0.055	-0.063
Number of students		2,089	2,102	2,085
NOTES: 1. Terra Nova test. OLS estimation. Effect sizes control for: subject-specific first/third grade scores; African American; Hispanic; female; free lunch (a proxy for income); unsubsidized lunch; class size; years of teacher experience. 2. *significant at 10 percent; **significant at 5 percent; ***significant at 1 percent.

The top panel of table 1 shows mixed effects according to voucher status. For reading, there are no differences across the four groups in second grade. For math, voucher users report the lowest scores. In contrast, for language, the public school group does considerably worse than the other three groups; however, voucher users gain the least—both nonusers and rejected applicants show larger advantages. Given the biases that might lead to statistically significant gains for voucher users, we cannot find evidence that voucher students outperform relevant comparison groups in second grade. Moreover, the bottom panel shows that by fourth grade the math penalty for voucher users persists, but the gains in language have dissipated to insignificance.

Additional testing using second grade data affirms the weakness of any voucher effects. When we do not control for prior achievement, the results do not favor voucher users: The math penalty is maintained, and the language advantage over the public school group is eliminated. Moreover, the rejected applicants report statistically significant test score gains in reading and language. When we compare users only against those who were not offered a voucher (the treatment effect), users report scores that are lower by 0.14 standard deviation in reading and 0.11 standard deviation in math, with no difference in language. When we examine whether those who have participated the longest in the program obtain the strongest effects, we find mixed effects across the three subjects.

Finally, we test for whether the impact of vouchers differs by race. Restricting the sample to black students, voucher users appear more disadvantaged: Their reading scores are now statistically significant and lower; the math penalty is now not statistically significant, but remains negative; and the language advantage is not discernible. These subanalyses give no indication that vouchers have a differential and beneficial impact for black students.

Overall, there is no clear advantage for voucher students; if anything, there is a slight academic penalty. However, given that in 2001 the value of the voucher was less than $2,400, and the opportunity cost in the public school system was approximately $8,800, it might be concluded that the voucher program is cost-effective. Yet, back-of-an-envelope calculations show that this conclusion is premature. The CSP voucher does not include transportation, which must be paid by the district and which may be as high as $1,400 under a large-scale voucher program with children going to many different schools (see Levin and Driver, 1997), and standardized assessments, which public schools must impose. It does not serve special education students or the most disadvantaged students (who do not make use of the voucher). And because the program is subadditive, for every three vouchers distributed approximately one student would have attended private school anyway. There are also additional administrative costs. Finally, the $8,800 figure for CMSD is the average cost per student, not the marginal cost. With approximate costs of transportation, assessment, special education, subadditivity, and administration taken into account, the anticipated fiscal savings falls from $6,400 ($8,800 – $2,400) to perhaps $2,500– $3,000. Although this is still a sizeable cost savings, it does not account for marginal costs, student disadvantage, or any reorganization costs.

Conclusion

Recent research sheds light on the efficacy of vouchers. Broadly, it may be questioned whether vouchers—even with some modest achievement gains—will be a catalyst for educational improvements. On the demand side, there are many other factors influencing the choices parents make. On the supply side, most participating schools are religious, with little evidence of new secular schools opening or accepting vouchers. Also, there is limited guidance on what makes for an efficient private school. Thus, competition between schools will probably be muted, and nonreligious students are unlikely to have more options. Finally, general equilibrium models explain why homeowning voters are wary about expanding voucher programs.

The Cleveland Scholarship program readily fits into this general pattern, evincing similar features to voucher programs in Milwaukee, Florida, and Washington, D.C. Although targeted at students from low-income families, these programs in fact serve those somewhat closer to the middle of the income distribution when usage rates and prior schooling are accounted for. Students are highly likely to be in religious schools, and black students are less likely to use their voucher.

Importantly, the CSP results are not encouraging with respect to achievement: The program shows a slight academic penalty for voucher users relative to other comparison groups. Thus, it seems unlikely that an expanded statewide program will radically enhance educational outcomes in Ohio.

Recommended Reading

For a full version of this paper, including the bibliography for all sources cited in this Commentary (except the additional one noted below), see:

Clive R. Belfield. 2006. “The Evidence on Education Vouchers: An Application to the Cleveland Scholarship and Tutoring Program.” Working Paper, no. 112, NCSPE.

Harry M. Levin and Cyrus Driver. 1997. “Costs of an Educational Voucher System.” Education Economics, vol. 5, pp. 265–83.

Clive R. Belfield is aprofessor of economics at Queens College at the City University of New York.

This Commentary was adapted from a paper Dr. Belfield presented at the Conference on Innovation and Education, held at the Federal Reserve Bank of Cleveland on November 17-18, 2005.

The views expressed here are those of the author and not necessarily those of the Federal Reserve Bank of Cleveland, the Board of Governors of the Federal Reserve System, or its staff.