The Illinois Network of Charter Schools, the state’s main pro-charter advocacy organization, took exception to my earlier analysis of Chicago’s market in high schools. In particular, they don’t like my conclusion that charter schools represent a particularly troubling failure of that market, in that mediocre or poorly-performing charter schools seem to be attracting just as many students as high-performing charters.

Last week, the INCS’ Catherine Deutsch, a senior manager there, wrote a response in Catalyst Chicago, entitled “Charters not the problem with high school ‘market.'” In explaining where my analysis went wrong, the piece focused on two things I should have done with my data, but didn’t. The points are as follows:

1. “Focus on growth instead of attainment.”

This was one of the arguments Matt McCabe made in his response, and which I mostly conceded. Ideally, we would measure school quality by how much they help students grow, not whether they attract good students to begin with. It’s not totally clear, though, that it makes sense to use growth measures to compare charters and regular public schools.

Here’s why. Growth measures depend on comparing test scores across years: a class’ average score in 9th grade, for example, compared to the same class’ average four years later. But since not all students, obviously, are in the same position to make large gains, it matters that schools don’t skew their student body towards the most-disciplined, best-positioned children.

But many charter schools very clearly do just that. The Noble network of schools – which, incidentally, makes up all ten of the top-performing charter schools in the city according to growth – admits that it loses about 35% of each freshman class by senior year. Urban Prep, another star performer, has lost as much as half of its freshmen four years later. Why? Probably a lot of reasons, although at least in the case of Noble, discipline so harsh that it attracted Congressional attention (and collected $400,000 in student fines) might have something to do with it.

In any case, if you’re measuring growth in a given cohort of students, removing the least-disciplined third of the cohort is a great way to skew your numbers. I should be clear that I’m not accusing schools of doing this on purpose; but regardless of intent, the effect is the same. Regular public schools, on the other hand, are much more constrained in their ability to remove students. Maybe this is a good argument for not using growth; maybe it isn’t. But the matter isn’t nearly as cut and dry as the INCS piece suggests.

2. “Exclude selective schools.”

This was also part of Matt McCabe’s response; as I’ve written before, whether you want to do this depends on what question you’re asking. If we’re trying to judge families’ choices about where to send their kids, then of course it doesn’t make sense to punish them for not enrolling in schools they can’t enroll in.

But if we want to have a debate about whether letting students choose new schools will give them meaningfully better educational opportunities, then I’m not sure why we would begin that debate by agreeing to ignore the fact that nearly all of the city’s acceptable schools – let alone best – don’t allow students to choose them. As I’ve said before, this obviously isn’t the parents’ fault; but my original point wasn’t that families are getting a failing grade in making the “right” school choices, but that for a variety of reasons, students aren’t moving to above-average schools in anything like the numbers you would want for this to be a viable strategy. Matt McCabe, the INCS, and I all agree that the selective enrollment system is a major cause of this problem. I assume we all agree that the selective enrollment system isn’t going anywhere anytime soon. We disagree, I guess, as to whether that matters for the future of school-market-based reform strategies.

But for the sake of argument, I’ll concede the point, so we can move on to the big reveal: INCS’s redone version of my graphs, adjusted for these two points.

I don’t get it. Remember that the rebuttal’s central accusation is that I was wrong when I concluded that, in Deutsch’s words, “families are activating their option to leave their low-performing neighborhood school, but sometimes in favor of an equally mediocre charter school.” Now let’s turn back to the INCS’s graph, which I have annotated:

The overall proportions, in fact, don’t appear to be significantly different from what I originally found:

Using a semi-exact method (tracing the bars on a piece of paper over my computer screen), I compared the enrollment gains to “equally mediocre” and non-mediocre charter schools across our two graphs. Here’s what I got (it looks unofficial, but you can easily check for yourself at your desk in about 2.5 minutes):

The INCS uses a different scale, so their bars are a bit shorter, but the ratio between the enrollment gains at “equally mediocre” and non-mediocre charter schools are almost exactly the same. If anything, my data is a bit more generous to charter schools, with a slightly higher unambiguously-better-to-“equally-mediocre” ratio.

So the Illinois Network of Charter Schools issued a rebuttal to my numbers by presenting their own numbers, which are the same as my numbers. In other words, regardless of what you think of points 1 and 2 above, there is no rebuttal here. The analysis is pretty much the same either way: something like a third to forty percent of enrollment increases at charter schools happened at schools that are “equally mediocre” as low-performing CPS schools, and another sizable chunk happened at schools that are only slightly better. If INCS thinks those numbers are bad – bad enough that they felt the need to defend themselves publicly from them – then they have a much more serious problem than an unflattering op-ed. The system isn’t working, even by their own standards.

*

I would like to take all of this one step further, and touch on something that I mentioned only in passing in the original article, and which was not mentioned at all in the INCS response. That is, let’s suppose that students who move to new schools actually do choose better ones, where they get meaningfully better educations. (Although note that both I and the INCS have just provided data that strongly suggests that is not the case.) What then? What happens to losing schools, and the kids who don’t leave them?

The best-case scenario, I guess, is that the schools that lose students realize they have to shape up, and do whatever it takes to improve, and then stop losing kids. In other words, the market forces them to get better. But that seems dubious; we already know that major enrollment losses lead to major financial problems, which seems unlikely to be a catalyst for academic success. What’s more, it seems likely that the kids who are savvy and proactive enough to move to better schools will, on average, be from more advantageous backgrounds, and more academically motivated, than the kids who choose to stay in their terrible schools. Both of those things would suggest that being on the losing end of the market in schools would actually lead to a sort of downward spiral, where the best kids leave, making the quality of education worse, which causes more good students to leave, and so on.

This is something we can make a graph of.

In fact, it appears that every 20% decline in a school’s enrollment is associated with a half-point decrease in that school’s average ACT score. This isn’t slam-dunk evidence, of course: it can’t distinguish between academic decline caused by students leaving, and students leaving because of academic decline. But if there were a downward spiral effect, this is what it would look like.

So okay. Good kids leave bad schools; bad schools get worse. What do we do then? Probably, at some point, we declare that these bad schools are failing and under-enrolled, and we close them, and force all the remaining students to disperse to other schools. This causes an awful lot of disruption, and costs a lot of money in that we’ve created an under-used school for several years, and then have to find something to do with the empty building. It’s also – if last year’s closures were any indication – a fairly traumatic event for the community around the school.

But okay. We’ve closed the school, forced everyone to move to better ones. Are we done? Have we solved the problem of school quality, at the cost of closing dozens upon dozens of additional schools?

No. Let’s pretend for a moment that scattering the last holdouts from the district’s very worst-performing schools won’t have a negative effect on the quality of education at the schools that receive them, which up until this point have benefited from essentially culling better-than-average students from the system. Let’s pretend their scores and growth stay just like they are now.

They’re still unacceptably low. Only the very best of the Noble network’s schools – the ones that, remember, kick out something like the bottom third of their students from each class – end up graduating cohorts that have, on average, the bare minimum ACT score required for going to a four-year college. Basically everyone else is way below that. If we take the city’s rhetoric about college-readiness at face value, this doesn’t cut it.

So do we open up another generation of charter schools, which will poach students from the current ones, in a repeating cycle of opening and throwing away schools in the hope that, eventually, we’ll reach some acceptable level of education quality? Does that make any sense? What are the social and financial costs to that kind of strategy?

I guess you can be the judge of that. But I would be very interested to hear what INCS thinks is the happy ending to the school market strategy.