The U.S. Department of Education released last week a summary of achievement trends in schools receiving School Improvement Grants (SIG).  This is the main federal strategy to turn around the nation’s lowest performing schools, which has directed $7 billion to 1,500 schools.

The reaction from the field is that once again, this is the same old news of mixed results.   “Only a little more than half of the schools improved, while the other half saw stagnant student achievement, or actually slid backward,” wrote Alyson Klein in Education Week.  Andy Smarick of Bellwether wrote that SIG could use a turnaround itself.  Among the very real problems: almost half of the grant recipients were excluded from the federal summary because of changes in how state tests are given.

But isn’t the look at whether or not SIG has effects “on average” missing the point?

Imagine that the $7 billion had been spent by a biotech company seeking a cure for stomach cancer.  After completing the clinical trials for “SIG-2”, they find it doubles a patient’s life expectancy.  The five-year survival rate for those with stomach cancer jumps from 24 to 48%.

Would you say that “SIG-2” fails a majority of the time?  NO!!! Tens of thousands people would live at least another year and the world would be overjoyed at these results.

Schools receiving SIG funds are chronically failing.  While they might have some talented individuals, as organizations, they lack the “know-how” that helps a school accomplish the basics.  Even getting the bells to ring on time can be difficult, to say nothing of ambitious instruction.  More money may or may not solve their capacity problems.  Even when making double-digit achievement gains, their move out of “crisis” is more likely into a stage of mediocrity rather than excellence.

Given these challenges, it makes sense for analysts to stop looking exclusively at average changes.  Even if positive effects are seen in only 20 percent of the schools, future analyses should examine under what conditions, and why, SIG gets the results it does.