School Satisfaction? Questions about the Survey
Posted on July 2, 2008
Mayor Bloomberg and NYC Schools Chancellor Joel Klein announced the results of the largest ever school satisfaction survey yesterday. More than 800,000 stakeholders, including students, parents, and teachers responded, a remarkable achievement. As the Dept. of Education noted, 800,000 people represents 1 in 10 New Yorkers, and over 200,000 more people than the entire populations of Boston, Washington DC, and Denver. But City Hall praised the survey results as evidence of dramatic improvements citywide, and exposes itself to criticism as a result.
Most misleading is the bold, italicized headline: “More than 90 Percent of Parents Satisfied with the Quality of Education their Children Received.” Deeper into the announcement, that number is significantly qualified, as only 40% of the eligible parents actually completed the survey. Ninety percent of the 40% who completed the survey is a lot less than 90% of parents overall.
I’m by no means a qualified statistician, but my introductory stats class taught that a scientifically selected random sampling would produce more reliable results than voluntary returns from every parent, student and teacher. Without reviewing the actual survey, it’s a fairly safe assumption that the response rates were significantly higher in well performing schools, where parents, students, and teachers have greater incentives to comply with school requests, than in chronically underperforming schools.
In fairness, school performance metrics have trended upwards the past few years — so much so that the NYC DOE received the Broad Prize for improvement last year.
Critics say the metrics have improved, at least in part, because of funny math that has fundamentally changed what the metrics measure and ignore entire student populations in some cases. For example, literacy metrics used to include ESL (English as Second Language) students’ test results, and now they don’t. Removing their scores artificially inflates the numbers; as a result touting any “improvements” by comparing those data sets is fundamentally misleading.
Another example: In the wake of No Child Left Behind, when schools were threatened with diminished funding after successive years of underperformance, the city began restructuring suspect schools. Specifically, they would take a building that housed a threatened school, “close” the poor performing school and reopen multiple smaller schools within the same building. Often these “new” schools had fundamentally the same students, teachers, and administration, plus an additional layer of bureaucracy to manage the new schools. But because they were technically new schools, their funding was safe as they had no history against which to hold them accountable.
All that said, Mayor Bloomberg and Chancellor Klein deserve credit for staking much of the legacy of his administration on the success, or lack thereof, of their education reform initiatives. The results remain mixed, but efforts to give voice to stakeholders — like this survey, flawed though it may have been — give us reason for hope.
20/20 Vision anyone?
» Filed Under education reform, nycdoe, research
