Intervention stories “dishonest” in presentation of data

Print More

Probably the most obviously dishonest thing about the way in which CATALYST’s August 1 “Web Special” on “intervention” presents the CPS data on the intervention schools is the failure to mention the fact that the 2000 data compare(s) the combined scores for 9th and 10th graders, while the data from 1996 through 1999 is comparing 9th and 11th graders. A footnote is not sufficient to this important material. An entire critical essay could be written about the Vallas administration’s TAP shift between May 1999 and May 2000 (and the ludicrous claim that the ACT test is equivalent to the TAP for use with 11th graders). There is no way that the data from “1996 to 2000” can be compared, despite how neatly they line up on your charts. The data from 1996 to 1999 can be compared with reservations. Assuming that the “9th and 10th grade” scores from 1999 and 2000 have any statistical reliability as of yet, the only thing that can’t be said about them is that they can be compared with the data from 1996 through 1998. From 1996 to 1999 (as the official reports show), we were comparing the combined scores of 9th and 11th graders. In fact, the data released by the CPS in June 1999 don’t even mention 10th grade TAP scores! Then, suddenly, we are not only told that they exist, but that a meaningful comparison can be made for the purpose of “proving” that intervention was in order for five high schools.

A bit of care with the numbers would demand that the data be challenged. But that approach wouldn’t be consistent with the new party line, and CATALYST has shown a consistent way of adapting its coverage of CPS “news” to each new permutation in the Vallas administration’s party line. Combined with your colleague’s fawning praise of Vallas (and pompous condemnation of the schools facing intervention) in the Op Ed pages of the Chicago Tribune earlier this month, the full weight and credibility of CATALYST (and its parent organization) once again prop up the official line of the present administration of Chicago’s public schools.

It would have been a greater service to the public (and consistent with any tradition of an independent press in this town) to scrutinize the data and their sources, rather than simply (once again) parroting the (now revised, again) party line from CPS headquarters.

Since the Vallas administration’s numbers games began more than five years ago, both the numbers and the people who have interpreted them have been crafting bigger and bigger lies based on them. Both the numbers and those who utilize them in ways that promote schemes like “intervention” lie.

By this time, hundreds of critics from across the nation have weighed in against Chicago’s abusive use of the Iowa and TAP data. Additionally, the administrator’s guides for the Iowa and TAP tests warn specifically against using the data the way Chicago has used them. The corporate public relations departments for Riverside and its corporate parent provide a fig leaf behind which the data are still defended in their Chicago uses. A handful of local academics and media, all of whom receive generous subsidies directly (in the form of grants and programs) or indirectly (in the form of data, information, and easy access to news sources) through the Vallas administration or City Hall, complete the circle of deceit. But that doesn’t make the lies truth.

From the beginning it’s been known widely (except in Chicago’s media and a couple of academic circles) that the Iowa and TAP tests should not be used (a) for promotion decisions regarding students, or (b) to evaluate the success of failure of school programs. During the past six months, our staff have traveled widely, and there has been no serious scholarly defense of Chicago’s use of the Iowa and TAP tests for these purposes. None. Everywhere there is only criticism or ridicule. Unfortunately, the tragedies (to individual children and to entire schools) continue here in Chicago because our colleagues in the press continue in bed with their sources.

The first lie imbedded in the data you’ve published slandering the communities, teachers, and students of Bowen, Collins, DuSable, Orr and South Shore high schools is to publish them at all as if they can be used to evaluate schools. TAP data are not to be used for these purposes, period. Everyone has warned against it, and yet you publish charts as simplistic as the current standings in the American League, Central Division, so that people then are led to the wrong conclusions.

Once you’ve published the “standings” in the form crafted by CPS officials, your readers are forced to draw the same conclusions the CPS wants drawn. The lie is then broadcast from a supposedly “independent” source, and CPS recycles that fact to its public relations profit.

But the abusive publication of the TAP data is only the beginning.

A second lie imbedded in the numbers follows. That lie involves the sleight of hand trick used by Phil Hansen’s Office of Accountability between the 1998-1999 school year and the 1999-2000 school year.

Until May 1999, the tests were the “combined 9th and 11th grade” scores.

Suddenly, in 2000, we were able to combine 9th and 10th grade scores and (if CATALYST’s chart is to be believed) make meaningful comparisons all the way back to 1996.

By publishing these (“reading” and “math”) data without any explanation of this serious difference, you serve your Office of Accountability sources and the Vallas administration well. The same is true in the other uncritical ways in which you and your reporters present the “data” the CPS carefully provides to you and to your publication.

Of course, there is always the question of whether the data from the TAP tests can yield any meaningful information about the performance of a school, let alone be used to judge the value of the work of teachers and principals. Since you have also accepted the widely debunked base line (the TAP and Iowa tests) as the measure of schools and students in Chicago, you were caught on that one before the latest debate ever began.

It would be interesting to see any of our colleagues in the press revise their utilization of the “Source: CPS” data, but since that would now require an honest reevaluation of everything going back at least five years, it’s unlikely. Better to continue discussing how well the emperor is dressed than to admit to having been once of those who didn’t notice the nakedness from the beginning. How much longer do the editors and reporters at CATALYST believe that charts such as the “intervention” indictment against Bowen, Collins, DuSable, Orr, and South Shore can be published without critical scrutiny?

And once the invalidity of the TAP scores (whatever their iteration of manipulation) is revealed and discussed in depth for this purpose, what then happens to all of those somber articles on Web pages going back all the way to 1996? The CATALYST clip files (not alone, of course) will then read like those cheerleading articles from Washington and Saigon between 1965 and 1968 — a compendium of government propaganda, rehashed by the “independent” media, and then foisted off as accurate — until the whole thing collapses. The Big Lie doesn’t become less of a lie just because it’s repeated on a smaller scale in a different context — and with less deadly impact.

One additional question that must arise (like a ghost from the past) is how those great CASE “pilot” tests you praised so avidly could suddenly become so useless (not only, by the way, in “English”) with the elimination of the “required” readings (as you put it in your other recent Web posting). After all, how can anyone tell since the tests have been “secret” since CATALYST and Substance were reading them back in 1998 and 1999?

To report with a straight face that the “Pilot” CASE of 1998 through 2000 will be the same after the elimination of what you delicately call the “required readings” is like reporting that someone became unpregenant without mentioning the abortion that terminated the pregnancy.

A large percentage of each of the multiple choice CASE English (I and II) tests I’ve seen was based on those “required” readings. Usually 100 percent of the “constructed response” sections was also based on those “required readings.” Now that the “readings” are no longer required, the “pilot tests” for which I was sued (while you sat in frightened silence) and fired (looming now) are, as I said nearly two years ago, a waste of time and money. To extrapolate to the point I was trying to make two years ago, before such a rude and expensive interruption to my career and life, the whole thing has been, and is, a hoax.

Your August 1 Web specials were quite interesting. I hope thousands of people get them before they become non events, since they reveal so much under scrutiny.

It remains to be seen how you will cover the stories when you reduce your material to print next month — and how long you can continue doing it with straight faces. Haven’t your reporters and other editors noticed the growing number of statistical anomalies in your coverage of the official version of history? Together, CASE, TAP scores, and intervention are a part of the larger hoax that CATALYST has helped perpetrate through its uncritical coverage of the Vallas administration.

I only wonder how much longer my colleagues in the press will continue with this goofy brand of “journalism”.

It’s time to give a careful and critical look at the numbers upon which these numbers games have been based since 1995, when City Hall and its public relations machines took over the public schools of Chicago. Rather than begin with the rougher numbers, I propose two challenges that might be easier to undertake as part of what we might call “Project Second Look.” Both involve numerical claims from the “CPS” that are smaller bites than the big two.

1. The Poverty Challenge. Poverty Data for the high schools. Amundsen versus DuSable.

Here is a challenge that’s independent of both of these touchy subjects. Why not examine the “low income” data using the sophisticated marketing tools now available to those of us familiar with the Web? Then we could look behind the numbers by comparing how Amundsen High School (located in marketing zones — not just the gross data provided by ZIP Codes) could maintain a “low income” rate higher than those of most of the schools on “intervention” — including DuSable! Could it be that the books have been cooked for years, and that what now has to be done is to pick apart each of the lies, starting with an easy one like Amundsen’s “poverty rate” and then (as our reporters master the learning curve on this) finally getting to the two big ones: test score data and the budget numbers.

2. The Academic Magnet Challenge. Track the fate of student scores within the “Academic Magnet High Schools.”

At least three of the Academic Magnet High Schools (Northside, Payton, and Jones) refuse to admit students whose standardized test scores are “below” the national average. Therefore, when the students walk in the doors of these schools, 100 percent of the students are scoring “at or above the national norm” in reading and math. Rather than simply publish the “Percentage at or above the national norm” when evaluating these schools beginning next year, why not demand that the board provide individual student data, using student ID numbers to protect anonymity, and see how many go “up” or “down.”

The problem is that a student who enters Northside with a score at the 99th percentile in “reading” on the Iowa test can drop all the way to the 51st percentile without having any impact on the school’s “scores” in the way they are presently reported. The school could be a complete failure and nobody would notice as long as the public is told to worship the “percentage at or above the national norm.” We have to know how individual students are doing to know how well the school is doing, and the percentages don’t give us any meaningful data when published as they are now.

Conversely, it is possible for a school to have a “score” of “0” based on current reporting methods and be very successful. For example, if all of the students entering a particular high school can in at, say, the 20th percentile in “reading”, the score at 9th grade would be “0”. If all of them “doubled” their score between 9th and 10th grade (to the 40th percentile), the school would have done a great deal to help them, but the school would still show no improvement by the current CPS way of measuring things.

Of course, if you were to demand these data, the Board and your convenient sources would refuse to provide you with them. I guess it just gets to be so warm when a source makes such a comfortable bed that a reporter will do anything rather than go outside of it.

George Schmidt, editor Substance