Researchers: ACT ‘not a test you can game’

Print More

Posted May 30, 2008: Forced to announce another year of disappointing ACT scores, Schools CEO Arne Duncan last summer sought to temper the news by talking up the district’s investment in strategic improvements, highlighting some $3.5 million earmarked for test preparation.

Later, after the school year began, high school officials say they quietly decided to “repurpose” that money to ramp up the curriculum. Michael Lach, who last August took over the office of high school teaching and learning, says he worried preparation programs, where students were given strategies rather than learning content, weren’t worth the investment.

Echoing previous research, a study released by the Consortium on Chicago School Research this week shows test preparation activities, especially those done during class time, do not improve scores on the ACT. In fact, schools where 40 percent of class time was spent on test prep mostly scored worse than schools that spent only one-fifth of their class periods on test prep.

“This is not a test that you can game,” says researcher Elaine Allensworth, co-author of the report. “There is no quick fix on this.”


Research summary

“From High School to the Future: ACT Preparation—Too much, Too Late” by Elaine Allensworth, Macarena Correa and Steve Ponisciak
 
In this era of high-stakes testing, the Consortium on Chicago School Research examined the impact of test preparation on ACT test scores. They found:

  • Schools where teachers spend more time doing test preparation post lower ACT scores than schools where teachers devote less time to it.
  • The misconception that strong test-taking skills leads to higher ACT scores is widely held by CPS students and teachers.
  • Students who do well on the ACT are the ones who exceed state academic standards when arrive as freshmen. Most high school students barely meet state standards.

The report comes at a critical time for Chicago Public Schools. As elementary school pass rates on state standards tests rose some 20 points in the past five years, high schools overall have failed to make such performance gains, with scores creeping up less than one point.

The benchmark high school test is the Prairie State Achievement Exam, which is given to juniors and includes the ACT and a job-readiness skills test. Chicago’s average ACT score is 17.2—less than the statewide average of 20.5 and the minimum of 20 that students need to get into selective colleges.

Consortium researchers found that the only type of test preparation that resulted in higher scores was taking a full-length practice ACT test. But even that had diminishing returns as students took it multiple times, according to the authors.

Instruction and content mattered more than test preparation. Researchers found surefire indicators of improved ACT scores were higher grades in junior year classes, attentive students who cared about their school work and more students with 8th grade score that exceeded state standards.

The district’s high-profile High School Transformation effort is geared toward improving instruction and cultivating students’ critical thinking and problem solving skills, Lach says. The $80 million effort also provides teachers with support on how to best relay the new curriculum to students.

(A Catalyst report released earlier this year showed that High School Transformation is having some success with motivating students and getting teachers to improve instruction, but obstacles at the lowest performing schools—late freshmen registration and high student absence rates—are getting in the way. See Catalyst Special Report Spring 2008.)

Pressure is on

Over the next several years, including this school year, the district is spending only about $500,000 on two online ACT test preparation programs: ACT Prep Online and Keytrain, according to Lach. But many high schools use their own discretionary funds to buy test-prep programs and training from private vendors.

Teachers and principals have misconceptions about the value of test prep, Lach says. According to the study, juniors say nearly every day, classroom teachers devote time to preparing for ACT. And close to 60 percent of teachers surveyed say they believe student scores are determined by test taking skills—a proposition that the study refutes.

“Most teachers want to do right by kids,” Lach says. “Our job is to get them smarter about how to do that.”

Meanwhile, the pressure on educators and students to do well on this high-stakes test is widespread, says Jason Nault, assistant principal of Global Visions, a small high school in South Chicago. Students know that their ACT scores determine which colleges they will have access to, and schools are judged—and sanctioned—in part on those scores, too.

“So much is put into the test. The pressure is tremendous,” he says. “I really think it is unfair.”

Still, Nault says he and the principal don’t want teachers to overdo test preparation, but they recommend teaching students certain skills, such as the process of elimination.

Next year, Global Visions pilot a new curriculum designed by ACT. It promises rigorous content and better preparation for college, Nault says. Eventually, the school may join the district’s High School Transformation initiative.

Almost no chance

On average, freshmen arrive at Global Visions scoring well behind their peers in the city—a sign that it will be nearly impossible for them to score 20 or more on the ACT, researchers say. Last year, only 30 percent of students made expected gains on the standardized tests from freshman to junior year.

And even those who are on grade level have a long way to go.

Researchers point out that most students come to high school meeting standards, but while they may have conquered the basic skills necessary to hit targets, they are not learning the deeper critical thinking and problem-solving skills needed for high school tests.

Of the 58 percent of CPS students who met 8th-grade reading standards, fewer than one in four will hit the reading benchmarks on the ACT three years later.

This is especially true for black and Latino students. Not only do black and Latino students come into high school further behind academically, they also post lower gains on tests than white and Asian students—further widening the achievement gap from about three to five points.

Researchers argue that this is at least partly explained by white students being more likely to perform better on tests in 8thgrade and more likely to go to better high schools where they post larger gains.

Ferdinand Wipachit, principal at Phoenix Military Academy, says he realizes that there are no easy answers. His school, which is almost 100 percent black, Latino and low-income, has seen a modest 1.7 point improvement in ACT scores since its first junior class took it in 2003. Still, the average score is 15.2.

The uptick at Phoenix is the result of teachers focusing on reading skills, such as how to analyze, synthesize and infer. Traditional test preparation activities are offered after school or on Saturdays. 

At schools with more advantaged students, test prep is seldom substituted for content and curriculum. At Chicago International Charter Northtown Academy, about 30 percent of students are white and the average ACT test score is 19.9—a 1.6 point improvement since 2004.

Jamie Troiano, the school’s humanities chair, says building students’ ability to think and learn is the best form of test preparation. Anything else would be “cheating” students, Troiano says.

After school, a private consultant holds test skills classes that about a third of students enroll in for $250 each. The school subsidizes the cost for students who receive free or reduced-price lunches, so some students pay as little as $50.

Researcher Allensworth says test preparation that does not interfere with classroom instruction seems to make a modest difference if students show up to each class. But the primary reason for this boost may be that these students are especially motivated to do well, she notes.

Consultant Gary Solomon of Princeton Review, which sells test preparation material to some CPS schools, says he tells principals that his products are not a replacement for academic content and instruction. 

Yet Solomon contends that Princeton Review’s strategies—like finding and eliminating answer options that are meant to distract—can help students perform better on the ACT.

Princeton Review also offers several versions of a full-length practice ACT test, a strategy that researchers say works to some degree.

“I would never, ever say that this should take away from class time, but with the right balance it can be good for the kid that just needs a little nudge,” he says.

Students want to do well

Given where they are academically, when Global Visions freshmen take the ACT and see how hard it is, they get frustrated, Nault says.

“It makes them feel stupid,” he says. “They get discouraged.”

At the Black Star Project on the South Side, students often show up disenchanted after a poor showing on the ACT, says Kirsten Rokke, who runs the tutoring program at the advocacy and educational organization.

Before taking the test, students who are doing well in school think that they will also do well on the ACT.

“They think I will get a 20 or a 25, but then they wind up with 14 and they don’t understand,” she says.

At the Black Star Project, they veer away from teaching strategies and instead have the students take practice tests and then hone in on what questions they are getting wrong and why.

Schools also are taking steps to make sure juniors understand the importance of the exam. Wipachit at Phoenix says sophomores decorate the halls with encouraging signs and banners for the juniors. “It’s almost like homecoming.”

He says almost 100 percent of juniors came to school on testing day this year.

However, research shows that students are already motivated to do well on the ACT.

Another Consortium study released in March also showed that the vast majority of high school students want to go to college. (Both studies are part of the consortium’s ongoing Chicago Postsecondary Transition Project, which uses a combination of surveys and interviews with ChicagoPublic School students and analysis of data.)

The earlier study showed that some 90 percent of seniors say they want to go to college, but only 61 percent of those who aspired to attend college were enrolled in one by fall.

Allensworth notes that never in the past have high schools been called upon to prepare all their students for college. This change for high schools means that they need to alter their curriculum to not just have a college track, as was often done in the past.

“It is a very ambitious goal,” she says. “No one anywhere has ever done it.”