STAR-Struck: Big Problems with High-Stakes Standardized Testing

0. Abstract

As a public school teacher, I believe that we should stop taking the student proficiency tests used in California to determine school budgets. What we're now learning is that the current system is a colossal drain on our economy at every level; our GDP would be trillions higher if we didn't rely on test results to fund schools.

Every year, public school students across this state take STAR tests, multiple-choice standardized exams on the content standards teachers cover in their classes. However, STAR tests are written, administered, and graded incoherently, and are based on obsolete ideas about what kids should learn in the first place. Therefore the data they yield doesn't actually tell us how "proficient" kids are, nor how well schools are "performing." And this is same the data that determines how much money our schools get, how much our homes our worth, and how our students' time in schools will be spent. There's a lot of work to do to fix schools here in California, and every reason to believe that our state and federal governments' next policy decisions will make things much worse. The first step citizens can take is to boycott the tests. It's legal, free, and easy. And it will send a powerful message to policy-makers, who too often put the interests of the testing industry before the interests of families—but it will only work as long as enough people do it in unison.

1. Trillions of Dollars Higher

I want to convince you that the standardized proficiency tests used to label and track our students are unfair. I don't simply mean that it's unfair to make kids sit in desks for days filling in bubbles with No. 2 pencils. It turns out that the tests are unfair to all of us. To begin with, here is some recent news about how our schools affect the economy, from the New York Times (emphasis added):

April 29, 2009: Persistent Racial Gap Seen in Students' Test Scores

The achievement gap between white and minority students has not narrowed in recent years, despite the focus of the No Child Left Behind [NCLB] law on improving the scores of blacks and Hispanics, according to the results of a federal test considered to be the nation's best measure of long-term trends in math and reading proficiency. Between 2004 and last year, scores for young minority students increased, but so did those of white students, leaving the achievement gap stubbornly wide, despite President George W. Bush’s frequent assertions that the No Child law was having a dramatic effect. (full article)

April 23, 2009: Study Cites Dire Economic Impact of Poor Schools

WASHINGTON — The lagging performance of American schoolchildren, particularly among poor and minority students, has had a negative economic impact on the country that exceeds that of the current recession, according to a report released on Wednesday. [...] The report concluded that if those achievement gaps were closed, the yearly gross domestic product of the United States would be trillions of dollars higher, or $3 billion to $5 billion more per day. (full article)

Across the country, something about the way we teach our students and then measure their learning is utterly disastrous. What about the neighborhood impacts of standardized tests? An Ohio State University study from 2006 details the relationship between testing and house prices. Here's a quote from the study's co-author, professor Donald Haurin:

“In Ohio, there are districts with 20 percent pass rates and some with 85 percent pass rates, so based on our findings that would result in about a 23 percent difference in house values solely because of the schools. It is not a trivial amount." (full article, and for data about places other than Ohio, try this)

So our home equity is closely tied to these proficiency tests; our GDP is too. Yet public monies from black, brown, and white paychecks alike keep the whole system afloat to begin with. It seems worth asking: What are these tests, and what is going on with them?

There are many, first of all. The regimen differs by state and grade, and I'm only familiar with California's high school level exams. Among others, we have the CELDT, the CAHSEE (about which I'll have more to say in another post), the Aprenda, and the EAP. There is also the SAT, the PSAT, the SAT II, and the ACT—but these don't influence our mortgages as much. I will focus for now on the one that does the most: the California Standardized Testing and Reporting (STAR) program.

2. Old School: The Caged Lion

We lived and died by the STAR tests at my old school (which someone wrote a book about) because we barely scored enough points to avoid the dreaded "state takeover." This was a working-class district; lots of students were not native English speakers, you had kids with after school jobs, kids taking care of younger kids, addicted kids, kids with addicted parents, kids in otherwise broken families, kids in gangs, and so on. Few prioritized academics and testing as we adults wished they would. Soon enough, news came that the other big high school in town let their scores drop too far and would be taken over by the state. They had their schedule torn up and remade, minus many elective classes. Bureaucrats from various county or state offices frequented campus, mandated terrible curriculum, and earned big paychecks. Excess teachers got shuffled to other schools or otherwise lost autonomy. It sounded like less fun than a funeral.

Meanwhile, our school hung on by framing the tests as a last line of defense. If you guys don't do well enough, we told the students, you can lose electives. You want to keep art class, right? Look what happened to our cross-town rival. Do you want that to happen here? Propaganda was another useful trick. The year before I was hired they threw a rally during which the school's mascot, Monty the Lion, marched out onto the football field in a cage to symbolize the loss of academic freedoms we faced. Teachers were still talking about how well this worked the following year ("The students really got it!"), and calling for an encore. It didn't happen, although when we got closer to testing season I do remember seeing flyers up around campus reviving the Monty-behind-bars motif. That's quite a pep rally when you reflect on the statistics that predicted how many of those kids were likely to end up in jail.

At the end of my first year, we left for the summer in suspense about whether we'd get taken over. We didn't. A nine- or ten-point increase would have been substantial—we jumped about 20. The whole school seemed to sigh with relief. But now we had a new problem: Although we were at less immediate risk of a takeover, we still had to increase our students' "proficiency." NCLB's punitive measures can kick in as soon as a school's test results dip at all, ever. After such a quantum leap, could we really expect another gain in the scores? Had we done too well, and maxed out the kids' brains all at once? Could they do any better this year? Would they?

Now the rhetoric shifted. It came from the administrators at faculty meetings, and I for one dutifully passed the new message along to my classes: You guys did great last year—thanks for your hard work! All we need this year is a couple more points, so please take it seriously and do your best, but don't feel like you have to kill yourself or anything. Just remember: A few points. That's all we need. This backfired colossally; the kids increased the school's results by another 40 points. Again the next September, there were high-fives all around and then the same sinking sensation as we asked ourselves, How long can this go on?

In my third year, our STAR scores dropped by two points. We did not get taken over, but the possibility loomed once again despite our earlier gains. Suddenly we were just as scared as when we started. If our scores fell again, or fell enough, the state would come down and we'd have to cage up the lion forever. I forget how the exact policies worked, but it hardly mattered. In our struggling district, nobody took the STAR tests seriously as a measure of student learning. Instead we saw them for exactly what they are: a direct threat to education. Testing didn't just warp our teaching, it warped our entire school culture.

3. New School: Australia Calling?

My current school is in a more affluent area and suffers less from the "caged lion" pathology. We have a higher STAR average—and the houses here are indeed worth more. As long as we keep coasting gently upward, state takeover remains excluded from our school administrators' stick-and-carrot arsenal. Nevertheless, when it's STAR season they are suddenly everywhere, popping into classrooms to make sure the tests are going OK, exhorting the students over the PA to go to bed early and have a big breakfast, and reading the test directions aloud at faculty meetings. It's the time of year when my principal and vice-principals appear to be working hardest and sleeping least. I do not want to criticize them; they are in a tough position and have to resort to histrionics to get kids' attention. After all, STAR scores can make or break a school's budget, but get this—they have almost no direct consequences for individual students.

A kid's score doesn't affect his GPA or—absent a takeover—his choice of electives. It doesn't show up on his transcript, so colleges don't see or care about it. Teachers can look them up with some effort, but few do. The students know this. Plenty blow the test off, filling in bubbles with wild abandon. Some like to "connect" their bubbles into larger designs (smiley faces are a favorite). If anybody asks them to retake the test properly, they say with total impunity, "But I didn't know any of the answers." One morning I walked into a colleague's class as she was handing out answer sheets, and there was a sharp marijuana smell from the back corner. Clearly there are lots of students who take the time to get the answers right or our scores would not be where they are. Those who do often tell me that the tests are too easy for them.

Still we try. "I wish they'd put the scores on your transcripts," I heard our desperate principal say to a class of juniors. "Some other schools already do. This data can really get around and lots of people have access to it. The other day I got a call from Australia—a family there was thinking of moving here and they'd already looked up our results online!" These days, my own pre-test pep talk is more or less: Your STAR score is nothing personal, but it can give or take away weight from your GPA. A 4.0 from a school with good test scores means more to some universities than a 4.0 from a school with lower scores. So tell everyone you know not to blow it off, because it puts a dent in everyone's shot at college. I realize that's not cool, but it's the way things are. I don't know how true any of our rhetoric is; in any case it fails to impress. The point: At two quite different Bay Area schools, students take STAR tests in an atmosphere of stress and coercion—hardly the conditions for successful bubbling. Meanwhile, the test's consequences are unbalanced to the point of absurdity.

4. One Must First Understand the Kernel Itself

Aside from these problems, is the STAR test a useful tool to measure student achievement? Emphatic No. The test is riddled with questions that are clumsy, contradictory, useless, or impossible, so the data it yields—data upon which so much depends!—is not as meaningful as it may seem. The following example questions come from a packet released by the state for teachers to use as test prep exercises. They come from the 2003-2008 English/Language Arts tests—and admittedly, my criticism is mostly confined to this discipline for now. Some of the items pertain to reading passages which I've removed, although they also warrant close critical analysis. Otherwise you're looking at what the kids look at. The first is my favorite:

89. Read these sentences: "Though the act of the kernel of corn bursting takes place in a split second, the science behind the pop is a bit more complex. To understand it, you must understand the kernel itself." How is the underlined sentence best written?

A If one wants to understand this procedure, you must understand the kernel.
B The kernel itself must be understood if comprehension of this process is to occur.
C First, you need to familiarize yourself with the kernel to understand the popping method.
D In order to comprehend this process, one must first understand the kernel itself.

The answer key says it's D. Alas, everything about this question demonstrates how I teach my students not to write. Relying on big words and overwrought grammar is a common trick insecure writers use to eke by, but it never works. More important to note here is that all students have to take this test regardless of language ability. (And language learners' scores weigh equally with everyone else's.) So imagine that you're a 15-year-old Afghani who immigrated last month, and take a look at question 89 again. Now it's not just bad, it's impossible. Fortunately you won't need much English for the next example:

63. The word volcano is derived from which of the following names for the god of fire?

A Odin
B Hephaestus
C Vulcan
D Pele

Everyone loves freebies; they offer a quick confidence-boost. On the other hand, they insult students' intelligence and mock the pretense that these tests measure learning in the first place.

102. When the final draft of [a] report is typed, what line spacing should be used?

A single-line spacing
B double-line spacing
C 1.5-line spacing
D 2.5-line spacing

Number 102 (or variants about margin widths) is a classic; according to my students, it's been showing up on STAR tests and the California High School Exit Exam for years. They never know how to answer. The correct answer is listed as B—but students learn to format their papers any way they're asked, and different teachers have their own preferences. This is a trivia question, not a language question.

52. In which sentence does the underlined word have the most negative connotation?

A The man's remarks were harmful.
B The man's remarks were mischievous.
C The man's remarks were malicious.
D The man's remarks were unpleasant.

Learning about connotations is useful. But words often connote ranges of ideas unique to each reader. Subjectivity is the whole point; poetry wouldn't exist otherwise. If we "standardize" connotations, as question 52 attempts to do, we turn them into denotations fit only for a dictionary. Thus our only guessing strategy here is retroactive mind-reading: Who would have written a question like this? Which one would they think is the most negative? Why would they have chosen these four words? Are they trying to trick me again? How should I know? I would have gone with A, because to me it connotes actual damage of some sort—isn't that logically the most "negative"? No, the correct answer is C.

86. Read this sentence: "Fortunately, due to a 1911 treety and to rescue and habitat protection efforts, the sea otter population has nearly been restored." What underlined word in this sentence is spelled incorrectly?

A treety
B rescue
C protection
D restored

It would be better if they offered four ways to spell "treety"; treetie, treatie, treaty, or tready. That way you'd at least know if the kid could actually spell it. But since these tests aren't meant to guide education positively, it doesn't matter if the questions provide meaningful data. The purpose is to figure out which schools to punish, so a single, final score is all that's needed for students and schools alike.

Another problem is that this is supposed to be a test about what students learned in a certain grade—but even if we find they can spell treaty, how do we know when they learned how? Examples like this one undercut the logic of yearly testing, which is that there must be new content every year. In fact, the California State English Standards for Grades 9 & 10 (.pdf) are exactly the same. (It's the same story for grades 11 and 12—one list of standards for two years.) The Department of Education doesn't specify which standards are for grade 9 and which are for grade 10. At my old school the English department came up with some rough agreements about which ones to focus on in each grade; at my current school we don't. So do the freshman and sophomore STAR tests cover the same stuff? If so, why make the students take it twice? If not, why are these two years' standards published as a single list?

Beyond English/Language Arts: Here are examples of released questions from other subject tests. In light of the astonishing developments in science and biotechnology over the last few years, it's embarrassing that California's science content standards—source material for the tests—are eleven years old. (.pdf) And although one might expect that at least in mathematics multiple-choice ought to be a fool-proof format, it's not. Here's a detailed analysis of Washington state's standardized math tests by Arthur Hu, columnist for Asian Week, test-scorer, and parent:

[...] most of the problems on the 10th grade test I reviewed could be solved with 5th grade level decimal arithmetic. Almost none of the 4th grade problems can be solved with 4th grade arithmetic alone. Even OSPI studies concluded that many problems were beyond their stated grade level. Committees which set the "standard" never evaluated whether the problems were developmentally appropriate or compliant with grade-level learning standards in the first place. [...]

I have also participated in scoring tests for one state writing project by the same company that scores the WASL. I observed officials who appeared to be moving the definition of scoring points to meet desired pass rates. We were instructed that NO paper was to be scored a "4", the equivalent of an A grade without special permission. (full post)

Do not be fooled. The solution to this problem is not to come up with better tests. In fact, the test that quantifies an entire year of California's thousands of students' learning has not ever been and can not ever be written. If you disagree, try writing one. How many questions should it have? When should students take it? Who will grade it? What should be the "desired pass rates," and how many students would we therefore desire to fail? What are the consequences for failure? Why should it be strictly multiple-choice in the first place? In how many languages do you plan to offer it? When you stop and think about it—especially if you can remember taking them yourself—these tests are an impossible waste of time. That's why this paragraph from the front of the Released Questions Packet is so sad:

All questions on the California Standards Tests are evaluated by committees of content experts, including teachers and administrators, to ensure their appropriateness for measuring the California academic content standards in Grade 9 English-Language Arts. In addition to content, all items are reviewed and approved to ensure their adherence to the principles of fairness and to ensure no bias exists with respect to characteristics such as gender, ethnicity, and language.

But bias does exist. It is intrinsic to NCLB itself. The way the law works is that the government divides school populations into racial "subgroups" which have to improve their collective scores by a certain amount every year—forever. The subgroups that score lowest have to increase their scores at a faster rate than their higher-scoring counterparts in order to "close the achievement gap." This endless game of ethnic catch-up is ridiculous on its face. The rules of the game assume a "perfect" world in which no new language learners show up to drag scores back down and students take seriously the duty of proving their academic worth according to government standards. And if the only way forward is to get fewer and fewer answers wrong, then the ultimate goal must be for everybody to get them all right. Imagine that!

5. Malicious is More Negative than Harmful

If I were feeling charitable, I would say that the "committees of content experts" and other bureaucrats responsible for our nationwide testing regime are hopelessly naive. Unfortunately, there is more to tell:

Thanks to the testing mandated by the federal [...] legislation, private companies are mining the testing field with all the power their accountants, test-makers, and marketers can muster. States are likely to spend $1.9 billion to $5.3 billion between 2002 and 2008 to implement NCLB-mandated tests, according to the non-partisan Government Accounting Office (GAO). Those GAO figures cover just the direct costs [...] Add in indirect costs, such as the amount of classroom teacher time devoted to coordinating and giving the tests and, increasingly, preparing students with ongoing "practice" tests, and testing experts say the figure could be 8 to 15 times higher. [...]

"There's very little oversight of the testing industry," notes Walt Haney, an education professor at Boston College and a senior researcher at its National Board on Educational Testing and Public Policy (NBETPP). "In fact, there is more public oversight of the pet industry and the food we feed our dogs than there is for the quality of tests we make our kids take." (full article)

In what's becoming a more and more familiar narrative in our national consciousness, we find that in order to provide a basic public service, the government relies on a cadre of private companies with proven histories of abject failure. Education officials buy their crude but expensive tests, and school administrators in turn must use them to answer Bush's big question: "Is our children learning?" The people who actually have to fill in the bubbles have little reason to cooperate with the process, but a school's collective punishment for failure can be extreme. So the children who do get left behind are the ones stuck in schools with ever-diminishing resources to pull their scores out of the hole. We rake open the old scars of segregation year after year, and wonder why so many kids vote with their feet:

[...] many states use an inflated graduation rate for federal reporting requirements under the No Child Left Behind law and a different one at home. As a result, researchers say, federal figures obscure a dropout epidemic so severe that only about 70 percent of the one million American students who start ninth grade each year graduate four years later. California, for example, sends to Washington an official graduation rate of 83 percent but reports an estimated 67 percent on a state Web site. (full article)

What about the current administration's plans? They want to give kids better schools than their predecessors did, but continue to assume that tests are the only feasible way to do it. Recently, Education Secretary Arne Duncan came through town to tell California the government has little intention of helping our schools out:

"Honestly, California has lost its way," Duncan told dozens of the state's mayors and education officials who packed into San Francisco City Hall on Friday morning. [...]

"Your state once had the best education system in the country. From cradle to career, you took care of your children. You made sure they were ready to enter your universities or be productive participants in the workforce.

"I ask you, is California going to lead the race to the top, or are you going to lead the retreat?" [...]

California's fiscal crisis - in which schools are being forced to cut programs and lay off teachers - means the state has a long way to go before it is seen as a state that can show others how to do education right, he said.

"I have huge hopes for what California can do," he said. "I'd love to have California at the table, but California has things it needs to change."

Since the system of state-by-state standards has been such an undeniable catastrophe, Duncan believes the next big step should be national standards and national tests. In the meantime, Washington is calling its next $5 billion in educational funding a "Race to the Top," which is sadly apt; the terms of the stimulus pit the states against each other:

Duncan said the few states that win the competition—he wouldn't say how many that could be—will have to show that they are innovative and that their creative efforts succeed in helping low-performing students succeed. [...]

[San Francisco Superintendent Carlos Garcia] had asked Duncan if a good school district [...] might be able to win a piece of the $5 billion pie even if the state does not.

The answer was no.

For a guy who claims to be on a "listening and learning tour" looking for "grassroots input," aren't these remarks a little condescending? He even wants to find a way to peg teacher salaries to student scores, a great way to drain educators from schools that need the most help. In short, Duncan is clearly leaning towards a lot more of the same, repackaged:

[...] the nation's top educator also talked about changes he's expecting in the federal No Child Left Behind Education Act—including changing its Bush-era name, which he called "toxic." [...]

As for the law's new name, Duncan was unsure.

"It's got to be aspirational and inspirational," he said. "Maybe we could ask a smart 10-year-old for an idea." (full article)

Mr. Duncan was also on the evening news, where he offered at least one innovative idea for California:

"And again, one of the benefits of a tough economy is, you've got good scientists, you've got good chemists being laid off; let's get 'em into our classrooms."

That says something about his attitude toward our profession. Does Mr. Duncan think that anybody can be a teacher, as long as they don't have some other more important job to do? If we're all soon going to be teaching to a national test, he might actually be right about that. In any case there's no reason Duncan should know any better; he's never been a teacher. And neither has California State Superintendent Jack O'Connell, who was on hand for Duncan's visit:

"We're planning on not only competing, but participating and succeeding in the Race to the Top," he told The Chronicle. "I agree with (Duncan) that this is the chance of a lifetime."

The chance to do what exactly? Where could this "race" be heading? Duncan dropped another interesting hint during his visit:

"There's a really important saying: Never waste a good crisis. And at a moment of crisis you have an opportunity to think about some things very differently. We want to bring unprecedented resources to this state, but the state has to push a very strong reform agenda in return."

Nothing about this situation is unprecedented. California is in shock; it's our turn for disaster capitalism. Broke school systems are looking at the same basic deal the IMF used to force on places like Argentina: We promise we can help you get out of this mess if you show you're worthy of our support by making big, permanent sacrifices we're not specifying yet that may go against your own values. How come the Obama administration doesn't take this stance with the banking sector? And if we take their bait, how can we be sure we won't get the same treatment New Orleans' schools have received since Katrina? From the Institute for Southern Studies:

[...] many activists criticize what they see as token attempts at community involvement, and a paternalistic attitude among many of the new decision makers.

For example, our education system was in crisis pre-Katrina, and certainly needed revolutionary change. Change is what we have gotten – the current system is in many ways unrecognizable from the system of three years ago – but this revolution has been overwhelmingly led from outside, with little input from the parents, students and staff of the New Orleans school system.

Shortly after the post-Katrina evacuation of the city, the entire staff of the public school system was fired. Not long after that, school board officials chose to end recognition or negotiation with the teachers' union – the largest union in the city, and arguably the biggest outlet of Black middle class political power in the city. Since then, the school landscape has changed remarkably – from staff to decision-making structure to facilities. According to Tulane professor Lance Hill, "New Orleans has experienced a profound change in who governs schools and a dramatic reduction of parent and local taxpayer control of schools." (full article)

Finally, what can we learn from Mr. Duncan's seven-year track record as CEO of Chicago Public Schools?

Duncan has used three strategies to fix high schools: Close them down and replace them with new, smaller schools [...] fire school staff and reopen under new management [...] or infuse classrooms with new curriculum and materials [...] On all fronts, long-languishing, often-ignored high schools got some much needed attention. Also, education experts laud the focus that these efforts have placed on what goes on in the classroom.

But problems with high schools are so entrenched and intertwined with poverty that it is difficult to predict whether these efforts will be enough. (full article)

6. All of the Above

There are plenty of educators with much better ideas about how to tell what kids are learning. All we need to implement them is the political will, which brings us back to where we started. I said at the beginning that my purpose was to convince you that these tests give us all a bum deal. With luck, I've also convinced you we need to stop waiting for our government to cook us up some decent educational policy and start taking serious homegrown action for our students, our schools, our homes, and our country.

If you are convinced, tell someone else the story. Plenty of folks—including parents and teachers—are sadly misinformed. When fewer people fall sway to the myth that you can boil a young person's unique and dynamic intelligence down to a single number once a year and stick it to the fridge, we'll at least be able to have the fresh debates about public schooling we need. One of the biggest obstacles to education reform is that so many ideas from well-meaning people are based on deep, unquestioned assumptions that do real damage. We have much unlearning ahead.

I also said at the beginning that we should boycott the STAR tests. Such actions have been tried before but only on small scales. Getting your own school to opt out will not do it; you'll just be inviting a takeover like my old cross-town rivals experienced. A STAR boycott would need to be state-wide if not bigger. Fortunately, new social technologies make collective action faster, cheaper, and easier than ever. The means for families to participate are easily and legally within reach; what better "grassroots input" can we give these top-down mess-makers? Bold moves like the one I'm proposing always carry risks, but we can't get in much more trouble than we're in already. Fresh from a months-long budget battle which everyone lost, Schwarzenegger is back to his earlier plan of cutting the school year by a week—just about the amount of time it takes to prep for and administer the STAR. How much more do California's kids have to lose before we stop playing along with this high-stakes nonsense?

Mr. Duncan says he expects to see a "flourishing of innovation" welling up in our schools. I'm all for that—let's begin by scrapping useless standardized tests that don't tell us anything about our kids and wreak havoc with our livelihood. If participation in such a protest taught our kids something about civic responsibility and the need to protect their own future, so much the better—as long as we promise not to test them on it afterward.