Race-bias on SAT

[Unpublished editorial, December 15th, 2003.  See the conclusion of this post for published [in print] articles on the SAT and the College Board.]

I once chaperoned a group of inner-city boys from Washington, DC, on an overnight camping trip along the Appalachian Trial. Most of the kids, all teenagers, had never been out of the Beltway. Some of them couldn’t fall asleep at night because it was too quiet.

Here’s my point: It might be difficult for execs and test-writers at the College Board and the Educational Testing Service (who develops the SAT for the College Board) to realize, but there may be more than a few test-takers out there who have never seen a bird’s nest in a tree much less stepped foot in a forest, which is why the following analogy question published in the latest College Board practice-test pamphlet may be culturally biased:

WALRUS: OCEAN ::

(A) bird : nest
(B) elephant : zoo
(C) automobile : highway
(D) moose : forest

D is the correct answer, but I can see how some kids (particularly from large urban areas) could interpret an automobile “living” on a highway in the same way that a walrus lives in the ocean. Just how wrong would that child be? One-fourth of a correct answer, according to the College Board, who readily admits that there are no right or wrong answers on the SAT, just best answers.

Aha, what a wonderful little trick. The Board admits that certain answers are more correct than others while simultaneously eschewing the idea that one’s culture might play a significant role in eliminating the non-best answers in the first place.

For the record, walruses don’t actually live in the ocean in the same way that moose live in the forest. Walruses don’t venture very far from shore and never reach depths of more than 80 meters; in fact, walruses spend more than one-third of their time outside of the ocean, “hauled-out” on coastland or ice floes in gathering areas called ooglits by the Eskimos (to be culturally precise)—but who’s splitting hairs. Certainly not the moose.

Back to the test. Black children who are being raised at home in the kind of black vernacular English most whites experience only when reading a novel by Toni Morrison or Zora Neale Hurston—or lip synching a rap song by Jay-Z or 50-Cent—are certainly at a cultural disadvantage when taking the SAT. The College Board is obviously not responsible for creating these cultural differences, but to deny that such things play no role in how the test is written, designed, and scored is condescending. Telling blacks to get with the dominant culture—or to stop complaining—is equally condescending.

Why?

First, because such bi-polar (us versus them) notions are insensitive and racist. Second, because with a little work any one in education can design a test in standard written English that would give an advantage to certain kinds of kids based on race and/or class. The Board already does that, just not systematically.

For example, over the past 10 years or so ETS has included a few questions on every SAT of an overtly multi-cultural nature. The following sentences appeared in a critical reading passage in the same practice-test brochure as the walrus-analogy mentioned earlier. These sentences were written by a black American woman about her experiences in Ghana, East West Africa:

“The customs people, taxi drivers, policemen, were all intensely and beautifully Black . . . I felt far less an outsider than I had sometimes felt in California. As a Black person in a Black country, I was very much at home. . . . Any failure of mine, I was convinced, reflected badly on 400 million Black people throughout the world. . . In Ghana I became just a woman. I let down my burden of responsibility. . . . I was free to enjoy myself and be something I have often missed intensely in the years since I came [back to the U.S.]—ordinary.”

Your average white male test-taker is going to be at a cultural disadvantage when answering the 12 critical reading questions that come after this passage, which deal with very mature gender- and racial-identity issues.

Does the SAT contain questions that are culturally biased? Certainly. The real question should be: Is the bias on the test systematic, persistent, and statistically significant? And, if so, what can be done about it? These questions do not have multiple choice answers, which is probably why people from the College Board have done such a lousy job responding to their critics.

What can the College Board do?

First, the Board can start by not being so arrogant and defensive when well-informed critics challenge the test. Roy O. Freedle’s essay on race-bias on the SAT, which appeared in last spring’s Harvard Educational Review (“Correcting the SAT’s Ethnic and Social-Class Bias: A Method for Reestimating SAT Scores”), is a case in point. (Freedle worked for ETS for nearly 31 years, retiring in 1998. Freedle is not a disgruntled employee; the major thrust of his Harvard piece was to suggest how the SAT could be revised to include a score that might benefit some minorities who seem to do better, on average, on certain types of questions.)

The Board’s response to Freedle was summed up in a one-page, unsigned, web article, which called Freedle’s study “so flawed that its conclusions are misleading.” The anonymous writer called Freedle’s observations mired in technical problems and suggested that his conclusions of bias could be explained away by mere chance (randomly guessing answers). A more comprehensive reanalysis of Freedle’s data and a more thorough explanation of the Board’s response has been promised for 2004. Stay tuned.

The Board’s near total disregard for any challenges of bias smacks of hypocrisy (since the SAT is supposed to measure and celebrate critical reasoning skills) and flies in the face of common sense. The Board acts as if this test, which has become the Goliath of all tests, and the nearly 1.4 million annual test-takers exist in some kind of cultural vacuum where all variables can either be kept constant or explained away by chance.

Second, the Board should open up all their data for research and review. Make it all available on the web (minus students’ names, of course), including information about race, gender, and educational level of their internal test-writers and psychometricians, and exactly how a question winds up on the SAT. I know that the SAT is the most analyzed test in history. Give us examples of questions that don’t cut the bias-mustard, and why, so we’ll better understand and appreciate the final product.

And while they’re at it, I’d like to know the racial breakdown of the exponentially increasing numbers of test-takers who are being granted testing accommodations on the SAT, particularly now that the Board is no longer reporting scores to colleges with any kind of footnote next to the score indicating that the test was taken under non-standard conditions (time and a half, or double-time, for instance) for students with special needs.

I bet such data will indicate that whites—particularly rich whites— more than any other group are getting testing accommodations faster and more often than any other group—why?—because they can afford it and are better able to navigate the parameters the College Board requires at the student’s high school months before the student arrives at the testing center.

This kind of data is not a national security risk. Full disclosure will do more than anything else to raise the awareness of what areas need attention, revision, or systematic re-structuring.

Finally, the College Board can stop acting like coaching and expensive courses either don’t help or provide only modest gains to a student’s score. Baloney. There is a reason why the College Board provides links on its web-site to order its own practice tests and test-taking tips in book form ($19.95) or on CD-ROM ($395.00). Add to cart? The College Board knows—though they don’t want to publicly admit it—that their Goliath has spawned a multi-million dollar “beat the test” industry that is certainly not benefiting the kind of kid I took camping along the Appalachian Trial. Those kids could tell the College Board a thing or two about the concrete jungle, but—don’t worry—they won’t be tested on it.

Note: If you enjoyed this piece, check out these other published pieces on the SAT and the College Board:

Time to Think, oped, The New York Times

A Small Step in the Write Direction, oped, The Philadelphia Inquirer

New SAT Writing Section Scores Low, oped, The Christian Science Monitor

[Mark Franek is the dean of students at the William Penn Charter School in Philadelphia. He also teaches English.]