Could Sonia Sotomayor have moved up the ranks in the New Haven Fire Department? The stark contrast between a Hispanic lawyer who has risen to the top of the legal profession and a group of African-American firefighters whose efforts to move up have been stymied by a test points to how differently equal opportunity has played out in education and in employment.

Educational and employment tests rose to prominence after the civil rights revolution of the 1960s. They signaled that a college or employer valued ability and was committed to equal treatment. With the SAT, the New Haven fire captain’s exam, or the popular General Aptitude Test Battery, each candidate’s numerical score could be plotted on a curve. A scientific judgment could be made about who was most able, most meritorious, and most deserving of a desk in the classroom or a seat on the fire truck.

While both kinds of tests were popularized by equal opportunity laws, educational tests have had a very different legal and practical history than employment tests. Berkeley and the University of Michigan and Yale and Harvard took SAT test scores with a grain of salt, in part because data from millions of test-takers showed that the test does not predict college performance very well for African-Americans. When elite colleges made a concerted effort to diversify the student body in the 1960s, they avoided hard numerical cutoffs for SAT performance and instead took a range of different indictors, like high school grades, into account. Non-elite colleges were admitting almost all comers, so the test mattered little for the average college applicant.

Many companies responded to the Civil Rights Act of 1964 by installing hiring and promotion tests on the theory that these could guarantee that placement was based on merit, not bias. While police and fire departments came to rely heavily on tests, the relationship between test and job performance was problematic, like the relationship between SAT and college performance. Employers often used general ability tests that covered basic high school math and English skills for jobs in auto factories and airplane plants that didn’t require these skills. People who do well on the GATB can make lousy bricklayers. The disconnect between test scores and job performance was the basis of a the Supreme Court’s landmark Griggs v. Duke Power decision in 1971, in which African-American plaintiffs argued that they had been denied posts at Duke Power because they scored poorly on a test that measured skills unrelated to the job. Substandard segregated high schools had prepared African-Americans poorly for such tests, and so many promising candidates for jobs botched the test. The Court ruled that hiring and promotion practices that have a “disparate impact” on certain groups, and serve no business purpose, can be discriminatory.

Personnel psychologists echoed the court’s call for tests to be validated — statistically proven to predict job performance. Lower courts encouraged validation, but Washington made no hard and fast rule that employers must validate job tests. As a result, HR departments sometimes used psychologists’ stringent criteria to validate employment tests, but more often they used guesswork or, as one HR manager told me in an interview, decided they would “throw themselves on the mercy of the court” if they were sued for using a discriminatory test. Even big city police and fire departments, many of whom were under court order to integrate, continued to use tests that had not been validated, and that were known to exclude almost all African-American or Latino candidates. New Haven’s captain’s test had not been validated. While some police and fire departments followed colleges in using supplemental criteria for selection, such as performance in a mock fire, many did something like New Haven, which gave the multiple choice exam a weight of 60% and an oral exam a weight of 40%. In effect, no one below a certain threshold could win a lieutenancy or captaincy.

In colleges, we know that the SAT isn’t a perfect predictor of success, and so we don’t exclude all candidates below a particular score. In fire departments, which use any number of different tests, fire chiefs frequently have no idea of whether the exam is a good predictor of job performance, and yet they rule out candidates below a particular score.

Would Sonia Sotomayor have passed the New Haven lieutenant’s exam? Perhaps, though there are some reasons to think she might not have. She reports that her SAT scores for Princeton and her LSAT scores for Yale were not on par with those of her peers, and that she had to teach herself grammar and vocabulary and the classics when she got to Princeton. She likely wouldn’t have gotten into either Princeton or Yale if they had used hard cutoffs, but that didn’t stop her from graduating summa cum laude from Princeton and making the Law Review at Yale. She may have done a middling job on the SAT for any number of reasons. Perhaps growing up in a housing project in the Bronx didn’t give the preparation that a childhood in Scarsdale and at Exeter Academy might have. Perhaps learning English as a second language put her at a disadvantage. Or perhaps she choked. Social psychologists find that minorities often blow tests when they are thinking of their status, and excel when they aren’t.

Soon to be Justice Sotomayor hoped as a child to become a detective. If she had followed that dream, she might well have struggled with the police department’s detective exam and been stuck as a beat cop. The world of policing would have lost a natural leader, and a great intellect. Police and fire departments could stand to learn one thing from colleges and universities. If multiple choice tests don’t predict college test-taking very well, they surely don’t predict policing and firefighting very well. Chiefs would be well advised to look at a wider range of criteria, as Princeton and Yale did in the case of Sonia Sotomayor.

Frank Dobbin is the author of Inventing Equal Opportunity

Frank Dobbins: Frank Dobbin is professor of sociology at Harvard. He has studied corporate equal opportunity and diversity programs for more than two decades. In his most recent work, with Alexandra Kalev, he is developing an evidence-based approach to diversity management, using a large sample of firms and thirty years of data to analyze the effects of popular diversity programs on workforce integration. His Inventing Equal Opportunity (Princeton University Press, 2009) traces the evolution of corporate equal opportunity and diversity programs put into place by human resources managers – programs that ultimately define discrimination in the American mind.