The History of Bivariate Ecological Regression in Judicial Politics

[Version 1.5]*

It is challenging to commence an organized analysis of the problems inherent in Segal and Spaeth’s Supreme Court decision-making literature. Indeed, one could enter this discussion from a number of areas. But rather than beginning where others have, I want to start with an entirely original point that is relevant to my own labors in this field: political science’s attempt to create a “bivariate ideology model.”

What is a bivariate ideology model? It is simply a mathematical model of decision making that estimates the relationship of a lone, single variable -- justice ideology -- upon the choices justices make in Supreme Court cases (votes on the merits). Five or ten years ago, it was common to find political scientists appealing to these models as “proof” of the primacy of politics over “law” and the exposure of a popular “mythology” surrounding Supreme Court judging.[1]  The creators of these models apparently still put forth these views.[2]  The truth, however, is that the bivariate ideology models created by political scientists never established the conclusions commonly attributed to them. For now, however, let us begin with an overview of the basic nature of these models as well as their history.

Any discussion of the history of the bivariate ideology model  in attitudinal literature must begin with Segal and Cover’s (1989) landmark article. It was this article that was said to provide the first systematic explanation of the voting behavior in civil liberties cases using an independent variable not derived from the justices’ votes (557). To accomplish this goal, attitudinal researchers created an empirical index called “Segal/Cover scores,” which were derived from the content of newspaper editorials appearing during justice confirmation hearings. The scholars coded editorials describing nominees as liberal or conservative and scaled the results, creating what in essence is a measure of each justice’s reputation for political bias at the time of his or her confirmation. Importantly, the scholars then decided to regress the scores not against the actual civil-liberties voting data that existed, but against a set of summary percentages derived from that data. In short, they regressed Segal/Cover scores against the justices’ percent-liberal ratings. The results of the regression showed a rather high correlation coefficient of 0.80 (561). Other scholars in 1995 joined in this example and created an updated analysis that again offered a robust correlation of 0.80 (Segal et. al., 1995).

Based upon these studies, many political scientists began concluding that the attitudinal model was now an empirically dominant explanation of justice voting behavior, and that justice ideology governed the bulk of the choices made in civil liberties cases.[3]  So popular did bivariate ecological regression become among judicial politics scholars that, even today, it continues to appear in the literature. Jeff Segal’s (2005) work, in fact, analyzes the Rehnquist Court with a bivariate ecological model that focuses exclusively on the 14 justices who served under William Rehnquist’s tenure as chief justice. He reports a correlation of 0.70 for civil liberties cases and 0.72 for the entire docket. There is also a bivariate ecological model that appears in Epstein, Knight and Martin’s recent work on civil rights voting (2004, 181; Figure 10.3). (Even the New York Times recently printed aggregated scatter plots).[4] Perhaps the best statement, however, of what political scientists thought they had proven with bivariate ecological regression can be found in Epstein and Knight’s (1998) work, which replicates the now-famous scatter plot and declares why it is relevant (35,36):

When it turned out that [Segal and Spaeth] could explain more than 60 percent of the variation in civil liberties votes based solely on the justices’ policy preferences, the researchers concluded that justices come to the bench with a set of policy preferences, which they pursue through their votes, at least in civil liberties cases.

Over the last five to ten years, many political scientists offered similar pronouncements.[5]  Of particular interest is Brisbin’s (1996) assessment of the evidence that appeared in political science’s top journal. It declared that the case in favor of Segal and Spaeth was so cogent that further study of the issue should actually cease (1004). It also gave to the APSR the following observation (1011):

If the fiction of a Court of law and not politics, like the tale of a fire breathing dragon, is now dead, why belabor it through further study? Perhaps it is because the dragon is dead, but like most dead reptiles, he is still twitching. So, for good measure, it is necessary to drive lances into him again and again and then draw and quarter him so that the heresy of a legal model of Supreme Court decision making cannot be regenerated.[6]

To understand why these bivariate ideology models are problematic, one must first understand the data that comprise them. The source is a large, publicly-available resource known as the Supreme Court Data Base, which contains voting and case data for every justice who served on the Court from 1946 through 2004.[7]  The format of the variable that observes the ideological choices of the justices is a simple binary entry coded with a "1" or “2” if the vote is liberal or not (Spaeth 1999, 69-72, 92). The total number of civil-liberties votes accounted for by this resources is over 31,049, covering 58 continuous years of Court activity by 32 justices. By aggregating this data into a handful of percentages and using the same as a dependent variable in a regression model, political scientists introduced ecological inference into their empirical analysis. This appears to have created confusion and exaggeration in the interpretation of model results. The truth is that Segal and Spaeth’s bivariate ideology model only accounts for about one-third of the level of explanation the researchers proclaimed. This is still a reasonable model, of course, but it is nowhere near the level of deconstruction many political scientists had proclaimed – and, in fact, supports a much more limited critique of the role that ideology plays in judging.  

---

[1]. Segal and Spaeth, The Attitudinal Model Revisited, 1, 8, 10 and 26-27.

[2]. Segal, Jeffrey. 2005. “The Rehnquist Court” Law & Courts. 15 (Spring): 14-17.

[3] Evidence of this is found in the following declarations: (1) “A prominent view, if not the prominent, view of U.S. Supreme Court decision making is the attitudinal model. It supposes that the ideological values of jurists provide the best predictors of their votes …” (Segal, et. al. 1995); (2). “[xx-get this quote]” (Peretti 1999, 105-111); (3) “Spaeth’s conclusion about the value of the attitudinal model is one echoed by many scholars of the judicial process, and not just those working in the area of decision making. ... Justices do not decide a priori to protect minority rights or to legitimate the ruling regime. Rather, they base their votes on their political ideologies, with a consequence being that liberal justices tend to protect minority interests, while conservative ones tend to legitimate the ruling regime” (Epstein 1995,  249-250); (4) “... attitude theory is still regarded by most judicial behavioralists as the most elegant and persuasive model for predicting appellate judge behavior” (Carp and Stidham 2002, 351); (5) “Among many political scientists, aspects of the attitudinal model have become a virtual truism” (Cross 1997, 251, 265); (6) “Today, few political scientists would dispute that, within their discipline, the leading approach to adjudication is the ‘attitudinal model,’ which hypothesizes that Supreme Court justices vote their political preferences or ideologies” (Feldman 2005, 89-90); and (7) “Indeed, these days it is difficult to argue credibly that the model utterly fails to perform its primary task. The evidence in support of its one observable implication – namely, that the policy preferences of the justices help predict their merits votes – is overwhelmingly in its favor” (Epstein 2003). See also, Gillman (2001, 465-466) (asserting that judicial behavioralist scholars believe that “law has almost no influence on the Justices” of the Supreme Court) and Brisbin (1996) (quoted, supra, p. 5).  But perhaps what says it best is Segal and Spaeth’s now famous (2002, 86; 1993, 65) summation of their research, “Rehnquist votes the way he does because he is extremely conservative; Marshall voted the way he did because he was extremely liberal.”

[4] See The New York Times, January 6, 2006.

[5] See Notes 12 and 15.

[6] The author continues later in the article: "If additional empirical analysis is coupled with a politically conscious interpretation of legal texts, judicial research could not just slay any claims for principled, legal models of Supreme Court decisions making, it could slay any prescriptive arguments that endeavor to separate legal decisions from politics. Using multiple levels of analysis deciphering the components of judicial attitudes, judicial scholars could in effect deconstruct any claim that American law is or can be a morally principled effort to write down the rules used to discipline political pathologies" (1014). (Although it is true that some of this exaggerated praise arose not only from the results of ecological regression, but from other models that Segal and Spaeth were producing, it is also true that if you take away the bivariate ecological model, not enough remains in the other models to support such an observation -- at least not empirically).

[7] There are several data sets available. See the Ulmer Project at: http://www.as.uky.edu/polisci/ulmerproject/

* copied paragraphs from manuscript version

Will Political Science Lose Out to ELS in Law Schools?

I have been doing some thinking lately. How secure would you say that the subfield of "judicial politics" or "public law" is in political science? Let's define these terms: I am specifically talking about the application of quantitative empirical study to the phenomenon of judicial behavior and legal institutions. What does the future of this endeavor look like for political science? For the reasons indicated below, it may not be all that bright.

First, let’s consider some relevant history before returning to the question. When political science began to study bureaucracy, a major portion of that inquiry turned itself into its own concern: "public administration." When political science began to analyze policy, a major portion of that inquiry turned itself into its own concern: "policy analysis."[1] When political science began studying trial courts and the criminal justice system, a major portion of that inquiry was taken over by criminal justice programs located within sociology departments or organized independently.

What is most disturbing about this trend is that, unlike philosophy, political science cannot claim as one of its virtues the creation of other fields of inquiry. That is, it is not the job of political science to create other departments. Philosophy's mission, on the other hand, is different. All science begins as philosophy, and, hence, the production of science is philosophy's expected byproduct.[2] But what happened to political science in the fields I mentioned is simply that the work produced by the discipline was unable to meet the needs of the subject. In short, these things were taken from political science; they were not birthed by it. Stated another way, because the policy analysis performed in political science departments was not fulfilling the needs of such an inquiry, the work was substantially overtaken and transformed by other scholarly organizations. The same is true with criminal justice.

Now let us return to the question. What will be the status of judicial politics and public law scholarship in political science in about 10 or 15 years? I ask this question for one simple reason: law schools are getting into the game. Legal scholarship is currently in the process of transforming itself into an empirical endeavor. Many law schools are hiring Ph.Ds and are talking about whether statistics should be a requirement for incoming students. I saw one debate recently about whether law schools should start offering their own Ph.D programs. Even popular disciplinary figures like Lee Epstein are now teaching in law school.

What is the long term implication of this? One view says "the more the merrier." But although I do think that "more is merry" from the standpoint of the production of knowledge, I also think it means that the future of "judicial politics" and "public law" is now destined to be secondary (if not tertiary) to that of future legal scholarship. Why? Some of the answer is familiar, some perhaps not: higher salaries, more resources and a greater appreciation for the phenomena being studied. One of the objections that people make to political science scholarship is that its authors are too far removed from the phenomena they study. Indeed, this claim is sometimes made in reverse when political scientists talk about legal “scholarship:” it is too invested in legal culture and the worth of its orthodoxy to produce scientific scholarship.  To see this problem, just listen to the two talk about judging. One of my central complaints with the Supreme Court decision making literature in political science is that too many of the authors: (a) don’t know enough legal doctrine; (b) don’t read the cases they model; (c) don’t have a firm understanding of legal epistemology when creating a priori theories about judging; and (d) wish (at times) to tear down legal culture as much as law schools are said to want to promote it.

Now I offer my caveats: I don’t mean all political scientists. I’m generalizing. (It’s a blog entry!). But my point is very simple: in the future, law schools may be in a better position to produce empirical models of legal phenomena because they are closer to the workings of the phenomena they study. And if I am right about this, in 10 years or so, the top readings that graduate students are assigned in political science departments may be from dual-degreed law school faculty -- or even worse, the students may be attending an empirical Ph.D program sponsored in law school itself.  

---

[1] Indeed, there was a great battle fought over whether policy analysis should be its own field, separate from political science. See Public Policy Analysis: An Introduction, William N. Dunn

[2] Witness the relatively recent birth of cognitive linguistics, which grew out of the concerns that occupied philosophy of language.  

What is Political Philosophy?

What is it that makes Rawls or Nozick “political philosophers,” but not Pat Buchanan or Newt Gingrich?  Is it because Nozick is more intelligent than Buchanan? I think the answer is clearly no for several reasons. First, some may say that Buchannan is quite sufficiently gifted with respect to intellectualism. He certainly can reason well with premises. Is it merely that one involves a different style of assertion? Both are simply conservative advocates who simply use different styles of espousal?  This view suggests that political philosophy is simply a form of fashion  for communication (a form of art, I suppose).

All of these views seem mistaken. The real difference seems to be the grounds of the starting point. One purports to reach a conclusion about the grounds or foundation of conservatism as a proof – as a construct unto itself – where the other is seen as reasoning from the assumption that the grounds need not be inspected, at least not philosophically.  When Pat Buchanan speaks, doesn’t he speak as a spokesman for something already adopted – sort of as an advertisement does? When Ronald Dworkin or Immanual Kant speaks, by contrast,  they are trying to demonstrate that starting point X lies on a ground unto itself. Although this, too, has policy implications, the nature of the craft purports to stand on its own making.  This means that Pat Buchannon's writings are really a form of applied philosophy.  It is not that Dworkin or Nozick could not enter the journalistic format and champion something. It is that, if they did, they would have to be applying philosophy instead of doing it. 

Segal-Cover Scores Statistically Insignificant for Some Voting Years

Version 1.1* Many scholars in the judicial politics world do not know this, so I thought I would make a quick note. During a paper I prepared for the 2006 Midwest meeting, I discovered something interesting: Segal/Cover scores on a few occasions actually generate statistically-insignificant parameter estimates for specific years of voting. Newspaper reputation is a statistically insignificant predictor for every civil liberties vote cast by justices in the years of 1950, 1954, 1964, and 1992 (95% confidence interval and a two tailed test). The p-values are also greater than .01 for the years 1949, 1951, 1952, 1953, 1965, 1968, 1991 and 1993.  The table below summarizes the results. The data comes from the Ulmer Project. It is a combination of Vinson Court and updated Supreme-Court data combined into a single, justice-centered data set using stata commands (available through Paul Collins and also the Law and Courts Newsletter).  What would attitudinal modelers make of this? Would they say that judging was so good in 1992 that no vote was influenced by justice ideology? Was that a special, vintage year or something? Better than the '69 Mets?  P-values for Segal/Cover Scores above .01 (civil liberties):
Year	P-value	Year	P-value	Year	P-Value
1949	.069	1953	.028	1968	.067
1950	.739	1954	.165	1991	.016
1951	.037	1964	.272	1992	.547
1952	.056	1965	.053	1993	.022

* The original version contained a spelling error in the title and a typo on the table. Both are corrected.

What is "Political Science?"

If someone asked whether political science was its own subject matter, we would need to clarify something: (1) do they mean it own activity (a methodological focus); or (2) do they simply mean its own topic of conversation? Let’s take the second one first. Clearly, political science is not distinct in what it talks about. Journalism talks about the same thing. So does history to some extent. Indeed, so do some in taverns on the weekends. But compare this to, say, statistics or engineering. Talking about these subjects seems to require its own expertise before the conversation is understood. Can you talk about engineering without first having to learn something about engineering as an activity? It is not that people in taverns do not talk about engineering, it is that, when the do, they are being “amateur” engineers. Can we say the same of political science? When plain people talk about politics, are they merely being amateur political scientists? Are journalists or historians when they talk about political phenomena just “winging it,” while the true masters of this subject dwell elsewhere? I doubt it. It seems to me that the conversation of politics is neither foreign to ordinary people or something that requires is own expertise to understand the conversation.

Having cast doubt upon the subject of whether politics is an exclusive conversation, let us consider the second question: is political science its own means of inquiry (the methods part)? That is, is political science its own intellectual or scientific craft? It seems to me that what political science is, is the application of history, political philosophy, statistics and psychology to the study of political phenomena. To be a political scientists requires you to be a little of something else. Too much of one at the expense of the others puts it in another box.

But how does this compare with other “disciplines?” History appears not to be its own topic of conversation, but it does appear to be its own craft. So, too, for psychology. But then again, I must be careful: isn’t statistics merely the borrowing of the craft of mathematics, which borrows from arithmetic? Here is the difference: statistics is a distinct kind of applied mathematics, as is, say, algebra and calculus. This is not true of political science. What it borrows is not an activity unique to itself. So the point is this: political science appears to have neither its own conversation nor its own activity. And so I must now ask: are we even a “science?” Or are we simply a collection of educated people who apply someone else’s science to an ordinary conversation?