How Ecological Inference Corrupts an Ideology Model

[version 1.1]*

In my last entry, I demonstrated that the bivariate ideology models constructed by judicial politics scholars over the last sixteen years had the unfortunate property of introducing ecological inference into the regression analysis. One may wonder why scholars did this to their models given the fact that there was no reason to do so (at least not since the mid 1990s). That is a topic for another day, however. For now, I want to consider a more direct question: what is “wrong” with relying upon ecological inference in a bivariate ideology model?

Although there are many problems with models that aggregate voting data, I want to focus upon one exclusive phenomenon in this entry: goodness of fit and model misspecification. I’ll hit the other problems in my next entry.

A. Goodness-of-Fit and Modeling Flaws

The best way to demonstrate the fit problem is with an example, followed by an interpretation. Let us assume that there are two hypothetical courts, Alpha and Beta, each with five justices who have the following voting data: 

Alpha Court:				Beta Court
Justice	N	Pct. L	Segal/ Cover	Justice	N	Pct. L	Segal/ Cover
Rove	10	.10	.10	Rove	10	.10	.10
O’Connor	10	.30	.30	Drudge	10	.10	.30
Stewart	10	.50	.50	“Teddy”	10	.90	.50
Clinton	10	.70	.70	Jesse	10	.90	.70
Nader	10	.90	.90	Nader	10	.90	.90

The difference between these two courts is that one has a distribution of liberal votes that is symmetrical, the other is polarized. The Alpha Court is anchored by two extreme justices, followed by those of proximate distance and a centrist. The Beta Court, by contrast, is plagued with two extreme “clans.” In both cases, however, the hypothetical Segal-Cover scores are the same. That is, in the case of Alpha, the scores correlate perfectly with the percentage of liberal votes cast -- think of it as the attitudinal model in heaven -- but in the case of Beta, newspaper editorials were simply not as accurate of a forecast.

If one were to regress the Segal-Cover scores against the votes in each of these courts, what do you think the difference would be in the goodness-of-fit for an aggregated versus non-aggregated model? The answer may surprise you. In the logit model, goodness of fit is around .5 for the Alpha Court (symmetrical) and .8 for Beta (polarized).[1]  However, in the aggregated model, goodness of fit is a perfect 1.0 for the Alpha court and .47 for Beta.[2]  In other words, the fit of the models is moving in an opposite direction.

And now the critical question: how come the aggregated model cannot properly distinguish between symmetrical and polarized voting for purposes of fit like the non-aggregated model can? That is, how can a model of bias report that a voting universe is less explained by the presence of bias when it is dominated by two extreme clans – the “Rehnquist Five” logic – versus when it has symmetrical variety? The answer is straight forward: when data exists in a binary format, cases that do not affiliate well with dichotomous outcomes – those that show little to no favoritism for “0s” or “1s” – are interpreted as not fitting the model framework well. Hence, Justice Stewart on the Alpha Court is pulling down the model’s fit. However, in an OLS regression of continuous-level data, median cases of X only fail to fit their model if they have extreme Y values (are outliers). Hence, “Teddy” on the Beta Court is pulling down the fit of the OLS regression. (Picture a scatter plot: Teddy is the highest from the line. I have the data posted below if you want to play with it).

So what do we make of this? The point is that the only way a median justice will fail to fit very well into an ecological regression that is already anchored by two extremes is if he or she is so extremely directional (biased) as to be an outlier (Teddy). Yet, if that same voting pattern happens in the logit regression, goodness of fit would increase, not decrease, because the more movement one sees “out of the middle,” the better those models perform. Hence, what aggregation does is it transforms poorly fitting cases in one model into perfectly fitting cases in the other. Stated another way, it transforms non-directional justices into optimally-biased justices.

Given what I have just demonstrated, it should be quite clear why aggregating votes is fundamentally objectionable from the standpoint of both measurement logic and model specification. Quite simply, a model that transforms median justices who do not affiliate with an observed measure of bias into cases that actually “jack up” the model’s assessment of how well bias explains the voting universe is nothing other than a kind of sophistry masquerading behind statistical software. It is sophistry because: (a) bias is supposed to be an observed, empirical phenomenon, not a manufactured one; and (b) a model predicated upon the idea that a median-measured justice could lower the overall picture of bias in a voting universe by becoming an extremist is simply an invalid theoretical design.

Some might be tempted to argue, however, that non-directional justices have “moderate ideology,” and that this is their true “bias.” The argument would be that aggregation is good because median-measured justices should bolster fit unless they become extreme. The reply to this view is straight forward: Segal and Spaeth do not have a criteria for observing moderation as a political subject matter at the case level. That is exactly what the whole objection is. What is a vote for moderate ideology? To determine whether non-directional justices are, in fact, expressing preference for a political subject matter that is different from liberalism or conservatism, one would need to observe it with a trichotomous variable that provides acceptable coding criteria for the three distinct ideological choices. Or, one would need to create a continuous level measure of quality liberalism for each choice available to a justice (McGuire and Vanberg). To date, neither of these options have materialized. Even if they ever do, it is doubtful that such an innovation will help the fit of ideology models. The reason is that the justices who we think are liberal and conservative may “defect” quite regularly for centrist alternatives. If this happens with any regularity, the goodness of fit will not be as high as some in our field would like.

Therefore, transforming justices who systematically resist a measure of bias into perfectly-biased justices through the magic of aggregation is a most objectionable way to conduct empirical analysis of the data that is currently available. In short, these ecological models are misspecified.  No longer can political scientists assert as an empirical matter that 60-to-80% of the choices justices make in civil liberties cases arise out of their political values -- at least not to the extent that researchers have observed such phenomena in a data set. There is absolutely no empirical truth in that assertion whatsoever.

OUTPUT FOR ALPHA AND BETA: 

The STATA file:  http://ludwig.squarespace.com/storage/experiment.dta;  Goodness-of-fit tables for the logistic regressions: http://ludwig.squarespace.com/storage/table.alpha-beta.doc

REFERENCES:

McGuire, Kevin T., and George Vanberg. 2005. Mapping the Policies of the U.S. Supreme Court: Data, Opinions, and Constitutional Law, paper presented at the Annual Meeting of the American Political Science Association.  

---

[1] Logit is estimated with maximum likelihood. The only way to achieve a 1.0 (perfect) goodness of fit in a logit model is if the classification table perfectly predicts complete polarization. There would be no classification errors whatsoever.

[2] For those wanting more information, the logit classification table appears at the end of this journal entry. Fit is assessed with the R-squared analogues of phi-p and tau-p.

* corrected the spelling error in the title; minor editing in the final paragraph.

The History of Bivariate Ecological Regression in Judicial Politics

[Version 1.5]*

It is challenging to commence an organized analysis of the problems inherent in Segal and Spaeth’s Supreme Court decision-making literature. Indeed, one could enter this discussion from a number of areas. But rather than beginning where others have, I want to start with an entirely original point that is relevant to my own labors in this field: political science’s attempt to create a “bivariate ideology model.”

What is a bivariate ideology model? It is simply a mathematical model of decision making that estimates the relationship of a lone, single variable -- justice ideology -- upon the choices justices make in Supreme Court cases (votes on the merits). Five or ten years ago, it was common to find political scientists appealing to these models as “proof” of the primacy of politics over “law” and the exposure of a popular “mythology” surrounding Supreme Court judging.[1]  The creators of these models apparently still put forth these views.[2]  The truth, however, is that the bivariate ideology models created by political scientists never established the conclusions commonly attributed to them. For now, however, let us begin with an overview of the basic nature of these models as well as their history.

Any discussion of the history of the bivariate ideology model  in attitudinal literature must begin with Segal and Cover’s (1989) landmark article. It was this article that was said to provide the first systematic explanation of the voting behavior in civil liberties cases using an independent variable not derived from the justices’ votes (557). To accomplish this goal, attitudinal researchers created an empirical index called “Segal/Cover scores,” which were derived from the content of newspaper editorials appearing during justice confirmation hearings. The scholars coded editorials describing nominees as liberal or conservative and scaled the results, creating what in essence is a measure of each justice’s reputation for political bias at the time of his or her confirmation. Importantly, the scholars then decided to regress the scores not against the actual civil-liberties voting data that existed, but against a set of summary percentages derived from that data. In short, they regressed Segal/Cover scores against the justices’ percent-liberal ratings. The results of the regression showed a rather high correlation coefficient of 0.80 (561). Other scholars in 1995 joined in this example and created an updated analysis that again offered a robust correlation of 0.80 (Segal et. al., 1995).

Based upon these studies, many political scientists began concluding that the attitudinal model was now an empirically dominant explanation of justice voting behavior, and that justice ideology governed the bulk of the choices made in civil liberties cases.[3]  So popular did bivariate ecological regression become among judicial politics scholars that, even today, it continues to appear in the literature. Jeff Segal’s (2005) work, in fact, analyzes the Rehnquist Court with a bivariate ecological model that focuses exclusively on the 14 justices who served under William Rehnquist’s tenure as chief justice. He reports a correlation of 0.70 for civil liberties cases and 0.72 for the entire docket. There is also a bivariate ecological model that appears in Epstein, Knight and Martin’s recent work on civil rights voting (2004, 181; Figure 10.3). (Even the New York Times recently printed aggregated scatter plots).[4] Perhaps the best statement, however, of what political scientists thought they had proven with bivariate ecological regression can be found in Epstein and Knight’s (1998) work, which replicates the now-famous scatter plot and declares why it is relevant (35,36):

When it turned out that [Segal and Spaeth] could explain more than 60 percent of the variation in civil liberties votes based solely on the justices’ policy preferences, the researchers concluded that justices come to the bench with a set of policy preferences, which they pursue through their votes, at least in civil liberties cases.

Over the last five to ten years, many political scientists offered similar pronouncements.[5]  Of particular interest is Brisbin’s (1996) assessment of the evidence that appeared in political science’s top journal. It declared that the case in favor of Segal and Spaeth was so cogent that further study of the issue should actually cease (1004). It also gave to the APSR the following observation (1011):

If the fiction of a Court of law and not politics, like the tale of a fire breathing dragon, is now dead, why belabor it through further study? Perhaps it is because the dragon is dead, but like most dead reptiles, he is still twitching. So, for good measure, it is necessary to drive lances into him again and again and then draw and quarter him so that the heresy of a legal model of Supreme Court decision making cannot be regenerated.[6]

To understand why these bivariate ideology models are problematic, one must first understand the data that comprise them. The source is a large, publicly-available resource known as the Supreme Court Data Base, which contains voting and case data for every justice who served on the Court from 1946 through 2004.[7]  The format of the variable that observes the ideological choices of the justices is a simple binary entry coded with a "1" or “2” if the vote is liberal or not (Spaeth 1999, 69-72, 92). The total number of civil-liberties votes accounted for by this resources is over 31,049, covering 58 continuous years of Court activity by 32 justices. By aggregating this data into a handful of percentages and using the same as a dependent variable in a regression model, political scientists introduced ecological inference into their empirical analysis. This appears to have created confusion and exaggeration in the interpretation of model results. The truth is that Segal and Spaeth’s bivariate ideology model only accounts for about one-third of the level of explanation the researchers proclaimed. This is still a reasonable model, of course, but it is nowhere near the level of deconstruction many political scientists had proclaimed – and, in fact, supports a much more limited critique of the role that ideology plays in judging.  

---

[1]. Segal and Spaeth, The Attitudinal Model Revisited, 1, 8, 10 and 26-27.

[2]. Segal, Jeffrey. 2005. “The Rehnquist Court” Law & Courts. 15 (Spring): 14-17.

[3] Evidence of this is found in the following declarations: (1) “A prominent view, if not the prominent, view of U.S. Supreme Court decision making is the attitudinal model. It supposes that the ideological values of jurists provide the best predictors of their votes …” (Segal, et. al. 1995); (2). “[xx-get this quote]” (Peretti 1999, 105-111); (3) “Spaeth’s conclusion about the value of the attitudinal model is one echoed by many scholars of the judicial process, and not just those working in the area of decision making. ... Justices do not decide a priori to protect minority rights or to legitimate the ruling regime. Rather, they base their votes on their political ideologies, with a consequence being that liberal justices tend to protect minority interests, while conservative ones tend to legitimate the ruling regime” (Epstein 1995,  249-250); (4) “... attitude theory is still regarded by most judicial behavioralists as the most elegant and persuasive model for predicting appellate judge behavior” (Carp and Stidham 2002, 351); (5) “Among many political scientists, aspects of the attitudinal model have become a virtual truism” (Cross 1997, 251, 265); (6) “Today, few political scientists would dispute that, within their discipline, the leading approach to adjudication is the ‘attitudinal model,’ which hypothesizes that Supreme Court justices vote their political preferences or ideologies” (Feldman 2005, 89-90); and (7) “Indeed, these days it is difficult to argue credibly that the model utterly fails to perform its primary task. The evidence in support of its one observable implication – namely, that the policy preferences of the justices help predict their merits votes – is overwhelmingly in its favor” (Epstein 2003). See also, Gillman (2001, 465-466) (asserting that judicial behavioralist scholars believe that “law has almost no influence on the Justices” of the Supreme Court) and Brisbin (1996) (quoted, supra, p. 5).  But perhaps what says it best is Segal and Spaeth’s now famous (2002, 86; 1993, 65) summation of their research, “Rehnquist votes the way he does because he is extremely conservative; Marshall voted the way he did because he was extremely liberal.”

[4] See The New York Times, January 6, 2006.

[5] See Notes 12 and 15.

[6] The author continues later in the article: "If additional empirical analysis is coupled with a politically conscious interpretation of legal texts, judicial research could not just slay any claims for principled, legal models of Supreme Court decisions making, it could slay any prescriptive arguments that endeavor to separate legal decisions from politics. Using multiple levels of analysis deciphering the components of judicial attitudes, judicial scholars could in effect deconstruct any claim that American law is or can be a morally principled effort to write down the rules used to discipline political pathologies" (1014). (Although it is true that some of this exaggerated praise arose not only from the results of ecological regression, but from other models that Segal and Spaeth were producing, it is also true that if you take away the bivariate ecological model, not enough remains in the other models to support such an observation -- at least not empirically).

[7] There are several data sets available. See the Ulmer Project at: http://www.as.uky.edu/polisci/ulmerproject/

* copied paragraphs from manuscript version

Segal-Cover Scores Statistically Insignificant for Some Voting Years

Version 1.1* Many scholars in the judicial politics world do not know this, so I thought I would make a quick note. During a paper I prepared for the 2006 Midwest meeting, I discovered something interesting: Segal/Cover scores on a few occasions actually generate statistically-insignificant parameter estimates for specific years of voting. Newspaper reputation is a statistically insignificant predictor for every civil liberties vote cast by justices in the years of 1950, 1954, 1964, and 1992 (95% confidence interval and a two tailed test). The p-values are also greater than .01 for the years 1949, 1951, 1952, 1953, 1965, 1968, 1991 and 1993.  The table below summarizes the results. The data comes from the Ulmer Project. It is a combination of Vinson Court and updated Supreme-Court data combined into a single, justice-centered data set using stata commands (available through Paul Collins and also the Law and Courts Newsletter).  What would attitudinal modelers make of this? Would they say that judging was so good in 1992 that no vote was influenced by justice ideology? Was that a special, vintage year or something? Better than the '69 Mets?  P-values for Segal/Cover Scores above .01 (civil liberties):
Year	P-value	Year	P-value	Year	P-Value
1949	.069	1953	.028	1968	.067
1950	.739	1954	.165	1991	.016
1951	.037	1964	.272	1992	.547
1952	.056	1965	.053	1993	.022

* The original version contained a spelling error in the title and a typo on the table. Both are corrected.