Unit No. 10:

Pitfalls of Comparative Research?

In This part of the Course we will discuss few problmes of case-oriented research as reflected in recent debate between John H. Goldthrope and Charles Ragin. The debate was published in the framework of a special issue of Comparative Social Research (Vo. 16, 1997). The new issues that will be prsented here include:

1. The Problem of Selection Bias:

Selection bias is commonly understood as occurring when the nonrandom selection of cases result in inferences, based on the resulting sample, that are not statistically representative of the population. The focus of the present discussion is on selection bias deriving from deliberate selection by the investigator. A common problem arising from such selection is that it may over-represent cases at one or the other end of the distribution on a key variable.

The statistical insight crucial, to understanding the consequences of such selection, is the observation that selecting cases - so as to constrain variation toward high or low values of the dependent variable - tends to reduce the slope estimate produced by regression analysis. This is the basis for warning about the hazards of "selecting on the dependent variable". This expression refers, not only to the deliberate selection of cases according to their scores on this variable, but to any mode of selection correlated with the dependent variable (i.e., tending to select cases that have higher, or lower, values on that variable) once the effect of the explanatory variable is removed. If such a correlation exists, causal inference will tend to be biased (Coolier, 1995, 461).


2. The small N Probelm:

The small N problem arises in that, if nations are taken as units of analysis, the number available for study is likely to be quite limited. Where individuals are the units, populations can be sampled so as to give Ns of several hundreds or thousands; but where nations are the units, N cannot rise much above one hundred even if all available cases are taken, and is often far less. In applying techniques of multivariate analysis, serious difficulties tend therefore to be encountered in that N is not much greater than the total number of variables involved. Statistically, this means that there are too few degrees of freedom, that models become "over-determined," that inter-correlations among independent variables cannot be adequately dealt with and that results may not be robust. Substantively, it means that competing explanations of the dependent variable may not be open to any decisive evaluation. Thus, it has been recently claimed (Huber, Ragin, and Stephens, 1993) that, for just these reasons, the research program on the determinants of state welfare provision - in which analyses based on a maximum of c. 20 nations have been typical - has by now reached a virtual "impasse". (Goldthrope, 1997, 5).

3. The Black-box Problem:

A quantitative analysis may be undertaken which is successful in "accounting for" a significant part of the variation in the phenomenon of interest - let us say, the sizes of welfare states. But such an analysis, it can be objected, still tells us rather little about just what is going on at the level of the social processes and action that underlie, as it were, the interplay of the variables that have been distinguished. We know the "inputs" to the analysis and we know the "outputs" from it; but we do not know much about why it should be that, within the black box of the statistical model that is applied, the one is transformed into the other. The problem is of course mitigated if "intervening" as well as independent variables are included in the analysis, so as to give it a more finely-grained character; and further if both independent and intervening variables are chosen on theoretical grounds, so that certain causal processes may at least be implied. None the less, it can still be maintained that the black box problem is seriously addressed only to the extent that such processes are spelt out quite explicitly, so as to provide a "causally adequate" account of the actual generation of the regularities that are empirically demonstrated.

4. The Galton Problem:

The "Galton" problem is named after the nineteenth-century British Francis Galton who famously criticized a pioneering comparative analysis by the anthropologist, Edward Tylor. Tylor claimed to show complex correlations among economic and familial institutions across a wide range of societies, past and present. These correlations he then sought to explain from what we would now think of as a functionalist standpoint. Galton questioned the extent to which Tylor's observations were independent ones, and pointed out that "institutional" correlations might arise not only under the pressure of functional exigencies, or through other processes operating within societies. They might also be the result of processes of what we would now call cultural diffusion among societies. The Galton problem could be regarded as potentially more damaging at the present time than ever before. Claims that the treatment of nations as independent units of analysis seem to be increasingly untenable in a global world.


Reading Assignment:

Read the following texts:

1. A debate on Mthodological Issues, Option: Download in Word Format

2. Turning The Tables: A challenge to variable Oriented Approach, Option: Download in Word Format



If you need the advise of the teacher write e-mail to levi@poli.haifa.ac.il

If you want to address the class write e-mail to method@research.haifa.ac.il

Do you have difficulties with some words? Use the Webster Dictionary on-line Or use / download the Babylon Translation Software

Return to previous page      Return to Class Home Page