Download WHAT IS RESEARCH DESIGN? 1 and more Study notes Design Patterns in PDF only on Docsity!
PART I
WHAT IS RESEARCH DESIGN?
THE CONTEXT OF DESIGN
Before examining types of research designs it is important to be clear about the role and purpose of research design. We need to understand what research design is and what it is not. We need to know where design Æts into the whole research process from framing a question to Ænally analysing and reporting data. This is the purpose of this chapter.
Description and explanation
Social researchers ask two fundamental types of research questions:
1 What is going on (descriptive research)? 2 Why is it going on (explanatory research)?
Descriptive research
Although some people dismiss descriptive research as `mere descrip- tion', good description is fundamental to the research enterprise and it has added immeasurably to our knowledge of the shape and nature of our society. Descriptive research encompasses much government spon- sored research including the population census, the collection of a wide range of social indicators and economic information such as household expenditure patterns, time use studies, employment and crime statistics and the like. Descriptions can be concrete or abstract. A relatively concrete descrip- tion might describe the ethnic mix of a community, the changing age proÆle of a population or the gender mix of a workplace. Alternatively
the description might ask more abstract questions such as Is the level of social inequality increasing or declining?',How secular is society?' or How much poverty is there in this community?' Accurate descriptions of the level of unemployment or poverty have historically played a key role in social policy reforms (Marsh, 1982). By demonstrating the existence of social problems, competent description can challenge accepted assumptions about the way things are and can provoke action. Good description provokes thewhy' questions of explanatory research. If we detect greater social polarization over the last 20 years (i.e. the rich are getting richer and the poor are getting poorer) we are forced to ask Why is this happening?' But before askingwhy?' we must be sure about the fact and dimensions of the phenomenon of increasing polarization. It is all very well to develop elaborate theories as to why society might be more polarized now than in the recent past, but if the basic premise is wrong (i.e. society is not becoming more polarized) then attempts to explain a non-existent phenomenon are silly. Of course description can degenerate to mindless fact gathering or what C.W. Mills (1959) called abstracted empiricism'. There are plenty of examples of unfocused surveys and case studies that report trivial information and fail to provoke anywhy' questions or provide any basis for generalization. However, this is a function of inconsequential descriptions rather than an indictment of descriptive research itself.
Explanatory research
Explanatory research focuses on why questions. For example, it is one thing to describe the crime rate in a country, to examine trends over time or to compare the rates in different countries. It is quite a different thing to develop explanations about why the crime rate is as high as it is, why some types of crime are increasing or why the rate is higher in some countries than in others. The way in which researchers develop research designs is funda- mentally affected by whether the research question is descriptive or explanatory. It affects what information is collected. For example, if we want to explain why some people are more likely to be apprehended and convicted of crimes we need to have hunches about why this is so. We may have many possibly incompatible hunches and will need to collect information that enables us to see which hunches work best empirically. Answering the `why' questions involves developing causal explana- tions. Causal explanations argue that phenomenon Y (e.g. income level) is affected by factor X (e.g. gender). Some causal explanations will be simple while others will be more complex. For example, we might argue that there is a direct effect of gender on income (i.e. simple gender discrimination) (Figure 1.1a). We might argue for a causal chain, such as that gender affects choice of Æeld of training which in turn affects
2 WHAT IS RESEARCH DESIGN?
Students at fee paying private schools typically perform better in their Ænal year of schooling than those at government funded schools. But this need not be because private schools produce better performance. It may be that attending a private school and better Ænal-year performance are both the outcome of some other cause (see later discussion). Confusing causation with correlation also confuses prediction with causation and prediction with explanation. Where two events or charac- teristics are correlated we can predict one from the other. Knowing the type of school attended improves our capacity to predict academic achievement. But this does not mean that the school type affects aca- demic achievement. Predicting performance on the basis of school type does not tell us why private school students do better. Good prediction does not depend on causal relationships. Nor does the ability to predict accurately demonstrate anything about causality. Recognizing that causation is more than correlation highlights a problem. While we can observe correlation we cannot observe cause. We have to infer cause. These inferences however are `necessarily fallible... [they] are only indirectly linked to observables' (Cook and Campbell, 1979: 10). Because our inferences are fallible we must minimize the chances of incorrectly saying that a relationship is causal when in fact it is not. One of the fundamental purposes of research design in explanatory research is to avoid invalid inferences.
Deterministic and probabilistic concepts of causation
There are two ways of thinking about causes: deterministically and probabilistically. The smoker who denies that tobacco causes cancer because he smokes heavily but has not contracted cancer illustrates deterministic causation. Probabilistic causation is illustrated by health authorities who point to the increased chances of cancer among smokers. Deterministic causation is where variable X is said to cause Y if, and only if, X invariably produces Y. That is, when X is present then Y will necessarily, inevitably and infallibly' occur (Cook and Campbell, 1979: 14). This approach seeks to establish causal laws such as: whenever water is heated to 100 æC it always boils. In reality laws are never this simple. They will always specify par- ticular conditions under which that law operates. Indeed a great deal of scientiÆc investigation involves specifying the conditions under which particular laws operate. Thus, we might say that at sea level heating pure water to 100 æC will always cause water to boil. Alternatively, the law might be stated in the form ofother things being equal' then X will always produce Y. A deterministic version of the relationship between race and income level would say that other things being equal (age, education, personality, experience etc.) then a white person will [always] earn a higher income than a black person. That is, race (X) causes income level (Y).
4 WHAT IS RESEARCH DESIGN?
Stated like this the notion of deterministic causation in the social sciences sounds odd. It is hard to conceive of a characteristic or event that will invariably result in a given outcome even if a fairly tight set of conditions is speciÆed. The complexity of human social behaviour and the subjective, meaningful and voluntaristic components of human behaviour mean that it will never be possible to arrive at causal statements of the type `If X, and A and B, then Y will always follow.' Most causal thinking in the social sciences is probabilistic rather than deterministic (Suppes, 1970). That is, we work at the level that a given factor increases (or decreases) the probability of a particular outcome, for example: being female increases the probability of working part time; race affects the probability of having a high status job. We can improve probabilistic explanations by specifying conditions under which X is less likely and more likely to affect Y. But we will never achieve complete or deterministic explanations. Human behaviour is both willed and caused: there is a double-sided character to human social behaviour. People construct their social world and there are creative aspects to human action but this freedom and agency will always be constrained by the structures within which people live. Because behav- iour is not simply determined we cannot achieve deterministic explana- tions. However, because behaviour is constrained we can achieve probabilistic explanations. We can say that a given factor will increase the likelihood of a given outcome but there will never be certainty about outcomes. Despite the probabilistic nature of causal statements in the social sciences, much popular, ideological and political discourse translates these into deterministic statements. Findings about the causal effects of class, gender or ethnicity, for example, are often read as if these factors invariably and completely produce particular outcomes. One could be forgiven for thinking that social science has demonstrated that gender completely and invariably determines position in society, roles in families, values and ways of relating to other people.
Theory testing and theory construction
Attempts to answer the `why' questions in social science are theories. These theories vary in their complexity (how many variables and links), abstraction and scope. To understand the role of theory in empirical research it is useful to distinguish between two different styles of research: theory testing and theory building (Figure 1.2).
Theory building
Theory building is a process in which research begins with observations and uses inductive reasoning to derive a theory from these observations.
THE CONTEXT OF DESIGN 5
if the theory is true then certain things should follow in the real world. We then assess whether these predictions are correct. If they are correct the theory is supported. If they do not hold up then the theory needs to be either rejected or modiÆed. For example, we may wish to test the theory that it is not divorce itself that affects the wellbeing of children but the level of conØict between parents. To test this idea we can make predictions about the wellbeing of children under different family conditions. For the simple theory that it is parental conØict rather than divorce that affects a child's wellbeing there are four basic conditions' (see Figure 1.3). For eachcondition' the theory would make different predictions about the level of children's wellbeing that we can examine. If the theory that it is parental conØict rather than parental divorce is correct the following propositions should be supported:
∑ Proposition 1: children in situations (a) and (b) would be equally well
off That is, where parental conØict is low, children with divorced parents will do just as well as those whose parents are married.
∑ Proposition 2: children in situations (c) and (d ) should be equally poorly
off That is, children in conØictual couple families will do just as badly as children in post-divorce families where parents sustain high conØict.
∑ Proposition 3: children in situation (c) will do worse than those in situation
(a) That is, those with married parents in high conØict will do worse than those who have married parents who are not in conØict.
∑ Proposition 4: children in situation (d ) will do worse than those in situation
(b) That is, those with divorced parents in high conØict will do worse than those who have divorced parents who are not in conØict.
∑ Proposition 5: children in situation (b) will do better than those in situation
(c) That is, children with divorced parents who are not in conØict will do better than those with married parents who are in conØict.
∑ Proposition 6: children in situation (a) will do better than those in situation
(d ) That is, children with married parents who are not in conØict will do better than those with divorced parents who are in conØict.
Parents divorced?
Parental conflict
No Yes
Low (a) (b)
High (c) (d)
Figure 1.3 The relationship between divorce and parental conflict
THE CONTEXT OF DESIGN 7
No single proposition would provide a compelling test of the original theory. Indeed, taken on its own proposition 3, for example, would reveal nothing about the impact of divorce. However, taken as a pack- age, the set of propositions provides a stronger test of the theory than any single proposition. Although theory testing and theory building are often presented as alternative modes of research they should be part of one ongoing process (Figure 1.4). Typically, theory building will produce a plausible account or explanation of a set of observations. However, such explanations are frequently just one of a number of possible explanations that Æt the data. While plausible they are not necessarily compelling. They require systematic testing where data are collected to speciÆcally evaluate how well the explanation holds when subjected to a range of crucial tests.
What is research design?
How is the term `research design' to be used in this book? An analogy might help. When constructing a building there is no point ordering materials or setting critical dates for completion of project stages until we know what sort of building is being constructed. The Ærst decision is whether we need a high rise ofÆce building, a factory for manufacturing machinery, a school, a residential home or an apartment block. Until this is done we cannot sketch a plan, obtain permits, work out a work schedule or order materials.
Theory
Propositions
Collect data
Analyse data
Implications for propositions
New theory
Inference Deduction
Starting point of theory building
Starting point of theory testing
Develop measures, sample etc.
Figure 1.4 The logic of the research process
8 WHAT IS RESEARCH DESIGN?
Quantitative and qualitative research
Similarly, designs are often equated with qualitative and quantitative research methods. Social surveys and experiments are frequently viewed as prime examples of quantitative research and are evaluated against the strengths and weaknesses of statistical, quantitative research methods and analysis. Case studies, on the other hand, are often seen as prime examples of qualitative research ± which adopts an interpretive approach to data, studies `things' within their context and considers the subjective meanings that people bring to their situation. It is erroneous to equate a particular research design with either quantitative or qualitative methods. Yin (1993), a respected authority on case study design, has stressed the irrelevance of the quantitative/ qualitative distinction for case studies. He points out that:
Design type
Method of data collection
Experiment
Questionnaire Interview (structured or loosely structured)
Observation
Analysis of documents
Unobtrusive methods
Case study
Questionnaire Interview (structured or loosely structured)
Observation
Analysis of documents
Unobtrusive methods
Longitudinal design
Questionnaire Interview (structured or loosely structured)
Observation
Analysis of documents
Unobtrusive methods
Cross-sectional design
Questionnaire Interview (structured or loosely structured)
Observation
Analysis of documents
Unobtrusive methods
Figure 1.5 Relationship between research design and particular data collection methods
10 WHAT IS RESEARCH DESIGN?
a point of confusion... has been the unfortunate linking between the case study method and certain types of data collection ± for example those focusing on qualitative methods, ethnography, or participant observation. People have thought that the case study method required them to embrace these data collection methods... On the contrary, the method does not imply any particular form of data collection ± which can be qualitative or quantitative. (1993: 32)
Similarly, Marsh (1982) argues that quantitative surveys can provide information and explanations that are `adequate at the level of meaning'. While recognizing that survey research has not always been good at tapping the subjective dimension of behaviour, she argues that:
Making sense of social action... is... hard and surveys have not traditionally been very good at it. The earliest survey researchers started a tradition... of bringing the meaning from outside, either by making use of the researcher's stock of plausible explanations... or by bringing it from subsidiary in-depth interviews sprinkling quotes... liberally on the raw correlations derived from the survey. Survey research became much more exciting... when it began including meaningful dimensions in the study design. [This has been done in] two ways, Ærstly [by] asking the actor either for her reasons directly, or to supply information about the central values in her life around which we may assume she is orienting her life. [This] involves collecting a sufÆciently complete picture of the context in which an actor Ænds herself that a team of outsiders may read off the meaningful dimensions. (1982: 123±4)
Adopting a sceptical approach to explanations
The need for research design stems from a sceptical approach to research and a view that scientiÆc knowledge must always be provisional. The purpose of research design is to reduce the ambiguity of much research evidence. We can always Ænd some evidence consistent with almost any theory. However, we should be sceptical of the evidence, and rather than seeking evidence that is consistent with our theory we should seek evidence that provides a compelling test of the theory. There are two related strategies for doing this: eliminating rival explanations of the evidence and deliberately seeking evidence that could disprove the theory.
Plausible rival hypotheses
A fundamental strategy of social research involves evaluating `plausible rival hypotheses'. We need to examine and evaluate alternative ways of explaining a particular phenomenon. This applies regardless of whether the data are quantitative or qualitative; regardless of the particular research design (experimental, cross-sectional, longitudinal or case
THE CONTEXT OF DESIGN 11
But these data are not compelling. There are at least three other ways of accounting for this correlation without accepting the causal link between school type and achievement (Figure 1.6). There is the selectivity explanation: the more able students may be sent to fee paying private schools in the Ærst place. There is the family resources explanation: parents who can afford to send their children to fee paying private schools can also afford other help (e.g. books, private tutoring, quiet study space, computers). It is this help rather than the type of school that produces the better performance of private school students. Finally, there is the family values explanation: parents who value education most are prepared to send their children to fee paying private schools and it is this family emphasis on education, not the schools themselves, that produces the better academic performance. All these explanations are equally con- sistent with the observation that private school students do better than government school students. Without collecting further evidence we cannot choose between these explanations and therefore must remain open minded about which one makes most empirical sense. There might also be methodological explanations for the Ænding that private school students perform better academically. These methodolo- gical issues might undermine any argument that a causal connection exists. Are the results due to questionable ways of measuring achieve- ment? From what range and number of schools were the data obtained? On how many cases are the conclusions based? Could the pattern simply be a function of chance? These are all possible alternative explanations for the Ænding that private school students perform better. Good research design will anticipate competing explanations before collecting data so that relevant information for evaluating the relative merits of these competing explanations is obtained. In this example of schools and academic achievement, thinking about alternative plausible hypotheses beforehand would lead us to Ænd out about the parents' Ænancial resources, the study resources available in the home, the parents' and child's attitudes about education and the child's academic abilities before entering the school.
The fallacy of afÆrming the consequent Although evidence may be con- sistent with an initial proposition it might be equally consistent with a range of alternative propositions. Too often people do not even think of the alternative hypotheses and simply conclude that since the evidence is consistent with their theory then the theory is true. This form of reasoning commits the logical fallacy of afÆrming the consequent. This form of reasoning has the following logical structure:
∑ If A is true then B should follow.
∑ We observe B.
∑ Therefore A is true.
THE CONTEXT OF DESIGN 13
If we apply this logic to the type of school and achievement proposition, the logical structure of the school type and achievement argument becomes clearer.
Initial proposition:
∑ Private schools produce better students than do government schools.
The test:
∑ If A then B^ If private schools produce better students (A) then their
students should get better Ænal marks than those from government funded schools (B).
∑ B is true^ Private school students do achieve better Ænal marks than
government school students (observe B).
∑ Therefore A is true^ Therefore private schools do produce better
students (A is true).
But as I have already argued, the better performance of private school students might also reØect the effect of other factors. The problem here is that any number of explanations may be correct and the evidence does not help rule out many of these. For the social scientist this level of indeterminacy is quite unsatisfactory. In effect we are only in a position to say:
∑ If A [or C, or D, or E, or F, or.. .] then B.
∑ We observe B.
∑ Therefore A [or C, or D, or E, or F, or.. .] is true.
Although explanation (A) is still in the running because it is consistent with the observations, we cannot say that it is the most plausible explanation. We need to test our proposition more thoroughly by evaluating the worth of the alternative propositions.
FalsiÆcation: looking for evidence to disprove the theory
As well as evaluating and eliminating alternative explanations we should rigorously evaluate our own theories. Rather than asking What evidence would constitute support for the theory?', askWhat evidence would convince me that the theory is wrong?' It is not difÆcult to Ænd evidence consistent with a theory. It is much tougher for a theory to survive the test of people trying to disprove it. Unfortunately some theories are closed systems in which any evidence can be interpreted as support for the theory. Such theories are said to be non-falsiÆable. Many religions or belief systems can become closed systems whereby all evidence can be accommodated by the theory and
14 WHAT IS RESEARCH DESIGN?
Summary
This chapter has outlined the purpose of research design in both descrip- tive and explanatory research. In explanatory research the purpose is to develop and evaluate causal theories. The probabilistic nature of causation in social sciences, as opposed to deterministic causation, was discussed. Research design is not related to any particular method of collecting data or any particular type of data. Any research design can, in principle, use any type of data collection method and can use either quantitative or qualitative data. Research design refers to the structure of an enquiry: it is a logical matter rather than a logistical one. It has been argued that the central role of research design is to minimize the chance of drawing incorrect causal inferences from data. Design is a logical task undertaken to ensure that the evidence collected enables us to answer questions or to test theories as unambiguously as possible. When designing research it is essential that we identify the type of evidence required to answer the research question in a convincing way. This means that we must not simply collect evidence that is con- sistent with a particular theory or explanation. Research needs to be structured in such a way that the evidence also bears on alternative rival explanations and enables us to identify which of the competing explana- tions is most compelling empirically. It also means that we must not simply look for evidence that supports our favourite theory: we should also look for evidence that has the potential to disprove our preferred explanations.
16 WHAT IS RESEARCH DESIGN?
