Some residents of Pennsylvania’s Northern Tier and New York’s Southern Tier received an opinion survey from Cornell University this week regarding “fracking.” It was constructed to appear unbiased, but is far from it.
A questionnaire distributed by Cornell University this week illustrates the subtle bias found in so much of academia and in superficial public opinion surveys. I have no doubt the researchers are sincere and believe they have constructed an objective research tool, but that’s the problem. The 28 questions asked, and the introductory explanation, as well as the cover letter and survey technique, all reflect a subtle bias against shale gas development that will almost certainly yield results tilted toward the anti-development view. They will also, without a doubt, and like a preceding study, conclude residents have unanswered concerns that require more study, as is all too typical with such research.
The survey was distributed by the Department of Natural Resources, Human Dimensions Research Unit, at Cornell University. The cover letter included the following message (emphasis added):
Cornell University is trying to better understand how residents in Pennsylvania’s northern tier and New York’s southern tier view and discuss shale gas development via hydraulic fracturing. Often called “fracking”, shale gas development has the potential to affect our communities in many ways. We are interested in how you think it will affect your community and your quality of life. We will use this information to inform local and state officials about community residents’ values and preferences. We are writing to ask that you participate in our study by completing the enclosed questionnaire.
You were randomly selected for this survey due to living in one of four counties in Pennsylvania
(Bradford, Lycoming, Susquehanna, or Wayne) or in one of six counties in New York (Broome, Chemung, Cortland, Delaware, Tioga, or Tompkins). Shale gas development has been discussed heavily in recent years in each of these areas. Even if you do not have strong views about shale gas development, please fill out and return the questionnaire so that our results reflect everyone’s thoughts.
Please complete your questionnaire as soon as possible, seal it with the white re-sealable label provided, and drop it in any mailbox; return postage has been paid. Your participation in the survey is strictly voluntary, but your response is very important to us. Your identity will be kept confidential and the information you give us will never be associated with your name.
The survey introduction also includes these paragraphs:
Hello. We are a team of researchers trying to learn more about communities in Pennsylvania’s northern tier and New York’s southern tier. You were selected for participation in this study because you live in an area where there has been conversation about shale gas development via hydraulic fracturing.
Shale gas development, often called “tracking”, refers to a range of processes used to prepare for, extract, and transport natural gas that is tightly locked within rock formations deep in the earth. A number of environmental, economic, social, and health-related effects may be associated with shale gas development.
We are interested in your thoughts about your community, its future, and potential effects of shale gas development.
While a first reading of these statements might suggest to the casual reader that they are innocuous, they reveal several problems even before the first question is asked.
The first, and perhaps most serious, problem may be found in the citing of several potential negative effects from shale gas development immediately followed by the suggestion the researchers want the respondent’s opinion on the effects. The survey, in other words, only asks for opinions on effects after first suggesting what those effects might be. There are no corollary suggestions of positive effects, so the respondent is already being led toward a negative conclusion.
The leading language is also a strong signal to anti-gas recipients that this survey is for them – their opportunity to send a message, through Cornell, to their local elected officials, thus encouraging higher return rates from this group. This is what is known as “response bias” and random surveys do nothing to alleviate it.
A second major problem may be found in the description of shale gas development as “fracking” and it as “a range of processes used to prepare for, extract, and transport natural gas.” This is simply inaccurate and reinforces negative stereotypes regarding natural gas development. “Fracking” is a slang term of negative connotation, the use of which already suggests negative impacts. The proper term is hydraulic fracturing and it is but one very specific part of natural gas development. It is involved with vertical as well as horizontal wells and can occur months or even years after a well has been drilled.
“Fracking” is not drilling or preparation for drilling, it is not transportation and it is not a “range of processes.” It is a single process designed to stimulate well production after drilling; nothing more and nothing less. Reinforcing the false stereotype that “fracking” includes everything to do with natural gas allows charlatans such as Josh Fox to claim it is responsible for flaming faucets, misleading the survey respondent from the beginning and further contributing to bias.
Still another front-end problem with the Cornell survey is that it is a mail survey especially vulnerable to “non-response bias.” This type of bias is very typical of mail surveys and reflects the fact some individuals may be unwilling or unable to participate in the survey for reasons, which can lead to a group of respondents quite different from the nonrespondents. Individuals working long hours in the natural gas or any other industry, for example, may simply lack the time to respond in a timely manner to a survey, whereas retirees or second-home residents with little interest in economic development may have all the time in the world to do so.
Likewise, white-collar workers (and college professors who work in places like Cornell, pictured to right) are more likely to complete surveys than blue-collar ones because they are accustomed to chasing paper around the room. It’s the sort of thing they’re always doing and, therefore, they are more comfortable with it and likely to do it. This is, from my experience of having done dozens of them over almost 40 years, a common problem with community surveys, which are almost always answered in disproportionately high numbers by seniors, managers and professionals, and in low numbers by members of the local workforce who might be expected to benefit most by natural gas development.
The questions themselves introduce still more inherent bias:
Question 6: Includes six vacuous agree/disagree statements such as “The land in my community is important because it provides for our people” and “The land has its own value, independent of what it provides for us humans,” none of which are clearly understandable or suggest a community might be able to benefit from natural resource development.
Question 11: Asks respondents to evaluate 24 impacts of shale gas development, only eight of which are remotely positive and 16 of which are negative. Giving respondents twice as many possible negative answers suggests the proper answer is negative and leads one to a particular conclusion. It is another example of response bias. Additionally, there are choices such as “lowers property values” but no opportunity to say what is objectively clear from the evidence; that property values increase in areas of natural gas development.
Question 12: Again suggests all negative impacts. Could not the question have asked about improved properties or busier stores, for example, which are among the most obvious positive impacts?
Question 13: Leaves out the most obvious answer – landowner groups. But, “environmental groups” are listed, of course.
Question 15: Asks about bans and moratoriums in a way that suggests they are appropriate and it’s just a question of where. There is no equivalent suggestion on the other side (e.g., “Do you support or oppose state pre-emption of municipal regulation of natural gas development?”)
Question 16: Provides nine choices, eight of which suggest the need for more regulation and one of which is a but a token reference to property rights, presented in such a simplistic fashion as to suggest no reasonable person would think property rights are unrestricted. It is yet another example of response bias; leading the respondent to what he or she thinks is a socially desirable answer.
Question 17: Includes this choice in an otherwise largely unbiased list; “Understanding “tipping points” in how much stress the local environment can handle.” This communicates clear bias by suggesting shale gas development is stressful on the environment, that there are “tipping points” and development threatens them. This is the language of environmentalism as a cause.
About You Questions: These include several useful questions for detecting bias in responses but leave out the really necessary ones regarding employment. Whether a respondent is gainfully employed today will reveal a great deal as to potential bias. Retired individuals who have “made it” or who are making their living elsewhere are going to have clearly different perspectives than those still struggling to put food on the table. Yet, there is nothing here to distinguish among those respondents.
Given these problems with the survey, the answers are predictable. The survey will be biased toward the anti-development side and the risks as opposed to the benefits of shale gas development. There’s nothing surprising about that because, Darrick Nighthawk Evensen, one of the two people signing the survey cover letter (and who probably designed it) is on record posing these questions in the Ithaca Journal:
Philosophy, like science, is essential to an informed decision-making process. Questions of how to weigh outcomes are only the beginning. Other questions abound: What role should distribution of risks and benefits play in decision-making? How do transparency and opportunities for participation in the policy process affect appropriate regulation? How should outcomes be treated differently if citizens are exposed to voluntary versus involuntary risks?
His questions in that instance suggest his own bias. He seems to believe the science doesn’t matter as much as the politics, the risks and rewards aren’t properly distributed, there is no transparency and some citizens are being exposed to invoiuntary risks. He says about the same thing here and here. His comments here about “distributive justice” (as if such a thing was possible) are also revealing of his bias, although, to be fair, they also indicate a naive sincerity. He’s also won a “sermon contest on fracking” that is revealing.
None of this to suggest the Cornell researchers intend bias, merely that they exhibit it. This survey is another shallow all too predictable example of the pseudo-science that passes for research in today’s academies of higher learning. We already know the answer it will produce; more study is needed and its about the politics, not the science. Not helpful…not helpful at all.