Cornell Twin Tiers Survey Random But Not Unbiased

Cornell -  Tom Shepstone Tom Shepstone
Shepstone Management Company, Inc.

 

Some residents of Pennsylvania’s Northern Tier and New York’s Southern Tier received an opinion survey from Cornell University this week regarding “fracking.”  It was constructed to appear unbiased, but is far from it.

A questionnaire distributed by Cornell University this week illustrates the subtle bias found in so much of academia and in superficial public opinion surveys.  I have no doubt the researchers are sincere and believe they have constructed an objective research tool, but that’s the problem.  The 28 questions asked, and the introductory explanation, as well as the cover letter and survey technique, all reflect a subtle bias against shale gas development that will almost certainly yield results tilted toward the anti-development view.  They will also, without a doubt, and like a preceding study, conclude residents have unanswered concerns that require more study, as is all too typical with such research.

The survey was distributed by the Department of Natural Resources, Human Dimensions Research Unit, at Cornell University.  The cover letter included the following message (emphasis added):

Cornell University is trying to better understand how residents in Pennsylvania’s northern tier and New York’s southern tier view and discuss shale gas development via hydraulic fracturing. Often called “fracking”, shale gas development has the potential to affect our communities in many ways. We are interested in how you think it will affect your community and your quality of life. We will use this information to inform local and state officials about community residents’ values and preferences. We are writing to ask that you participate in our study by completing the enclosed questionnaire.

You were randomly selected for this survey due to living in one of four counties in Pennsylvania

(Bradford, Lycoming, Susquehanna, or Wayne) or in one of six counties in New York (Broome, Chemung, Cortland, Delaware, Tioga, or Tompkins). Shale gas development has been discussed heavily in recent years in each of these areas. Even if you do not have strong views about shale gas development, please fill out and return the questionnaire so that our results reflect everyone’s thoughts.

Please complete your questionnaire as soon as possible, seal it with the white re-sealable label provided, and drop it in any mailbox; return postage has been paid. Your participation in the survey is strictly voluntary, but your response is very important to us. Your identity will be kept confidential and the information you give us will never be associated with your name.

The survey introduction also includes these paragraphs:

Hello. We are a team of researchers trying to learn more about communities in Pennsylvania’s northern tier and New York’s southern tier. You were selected for participation in this study because you live in an area where there has been conversation about shale gas development via hydraulic fracturing.

Shale gas development, often called “tracking”, refers to a range of processes used to prepare for, extract, and transport natural gas that is tightly locked within rock formations deep in the earth. A number of environmental, economic, social, and health-related effects may be associated with shale gas development.

We are interested in your thoughts about your community, its future, and potential effects of shale gas development.

While a first reading of these statements might suggest to the casual reader that they are innocuous, they reveal several problems even before the first question is asked.

The first, and perhaps most serious, problem may be found in the citing of several potential negative effects from shale gas development immediately followed by the suggestion the researchers want the respondent’s opinion on the effects.  The survey, in other words, only asks for opinions on effects after first suggesting what those effects might be.  There are no corollary suggestions of positive effects, so the respondent is already being led toward a negative conclusion.

Screen Shot 2013-09-06 at 6.48.30 AMThe leading language is also a strong signal to anti-gas recipients that this survey is for them – their opportunity to send a message, through Cornell, to their local elected officials, thus encouraging higher return rates from this group.  This is what is known as “response bias” and random surveys do nothing to alleviate it.

A second major problem may be found in the description of shale gas development as “fracking” and it as “a range of processes used to prepare for, extract, and transport natural gas.”  This is simply inaccurate and reinforces negative stereotypes regarding natural gas development.  “Fracking” is a slang term of negative connotation, the use of which already suggests negative impacts.  The proper term is hydraulic fracturing and it is but one very specific part of natural gas development.  It is involved with vertical as well as horizontal wells and can occur months or even years after a well has been drilled.

“Fracking” is not drilling or preparation for drilling, it is not transportation and it is not a “range of processes.”  It is a single process designed to stimulate well production after drilling; nothing more and nothing less.  Reinforcing the false stereotype that “fracking” includes everything to do with natural gas allows charlatans such as Josh Fox to claim it is responsible for flaming faucets, misleading the survey respondent from the beginning and further contributing to bias.

Still another front-end problem with the Cornell survey is that it is a mail survey especially vulnerable to “non-response bias.”  This type of bias is very typical of mail surveys and reflects the fact some individuals may be unwilling or unable to participate in the survey for reasons, which can lead to a group of respondents quite different from the nonrespondents.  Individuals working long hours in the natural gas or any other industry, for example, may simply lack the time to respond in a timely manner to a survey, whereas retirees or second-home residents with little interest in economic development may have all the time in the world to do so.

Cornell

Likewise, white-collar workers (and college professors who work in places like Cornell, pictured to right) are more likely to complete surveys than blue-collar ones because they are accustomed to chasing paper around the room.  It’s the sort of thing they’re always doing and, therefore, they are more comfortable with it and likely to do it.  This is, from my experience of having done dozens of them over almost 40 years, a common problem with community surveys, which are almost always answered in disproportionately high numbers by seniors, managers and professionals, and in low numbers by members of the local workforce who might be expected to benefit most by natural gas development.

The questions themselves introduce still more inherent bias:

Question 6: Includes six vacuous agree/disagree statements such as “The land in my community is important because it provides for our people” and “The land has its own value, independent of what it provides for us humans,” none of which are clearly understandable or suggest a community might be able to benefit from natural resource development.

Question 11: Asks respondents to evaluate 24 impacts of shale gas development, only eight of which are remotely positive and 16 of which are negative.  Giving respondents twice as many possible negative answers suggests the proper answer is negative and leads one to a particular conclusion.  It is another example of response bias.  Additionally, there are choices such as “lowers property values” but no opportunity to say what is objectively clear from the evidence; that property values increase in areas of natural gas development.

Question 12: Again suggests all negative impacts.  Could not the question have asked about improved properties or busier stores, for example, which are among the most obvious positive impacts?

Question 13:  Leaves out the most obvious answer – landowner groups.  But, “environmental groups” are listed, of course.

Question 15:  Asks about bans and moratoriums in a way that suggests they are appropriate and it’s just a question of where.  There is no equivalent suggestion on the other side (e.g., “Do you support or oppose state pre-emption of municipal regulation of natural gas development?”)

Question 16: Provides nine choices, eight of which suggest the need for more regulation and one of which is a but a token reference to property rights, presented in such a simplistic fashion as to suggest no reasonable person would think property rights are unrestricted.  It is yet another example of response bias; leading the respondent to what he or she thinks is a socially desirable answer.

Question 17: Includes this choice in an otherwise largely unbiased list; “Understanding “tipping points” in how much stress the local environment can handle.”  This communicates clear bias by suggesting shale gas development is stressful on the environment, that there are “tipping points” and development threatens them.  This is the language of environmentalism as a cause.

About You Questions:  These include several useful questions for detecting bias in responses but leave out the really necessary ones regarding employment. Whether a respondent is gainfully employed today will reveal a great deal as to potential bias.  Retired individuals who have “made it” or who are making their living elsewhere are going to have clearly different perspectives than those still struggling to put food on the table.  Yet, there is nothing here to distinguish among those respondents.

Given these problems with the survey, the answers are predictable.  The survey will be biased toward the anti-development side and the risks as opposed to the benefits of shale gas development.  There’s nothing surprising about that because, Darrick Nighthawk Evensen, one of the two people signing the survey cover letter (and who probably designed it) is on record posing these questions in the Ithaca Journal:

Philosophy, like science, is essential to an informed decision-making process. Questions of how to weigh outcomes are only the beginning. Other questions abound: What role should distribution of risks and benefits play in decision-making? How do transparency and opportunities for participation in the policy process affect appropriate regulation? How should outcomes be treated differently if citizens are exposed to voluntary versus involuntary risks?

His questions in that instance suggest his own bias.  He seems to believe the science doesn’t matter as much as the politics, the risks and rewards aren’t properly distributed, there is no transparency and some citizens are being exposed to invoiuntary risks.  He says about the same thing here and here.  His comments here about “distributive justice” (as if such a thing was possible) are also revealing of his bias, although, to be fair, they also indicate a naive sincerity.  He’s also won a “sermon contest on fracking” that is revealing.

None of this to suggest the Cornell researchers intend bias, merely that they exhibit it. This survey is another shallow all too predictable example of the pseudo-science that passes for research in today’s academies of higher learning.  We already know the answer it will produce; more study is needed and its about the politics, not the science.  Not helpful…not helpful at all.

follow-us-on-twitter  like-us-on-facebook  follow-us-on-linkedin

Print Friendly, PDF & Email

10 thoughts on “Cornell Twin Tiers Survey Random But Not Unbiased

  1. Is the survey truly random? From what data base did the Cornell team select the recipients? Are they homeowners who own houses over a certain value? Renters, Lanlords? Were any specifications put in place to prevent an accidental majority of either urban or rural recipients? Were areas excluded or included based on whether they had active gas drilling in their town? Were areas included or excluded depending on whether there is or was an existing ban or moratorium? A random series of coin flips can still yield one hundred consecutive “heads” even thought the odds are 50/50. How the data base was selected and how the specifications guarded against ‘random’ bias is crucial to the credibility of the survey. What degree of effort was expended to assure a true cross sample? Were the survey format and questions reviewed by a polling or survey company for compliance with ethical and intellectual integrity standards? This survey should be examined with the same degree of scrutiny as the opponents of fracking examine oil and gas drilling.
    Dr. Johnson provided wise words: Nothing will ever be attempted if all possible objections must first be overcome. Or Voltaire: Doubt is uncomfortable, but certainty is ridiculous.

    • Hello Paul,

      Thank you for your questions. Yes, the survey sample was as random as one could generate if relying on a mail survey (other types of surveys, such as telephone or Internet surveys have their own issues). We used a database, updated within 90 days of when the survey was sent out, of all USPS mailing address (including rural routes, highway contracts, throwbacks, all forms of PO Boxes, and many other postal service designations).

      We carefully sampled from both urban and rural areas, areas with high levels of drilling, areas will little drilling, and areas without drilling, areas with bans/moratoria and areas with legislation supporting development, as well as range of other community level variables. The surveys go to anyone who receives mail; therefore, homeowners, renters, and landlords could potential receive the survey. We spent over two months making careful decisions on each of the questions mentioned here and many more.

      All surveys that are distributed from a university, unlike many that generate in the corporate or non-profit world, require pre-approval by the university’s Institutional Review Board (IRB), which ensures ethical compliance. Our survey went through the same process as every other survey would to receive IRB approval.

      Thank you for this opportunity to provide additional information about our survey methodology. I hope readers find that our survey was constructed and administered by relying on the best methods available for survey research.

      Best regards,

      Darrick Evensen

  2. Hello all,

    As one of the authors of this survey, I am sorry that some people perceive this questionnaire as so biased. Tom is correct in stating that the researchers who created the survey were sincere and that we believed (and still believe) that we created an objective tool. We designed this survey by interviewing 50+ individuals living in northern PA and southern NY, who are heavily engaged in the issue of shale gas development. Because we are interested in how this issue is discussed in communities, we used the data from those interviews to generate questions for the survey. Tom Shepstone was one of the people I interviewed in April, and who, through his helpful comments in our interview, contributed to some of the questions in the survey.

    If people suspect bias in the survey, the best recommendation I can offer is to fill out and return the survey so that we can be certain we have captured your perspectives.

    One of our primary goals in conducting this research is to understand how discourse develops on this issue in communities most affected by it. I strongly believe that our survey will provide us with such understanding. Our goal is not to push for further study of shale gas development, as Tom suggests above. Our goal is to better understand factors that influence people to think about this issue as they do. There are a range of social, cultural, historical, and other factors that could shape views on shale gas development. We are interested in how those factors play out on this issue. I should think that this research would be relevant to anyone interested in the shale gas development debate, regardless of where they fall on the issue.

    Below I address a few of Mr. Shepstone’s specific concerns:

    He mentions that we initially introduce the survey by listing negative effects. If you look at our wording, you will note that we only mention “effects”, not negative effects. The wording was very deliberate. We mention economic, environmental, social, and health-related effects. Tom mentioned to me, during our interview, the positive effects shale gas development can have on the environment due to it being clean energy. He also mentioned myriad positive economic effects and many positive social effects, such as providing local jobs for youth, thus, keeping the youth in the area, and contributing to energy independence. Some interviewees mentioned that shale gas development could lead to better health services infrastructure in communities, improving health overall. Therefore, there are clearly potential positive effects in all of these categories; it is unclear to me why Mr. Shepstone states that we prime people to think of negative effects.

    Tom also mentions response bias. The survey has many means of characterizing our response population and comparing it to the general population, to check for over- or under-representation on a range of variables. Our survey follows a four-wave mailing that is consistent with the best recommendations that exist in methodological literature for how to achieve robust response rates. Additionally, we will be conducting 150 telephone non-respondent follow up interviews with people who received but did not fill out the survey. We will compare those follow ups to the people who sent in the survey to check for non-response bias on the range of variables that we examine.

    Finally, Tom critiques our use of the word “fracking”. Many of us know that this word creates much confusion and that it does carry an inherent bias. We use the word in the survey merely because some people who are not as familiar with the issue as Tom and I are may know shale gas development primarily by this word. “Fracking” has been used (frequently in mass media and other public fora) to describe a range of processes, even though “hydraulic fracturing” only refers to one specific process. My colleagues and I have written a six-page article which is currently under review for publication in which we highly critique this word and state why it should generally be avoided. I could not, however, include such a discussion in the cover letter to the survey. We used “fracking” merely to draw attention to the social construct that many people associate with the various processes. The reader of the survey will note that we use “shale gas development” throughout the entire survey. Not once in an actual question do we use the word “fracking”; we only use it in the introduction. It is worth mentioning that individuals opposed to shale gas development have critiqued use of “shale gas development”, suggesting that this phrase shows positive bias toward the processes. We do not mean to include bias, but alternatives for what to call the range of processes are lacking.

    I hope I have been able to shed some light on why we chose to construct the survey as we did. I would be happy to speak with anyone who has concerns about the survey.

    Best regards,

    Darrick Evensen

  3. The problem of crusaders attempting to disguise themselves as objective researchers seems quite common in debates over evironmental issues, especially in academic settings. Mr. Evensen’s acrobatic and insistent defense of this “survey”…. particularly his defense of the uses of the headline word “fracking”, which he justifies on the grounds that it is, in fact, a widely used, if loaded and bias inducing term…. speaks volumes about his agenda. Mr. Shepstone has done a very good job pointing out numerous frailties in the survey questionnaire. They remain unrefuted despite Mr. Evensen’s two attempts to do so.

  4. “Fracking” is widely used — by activists who oppose the industry. Its connotations are inherently negative. Evensen’s insistence that it is the best term to describe shale gas or natural gas development definitively proves his bias. If he had spent any amount of time in the four counties listed he would know that locals only ever hear the term used as a pejorative by the New York activists who come to Pennsylvania use it in their chants, on their signs and whenever they confront locals, and by uneducated media looking for the next provocative headline.

    An even bigger problem is the survey’s association with Cornell. Between the widely discredited work of Robert Howarth and Tony Ingraffea — often cited by the activist media juggernaut — as well as their cozy relationships with activist organizations, it is hard to think of anything unbiased coming from the campus. This survey language and Evensen’s spinning defense of it shows just how ingrained that bias has become.

  5. Again, allow me to state that, despite what the commentators above imply, the word “fracking” is never used in a question in our survey (for confirmation, please view the scanned copy of the full survey; the first link that Tom provides in his post). The title of the survey also only uses the phrase “shale gas development” (the phrase used throughout the survey). The commentators do not deny that “fracking”, however confusing and biased, is used frequently in the mass media and may be the primary term by which some citizens are familiar with the range of industrial processes associated with shale gas development via hydraulic fracturing. I contend that such usage justifies inclusion of “fracking” once in the entire survey compared to the twenty uses of “shale gas development”. I have been at meetings where gas industry executives have used this term as well.

    That anyone could critique research from a given university, writ large, based on research from a couple professors at that institution displays a lack of familiarity with the structure of such institutions. There are 79 distinct academic departments at Cornell distributed across 11 colleges on the central campus. The departments of civil and environmental engineering and ecology and evolutionary biology, with which the mentioned professors are associated, are in entirely different colleges than the department of natural resources. Scott does not mention the professors in the department of earth and atmospheric sciences (also a different college) who have strongly supported shale gas development publicly in their academic writing.

    The comparison of the researchers who constructed this survey to academics strongly opposed to shale gas development is inappropriate on another level – Scott himself states that such faculty identify with activist organizations. None of the researchers who worked on this survey are members of or affiliated with any partisan organizations focused on this issue (pro- or anti-shale gas development). All the researchers on this project do, however, spend substantial time and effort providing outreach to community members who wish to know more about shale gas development; we have frequently worked with residents in favor of gas development as well as those opposed to it.

    Finally, because these posts are in written form and lack the tone of a human voice to accompany them, allow me to state that I mean in no way for my words to sound antagonistic. My only concern is to address misunderstandings. As I mentioned in my first post, Tom himself provided very useful comments in his interview with me that helped to shape some of the questions in the survey. I also respect Tom as an honest man who cares for his community. The very purpose of my survey is to understand how and why people think shale gas development will affect their communities in certain ways.

    Again, I would be happy to discuss further any concerns individuals may have with this survey. My e-mail address is listed on the cover letter to the survey that Tom copied in his PDF scan.

    Best regards,

    Darrick Evensen

  6. Allow me to relay an interesting anecdote from my research on this issue in the province of New Brunswick, Canada, where I am also investigating perceptions of shale gas development:

    Of the 23 individuals I interviewed, associated with industry groups, activist groups, government, non-profit groups, and unaffiliated civically-engaged residents, nearly every one mentioned that mass media coverage on the issue of shale gas development is biased, and not only biased, but, on the whole, coverage is biased against his/her point of view. This clearly cannot be the case. How could coverage, on the whole, be inappropriately anti- and pro-shale gas development at the same time?

    I am studying this phenomenon further, but one hypothesis for what may be at work here is a theory called the “hostile media effect”. Much research on this theory has shown empirically that on contentious issues, people on both (or all) sides of the debate often feel that the exact same news articles and reports are biased against their point of view. I am not suggesting this has any relevance to mass media coverage in PA or NY. I think it may, however, have some bearing on perceptions of bias in the survey.

    Best regards,

    Darrick Evensen

    • I don’t wish to be argumentative, as you have responded very professionally in everything you’ve offered here, Darrick, but I often hear that answer – that if both sides feel there’s bias, there probably isn’t any but I’m not convinced. Mainstream media outlets are fond of saying this, for example, but the fact they receive complaints from both sides isn’t proof of anything. The real proof is in the numbers themselves and using mainstream media as an example, we know 90% or more are typically affiliated with one political view and the counts of news stories and mentions demonstrate bias beyond any doubt, yet these media say there is none because they get complaints from both sides. Unfortunately for them, they’ve only fooled themselves and that’s why every major mainstream media outlet is in deep trouble today – they’ve failed to see their own bias and rationalized it away. I’m not saying that’s what you’re doing but be careful in relying upon the uniformity of complaints from both sides because it really tells you nothing.

      • Tom, I agree with you — it does not mean that both “sides” are wrong; yet, it does mean that at least one of them is. Objectively, media coverage or a research instrument cannot be biased against two opposing sides at the same time. One of my closest collaborators, with whom I have worked for years on Marcellus Shale research, has recently been labeled a pawn for pro-gas-development causes. Because he and I use many of the same survey questions (indeed, we have developed some of them together), it is difficult to believe that his research is biased in favor of shale gas development while mine is biased against it.

        However, it is quite possible that media coverage could be inappropriately biased against each of two opposing sides in a debate, based on those “sides” own definitions of inappropriate bias.

        For example, a pro-shale-gas-development person might assert that 80% of the evidence supports moving ahead with shale gas development, while 20% of the evidence may cause us to rethink it. Vice versa, an anti-shale-gas-development person might assert that only 20% of the evidence supports moving ahead with shale gas development, while 80% may cause us to rethink it. In this case, an article that presents a 50%/50% balance of evidence for each side would be seen as inappropriately biased by both sides.

        Issues often exist on which both sides of the debate should not be treated equally because the evidence is clearly in favor of one perspective over another. When media coverage does treat both sides equally, this is known as “false balance”. In this sense, “balanced” reporting is actually biased. An example could be the controversy over whether certain vaccines cause autism. The scientific evidence for this claim is literally non-existent (or has been definitively refuted or shown to have been fabricated), yet some reporting treats both sides of the argument equally (the one based in science and the one in anecdotes).

        In my survey, I tried to avoid bias and false balance by simply drawing my items from my interviews with people highly engaged in discourse on this issue. This approach is by no means perfect, but I found it to be a more defensible approach than the alternatives available to me.

  7. I actually admire your persistence in defending the study and standing up to our criticisms. I understand academics would love for everyone to understand university structure, so that the activities of one department do not reflect upon those of another. The reality however is that a Google search for “Cornell methane study” returns nearly 360,000 results, while a search for “Howarth methane study” returns just 59,000 results. You are, in essence, asking the public to understand that “Cornell” is an overly broad term to apply to the research coming out of that university. Does it remind you of anything? (Hint: it’s a word that rhymes with cracking.)

    Lastly, it is not just me who associates certain professors with activist organizations. See this Eco-Watch story (wherein, incidentally, the author also notes that the methane “research” is more commonly known as “The Cornell Study”: perhaps the faculty senate or marketing department should have a say in that usage going forward): http://ecowatch.com/2013/industry-insider-to-fracking-opponent/

Leave a Reply

Your email address will not be published. Required fields are marked *