I promise I'll make this more interesting, eventually.
dis : adventure header image 1
  • Tags

    Advent America Apple Australia blogging blogosphere Books Canada conference Consumer advocacy current events disability disability/ability earth/health Faith Family film Health in/equality inanity internets Me media Neato! Obama photos Pics Politics poverty running School Science service Soapbox Social Musings TalkingToABs the arts transit TRAVELLLL!! Vancouver vidblogs video work world Writing
  • The Un-Scientific Method: or, how good science becomes bad press

    October 2nd, 2009 · 7 Comments

    I picked this one because it best represented what I look like at work every day!

    I picked this one because it best represented what I look like at work every day!

    Though I try not to be too open about my employer on this blog, I have previously mentioned that I am a researcher. I work on population data sets that encompass thousands of data points covering almost 4,000 people. It’s been a pleasure to learn more about how this data is accumulated, how it’s analyzed and finally, what value it brings to the public, to the funders, and to its stakeholders. One thing, however, has consistently troubled me.

    We researchers are not very good at talking to the public. And to be honest, the public is not very good at listening.

    Maybe it’s that we don’t have enough emphasis on basic science literacy in our increasingly technological and scientific world. (It’s always amused me how though our media and culture steams full-speed toward embracing science at near-religion status, the basic science literacy among the disciples of this new religion is pretty poor. I guess people just want something to believe in.) Maybe it’s that we scientists have not bothered with bringing the research back to the people - though this is changing, it definitely was the case. Research could sit on dusty shelves for years before ever being used for anything - or worse, it was done without thought to what use it might be. Talk about a waste!

    Either way, the fact that research is just not well understood by the public at large has been one of the strongest messages I take home from my job. Recently, our lab published two papers - one of them had been submitted to several journals, being turned down not because it was poor quality but because the editors flat-out didn’t believe it, but could not find a flaw in the method. (Don’t ever think that the scientific establishment doesn’t suffer from the same temptations to censor information that challenges them). The second paper is the first in what will be a series of papers on our research project, giving a “high level” view of what we found in our data. Both were published in Canadian journals, both were peer-reviewed (sent out to experts in the subject area for reading and critique) and accepted, and then…

    …they were shared with the media. (dun, dun DUNNNN)

    Now I confess to a fascination with media. I would consider it a fun and interesting job to be a journalist (”foreign correspondent” is high on my list of “jobs I’d do because they involve travel and mystery.”), and I respect their role as advocates and shit-disturbers. We could not have a free society without a strong press. However - corporate and political interests as they are, I’m incredibly disappointed with the way media deals with science, and I Feel this is partly to blame for the public’s general misconception of research.

    The 20-page article we got published (paper #2) involved various themes that the data fell into, had 10 figures and a data table, and went into great length about health providers and their responses in our study. Unfortunately, the first media story based on our article was given a very unfortunate headline that was based on one single datapoint in our survey, and pointed blame directly at the most sensitive of our study population. That was it - the rest of the media outlets picked up this study off the wires and re-used the same headline, the same angle, the same questions over and over, and we were busy doing damage control the whole week long. IT didn’t matter that our study had hundreds of other points, it was the editor who added the headline to that initial story that cast the die.

    The danger in this is not just that researchers look bad or the public gets the wrong idea. In our field of research, this means that our participants, or people we hope will act on this research, are given the sense that our research was biased, that we “spun” the data to say what we wanted to say. This throws doubt on all of the research we spent more than $800,000 doing for the last four years - and is a huge waste of money. These irresponsible misconstructions waste research dollars, limit access to research populations, and cast doubt on scientific participation at all. there are some research populations who have been so stung by this that we simply can’t research them anymore. They refuse to participate.

    I’ll acknowledge that there is the occasional unethical researcher or biased investigator who can occasionally warrant this sort of doubt. However the constraints that researchers - especially health researchers - operate under keep us in an almost cast-iron straightjacket. Though the journal publication system could be improved, it largely does promote good research. And because only the researchers who understand how to carry out rigorous research get publications in journals (I consider this emphasis on journal publications as our equivalent of “pieces of flair”), and the number of publications and the quality of the journal they are in is considered proof of your chops as a scientist, you can at least be sure that scientists will pour a lot of work into ensuring their publications are accurate, well-written, and relevant.

    I wish we could say the same for media articles.

    My sister recently wrote on her new blog about her frustration over a recent piece on Global National about a recent UK study of 5-year-olds. Her anger was at the barely 60-second piece and how it blamed parents for the findings of the study: that the children of dual income parents were more likely to be overweight or inactive. Buffy was rightfully angry at this judgement - especially since the researchers expressly indicated that their work should not be a judgment on parents. But GlobalNational has a much, much larger circulation than the Journal of Epidemiology and Community Health, so what do you think the lasting message was from this? Not “double-income households need more support to ensure they can provide the best nutrition and activity for their children” or “we need to find out whether children of double-income parents are exposed to disproportionate amounts of unhealthy food messaging which needs policy change.” No. The message was, “double-income parents are purposefully putting their children’s health at risk.”

    Now, I can’t blame this completely on the media. One of the major problems I have with the research publication system we currently have is that research articles - boring as they usually are - are generally not available to the public. Only a handful of open-access journals make their articles available for free. The rest are behind paywalls, requiring often $25-40 per article. Mass media agencies often have the same types of subscriptions that academic institutions do, allowing almost unrestricted access to articles. But they don’t have the seconds or column inches to publish the whole story - and the corporate pressure is to reduce even the most complex messages into stupidly basic, short-sighted and “sexy” headlines. I’m not sure that any Joe Plumber would have an interest or even know where to look for the article even if it was free, but it’s still part of the picture. Until the journal system allows free access to research articles, science is not blameless in this discussion.

    However, the first of our lab’s two articles to be published in September was published by an open-access journal - some of the media about it linked back or cited the article itself as freely available on the journal’s website (one of Canada’s better-known journals too). But to read the comments on the multitude of media stories published online was to see that the public had no clue how research worked. Very few had even read the article - if they had, they would never had asked the questions or made the comments they did. It was a paper about a particular type of healthcare provider and the safety of their type of practice. The dominant comments relayed the utter ignorance of this type of provider among the general public, and would have been quickly addressed if the commenters had clicked the link in the story to read even the abstract (the 250 word summary at the beginning of every journal article). This was eye-opening. There we were, in our lab, gathered around a computer, reading the comments to the news stories about our article, reading comments like “Who are these researchers, anyway? They are probably OWNED by pharma companies and how can we trust them anyway?”

    It was bizarre! WE were the researchers, and our group hasn’t ever seen private money. Canadian research is overwhelmingly taxpayer funded, or funded by endowments like the Vancouver Foundation or the Michael Smith Foundation for Health Research. Ultimately, they are all donated money. Our topic just isn’t marketable enough to attract private dollars. I think ultimately the responsibility lies with the media to report accurately - it’s too much to ask an uninformed public to seek out the journal article when the vast majority of the public don’t know what peer-reviewed articles are or how to find them, and most journals aren’t free. Forget the fact that most articles are written in highly technical language that would put Joe Plumber in a coma of boredom. The function of the media is to get the news out, and to get it right. Currently - neither is happening appropriately.

    My top ideas for fixing this problem:
    - better science literacy in schools - all schools: mandatory basic science curriculum plus incorporation of discussion about research and how the knowledge taught in schools is generated and ends up in their Math, Chemistry, Biology and social science textbooks. It’s a language that is increasingly depended on to drive the worlds advances, people. Y’all need to know it exists, if not speak it.
    - mandatory research process course in J-school: the best and brightest who will fill seats on nightly news shows and write column inches in every local paper in the country 5 years from now should be compelled to learn about science writing in school. If they don’t understand it, how can they tell it?
    - Regulation of science citations in print media: anything online or in the paper printed based on a research study should have one line at the end mandated by law: “You can read the actual study here:” and a link or citation that is understandable to the public.
    - Mandate that all research articles should be made available to the public for free in PDF form on the journal’s website for 3 weeks following the issue’s publication.

    Tags: Uncategorized

    7 responses so far ↓

    • 1 Benjamin Geer // Oct 4, 2009 at 11:03 am

      I agree with everything except “for 3 weeks following the issue’s publication”; they should be freely available permanently! Especially since, as you point out, research is overwhelmingly taxpayer funded. The taxpayer has already paid for the research, and shouldn’t have to pay again for the privilege of reading the results.

    • 2 bill // Oct 4, 2009 at 11:13 am

      “Mass media agencies often have the same types of subscriptions that academic institutions do…”

      This is the first time I’ve heard that claim. Can you point me to some concrete examples?

      “… allowing almost unrestricted access to articles.”

      My own experience does not line up with this claim. Researchers from all over the world, including many large institutions, regularly have to email friends or authors to get hold of articles their subscription access does not cover. I’ve never worked at, say, a Harvard or Cornell, but I’ve been at a couple of decent sized schools and frequently wanted papers that weren’t covered by subscriptions.

    • 3 josiejose // Oct 4, 2009 at 1:57 pm

      Ben - first of all - thanks for stopping by!
      I agree with you in principle, I think there should be free access to research proceeds. But I’m also pragmatic - I’m aware that Journal publication is an industry, and I don’t necessarily think that it’s a good use of money, nor an effective way to distribute information, for us to make journals provide all of their content for free. the Peer review process is incredibly expensive (needlessly so at times, that whole process is another blog post, and a bit of a gong show in my mind).
      the publication industry is going the way of the dodo now that people can get reasonable access to research info on the internet with nothing more than their university library card. but without journal publishers, nobody would be able to pay for collecting, triaging, reviewing and accepting articles, let alone arranging peer review and putting accepted articles to press. Somebody’s got to do it, and the audience is just not big enough for this market to be exclusively ad-funded.
      For now, I would settle for a period of free access when research has just been published so that when research is publicized in the media, the public can see it for themselves. Ultimately, if we can come up with a model that is reasonably profitable and efficient for peer review and allows free access all the time, then that would definitely be preferable!

    • 4 josiejose // Oct 4, 2009 at 2:09 pm

      Bill - welcome as well.

      In this comment I was specifically meaning that mass-media outlets (such as in Canada, CanWestGlobal, or for example CNN) have subscriptions to major journal services so they can research and publish on stories that require that research. With our recent publications related to my work, the reporters who wrote the stories all had access to our articles before we sent them. I know this isn’t the case with all outlets, but my feeling is that the major news services do - or at least their science/health reporters do.

      I’m familiar that university libraries vary in their access to the literature. The large research university I work for has a medical school, and so has pretty much every journal under the sun that is health related. The school I went to for both undergrad and grad school is a more “comprehensive” school, with no med school - so once in awhile I’d come across a health-related journal it didn’t have access to. However, it’s legal and current events periodical collections were unparalleled.

      We also have a system in canada where any article can be ordered from any university which has a subscription - whether it’s from a paper journal or from an online subscription. If it’s not available from another university in Canada, sometimes it can be obtained from CISTI, a federal government science/tech library. Now that communication is so easy, often librarians can fax or email the file to you.

      Because there is a cost involved (often libraries have quotas to ensure that the trade in articles across their subscriptions to journal databases is fair), often this is restricted to staff, faculty and grad students - though as an undergrad I was able to get access to a lot of information just by requesting it through a librarian.

      IT all comes back to whether this is what we are willing to pay for. I feel that the access our public universities in Canada have is very good, and not in immediate need of improvement. What troubles me far more is the lack of access to public research that the general public has.

    • 5 Benjamin Geer // Oct 5, 2009 at 12:36 pm

      Josie, how can peer review be expensive when peer reviewers aren’t paid to do it? See here:

      http://www.phdcomics.com/comics.php?f=1200

      And here:

      http://www.slideshare.net/brembs/whats-wrong-with-scholarly-publishing-today-ii

    • 6 Recent links on Open Access « Free Our Books // Oct 5, 2009 at 12:40 pm

      [...] The Un-Scientific Method: or, how good science becomes bad press: What every journalist should know before reporting on scientific research, and how OA can help. Possibly related posts: (automatically generated)Recent links on Open AccessFriday Blog RoundupROAR & DOAR: Registry / Directory of Open Access RespositoriesAcademic authors, scholarly publishing and open access in Australia [...]

    • 7 josiejose // Oct 5, 2009 at 10:16 pm

      Ben,
      I’m familiar with the PhD comic you linked to - it is pasted above my desk along with Jorge’s brilliant comic about “fill-in-the-blank abstracts.” A staple for any researcher’s office!

      I don’t think peer review itself (the act of sending the journal out to unpaid experts to be reviewed) is expensive - the process and structure of journal publication is expensive. It’s not just a matter of some guy, sending out electronic files to other folks listed on the journal’s board - before it gets to that guy, there’s an editorial assistant who triages, an editor who determines whether it fits the issue and approves the suggested reviewers - perhaps even finding another external reviewer in addition to those suggested. Then the editorial assistant (and her assistants) babysit 40 or so papers and their authors, all at various stages in the process, answering hundreds of questions, dealing with proofs, checking facts, examining citations, etc.
      Then they hound the reviewers - again and again. They deal with antsy authors who want to see their paper published before the grant is due. They send the reviews back to the authors and deal with the graphic designers about the cover and images in this month’s issue. They take revisions from authors and stickhandle the article to final publication.

      Occasionally, they field or solicit media contact on the articles in that month’s issue.

      Even the journal who published our paper #2, which isn’t a huge journal and which uses a large publisher’s submission management website to ease their task of vetting submissions and publishes only once a month, takes a full-time staff of two plus one part-time editor (who is a full-time health professional as well) to manage their content. Multiply that by however many thousands of healthcare journals there are out there and even without hard copy journals there would be costs inherent in the system.

      I’ll give you that if all journals were OA, the costs for printing, hard copies, etc. would all be gone, and perhaps then the page fees for OA articles would be drastically reduced. I think this would be a great improvement, as long as the model still involves formal, independent, anonymous peer review.

      I’ve been a bit scared of what I’ve heard lately out of OA proponents, suggesting that we do away with the current peer-review system entirely and all costs associated with journal publishing, suggesting that journals could be run completely open, with selected articles published on the website, then the comments from the readers of the article serve as a sort of “peer review.” To me, this defeats the whole purpose of a separate track of academic publication… if we were just going to have any old non-scientist be our “peer review,” why don’t we simply forget journals altogether and focus on getting in People Magazine’s website? Then we’d have millions of people to “review” our article.

      So - much as we’d like to think you could run Science like a blog, there’s a lot more to it than that, and mostly it’s expensive because there are human resources involved. We may solve the OA puzzle when we discover how to conduct journals like blogs with low-paid admin staff at the helm.

    Leave a Comment

  • Recent Comments

  • Archives