Fallout from John Bohannon's "Who's afraid of peer review"

As many many scientists, librarians and concerned folk who are interested in scientific publishing and the state of peer review are aware, the whole 'verse' was talking about the "news feature" in Science by John Bohannon entitled "Who's afraid of peer review?".

The basics of the article was a year long "sting" operation on a "select" group of journals (that happened to be open access.. more on this in a second) focusing in part on predatory/vanity journals. That is some of the journals had the "air" of a real science journal, but in fact would publish the paper (?any paper?) for a fee. Basically Bohannon generated a set of faux scientific articles that at a first (and superficial) glance appeared to represent a serious study, but upon even modest examination it would be clear to the reader (i.e. reviewers and editors for the journal) that the experimental methodology was so deeply flawed that the results were essentially meaningless.

Bohannon reported that a large number of the journals he submitted to accepted this article, clearly demonstrating insufficient (or non-existent peer review). This and the head line has apparently lead to a large amount of popular press, and many interviews (I only managed to catch the NPR one I am afraid).

 However, this sting immediately generated a great deal of criticism both for the way it was carried out, and more importantly the way the results were interpreted. First and foremost (to many) that ALL of the journals that were used were open access, and thus no control group for journals with the "traditional" subscription based models (where libraries pay for subscription to the journals). In addition, the journals were sieved to over-represent the shadiest predatory journals. That is it did not represent a random sample of open access journals. One thing that really pissed many people off (in particular among advocated of open access journals, but even beyond this group) that Science (A very traditional subscription based journal) used the summary headline: "A spoof paper concocted by Science reveals little or no scrutiny at many open-access journals.", clearly implying that there was something fundamentally wrong with open access journals. There are a large number of really useful critiques of the article by Bohannon including ones by Michael Eisen, The Martinez-Arias lab, Lenny Teytelman, Peter Suber, Adam Gunn (including a list of other blogs and comments about it at the end). There is another list of responses found here as well.  Several folks also suggested that some open access advocates were getting overly upset, as the sting was meant to focus on just the predatory journals. Read the summary line from the article highlighted in italics above, as well as the article and decide for yourself. I also suggest looking at some of the comment threads as Bohannon does join in on the comments Suber's post, and many of the "big" players are in on the discussion.

A number of folks (including myself) were also very frustrated with how Science (the magazine) presented this (and not just for the summary line). Making the "sting" appear to be scientifically rigorous in its methods, but then turning around and saying this is just a "news" piece whenever any methodological criticism is discussed. For instance, when readers commented about both the lack of peer review and the biased sampling of journals used for the "sting" operation for Bohannon's article, this was a response by John Travis (managing editor of News for Science magazine):

I was most interested in the fact Science (the journal) had an online panel consisting of Bohannon, Eisen and David Roos (as well as Jon Cohen Moderating) to discuss these issues. Much of it (especially in the first half hour) is worth watching, I think it is important to point out that Bohannon suggests he did not realize how his use of only OA journals as part of the sting operation would be viewed. He suggests that he meant this as largely a sting of the predatory journals, and that if he did it again he would have included the subscription based journals as a control group. You can watch it and decide for yourself.

The panelists also brought up two other important points that seem to not get discussed as much in the context of open access vs. subscription models for paying for publication or for peer review.

First, many subscription based journals (including Science) have page charges and/or figure charges that the author of the manuscript pays to the journals. As discussed among the panelists (and I have personal experience with paying for publication of my own research), these tend to be in the same ballpark as for the publication of open access papers. Thus the "charge" that the financial model for publication for OA journals would lead to more papers being accepted is true for many of the subscription journals as well (in particular for journals that are entirely online).

Second (and the useful point to come out of Bohannon's piece) is that there are clear problems with peer review being done sufficiently well. One suggestion that was made by both Eisen and Roos (and has been suggested many times before) is that the reviews provided by the peer referees of the manuscript and the editor could be published alongside (or as supplemental data on figshare) the accepted manuscript, so that all interested readers can assess the extent to which peer review was conducted. Indeed there are a few journals which already do this such as PeerJ, Embo J, ELife, F1000 Research, Biology Direct and some other BMC-series (see here for an interesting example), Molecular Systems Biology, Copernicus Journals. Thanks to folks on twitter for helping me put together this list!

 This latter point (providing the reviews alongside published papers) seems to be so trivial to accomplish, and the reviewers names could easily remain anonymous (or they could provide their names providing a degree of academic credit and credibility to the scientific community) if so desired. So why has this not happened for all scientific journals?  I am quite curious about whether there are any reasons NOT to provide such reviews?

Learning something new: Integrating computer vision into my research.

I recently started my sabbatical year, where I get a chance to re-tool myself, and my research knowledge. The game plan is to learn how to integrate computer vision (machine vision) and machine learning approaches into my research, in particular with respect to the study of animal behaviour and the analysis of images (and videos). We study the evolutionary genetics of complex phenotypes in my lab. While this used to (mostly) mean the complex structure of the shape and size of fruit-fly wings, we are moving more into the study of animal behavior. In particular how flies evade and escape being eaten by predators (more on that at a later date).

 The analysis of such data (both huge sets of wing images as well as video, which is effectively a series of images) can be time consuming and what can be done manually is somewhat limited (such as with JWatcher), in particular if you want to do "high throughput" work with many samples. I have over the past few years interacted, and begun to collaborate with scientists who are using all sorts of techniques from computer vision which have amazed me, both with respect to the speed of the analysis, but also the detailed information gleaned from such approaches. So I am trying to get up to speed and see how to utilize these approaches for my own work.

 To that end I will be now posting about this experience (as well as all of the more usual genetics). This will include useful new tidbits, programming scripts, software I have tried (and tutorials), books, and anything else I can think of. Basically my research progress journal for this new endeavour. I hope that this will help me stay nice and organized, and perhaps will be useful more generally. If you start to follow this thread, and have suggestions for anything, please let me know in the comments or on twitter.

More to follow soon!

Our new pre-print: An integrative genomic approach illuminates the causes and consequences of genetic background effects

This is a guest post by Dr. Chris Chandler. Cross posted from Haldane's Sieve.

Biologists have long recognized that a mutation can have variable effects on an organism's phenotype; even introductory genetics classes often make this observation by introducing the concepts of penetrance and expressivity. More mysterious, however, are the factors that influence the phenotypic expression of a mutation or allele. We know, for instance, that introducing the same mutation into two different but otherwise wild-type genetic backgrounds can result in vastly different phenotypes. But what specific differences between these two genetic backgrounds interact with the mutation, and how? And how does gene expression fit into this puzzle? Answering these questions has not been an easy task, which is not too surprising when you realize that penetrance and expressivity are, in reality, complex quantitative traits. We therefore adopted a multi-pronged genetic and genomic approach to tease apart the mechanisms mediating background dependence in a mutation affecting wing development in the fly Drosophila melanogaster.

The phenotypic patterns seen in our model trait have already been characterized: the scalloped[E3] (sd[E3]) mutation has strong effects in the Oregon-R (ORE) background, resulting in a tiny, underdeveloped wing, while its effects in the Samarkand (SAM) background are still obvious but much less extreme, resulting in a blade-like wing.

To try to find out what causes these differences, we generated and combined a variety of datasets: whole-genome re-sequencing of the parental strains and a panel of introgression lines to map the background modifiers of the sd[E3] phenotype; transcription profiling (using two microarray datasets and one RNA-seq-like dataset), including analyses of allele-specific expression in flies carrying a "hybrid" genetic background; predictions of binding sites for the SD protein, which is a transcription factor; and a screen for deletion alleles that enhance or suppress the sd[E3] phenotype in a background-dependent fashion.

Our results point to a complex genetic basis for this background dependence. We found evidence for a number of loci that are likely to modulate the effects of the sd[E3] allele. However, some unexpected inconsistencies provide a cautionary tale for those intending to take a similar mapping-by-introgression approach for their trait of interest: do multiple replicates, and introgress in both directions, or you may inadvertently end up mapping some other trait! Although the number of candidate genes we identified were generally large, by combining those results with data from our other datasets, we were able to narrow our focus to those showing a consistent signal, yielding a robust set of candidate genes for further study. Without getting into too much detail, we also used a novel approach to show that background-dependent modifier deletions of the sd[E3] phenotype (of which there are many) involve higher-order epistatic interactions between the sd[E3] mutation, the deletion, and the genetic background, rather than quantitative non-complementation (so more than two genes were involved).

Overall, we think that an integrative approach like this could be useful for others trying to understand complex traits, including genetic background-dependence of mutations. In addition, if you're a Drosophila researcher working with the commonly used Samarkand or Oregon-R strains, our genome re-sequencing data (raw and assembled), including SNPs, will soon be available in public repositories for genetic data.

Go to the pre-print of the paper on arXiV.

Evolution and Core Processes in Gene Regulation – some “terminal” thoughts (guest post from David Arnosti)

Guest appearance by David Arnosti here; I’m Ian’s colleague at Michigan State University and co-organizer for the ASBMB Special Symposium.

Our symposium last week brought together some folks who don’t regularly appear at the same conferences, including the structural biologists and biochemists who’ve devoted a good part of the last 30 years to unraveling the mechanisms of the central dogma – gene transcription, RNA metabolism, and protein translation. The trend has been to identify the central machinery, publish high-impact papers in single-word-title journals, and then years later discover that things don’t necessarily work the same in all contexts, organisms, developmental settings. Not surprising in eukaryotic transcription, for instance, where the “basal machinery” comprises ~200 proteins. Zach Burton, conference participant, refers to this complexity as a “molecular horror”, but from an evolutionary perspective, it is also an opportunity.

With respect to tracking the changes inherent in functioning of regulatory circuitry, we have more complete understanding of how bacterial systems work. Nice presentations by Saeed Tavazoie and Eduardo Groisman highlighted how easily such systems can transition between regulatory states, with just a few genetic changes in regulatory factors. Robert Landick and Seth Darst, in studies of E. coli RNA polymerase, pointed out that this bacterium actually features a novel outgrowth, an insertion of 188 amino acids that connects directly to the conserved trigger loop in the catalytic site. Nobody knows what it is doing there, but it is lineage specific, and mutations accumulate in this part of the protein when cells are grown under nutrient limitation, suggesting a short-circuit way to globally fiddle with gene expression.

Eukaryotic systems likewise have novel structures to their core machinery; Lawrence Myers described the gene amplification of Mediator subunits in Candida albicans that are linked to pathogenicity, while Jean-Marc Egly pointed out the pervasive effects of mutations in human Mediator and transcription factor TFIIH. With eukaryotes, the importance of variability in core machinery for generating important changes on the population or species level is obscure. Studies such as those presented by Ian Dworkin (host of this blog) and Aviv Regev showed how we are able to identify numerous loci involved in genetic background effects relating to development and immune function – but the overall importance of pervasive impacts generated by changes in an RNA polymerase subunit, for instance, vs. subtle changes in an enhancer is not clear. One mystery I dropped on the conferees was the special features of the RNA Pol II CTD found in Drosophila, but not other eukaryotes – a reflection of their unique developmental gene expression, discussed by Melissa Harrison and Julia Zeitlinger?

Aside from being mistaken once for Bill Gates by a visitor from Shanghai while walking around the University of Chicago, I was able to maintain my identity as a gene regulation specialist who takes to heart Theodosius Dobzhansky’s mantra that “Nothing in Biology Makes Sense Except in the Light of Evolution”. We will see how the combination of biochemical detail and evolutionary perspective can propel us into a future where gene regulation, in all its rich variation, makes sense. Many thanks to Joan Geiling and Barbara Gordon from the ASBMB for making this an outstanding conference!

Wrap-up for day one of: Evolution and Core Processes in Gene Regulation

Yesterday was the first day of the meeting on the "Evolution and core processes in gene regulation". A small conference (~75 people) interested in various aspects of gene regulation. As I mentioned in a previous post, the participants represent a really diverse mix of biologists interested in gene regulation (how, where why genes are turned on and off), including many who do not speak a common "scientific" language.

To start with, Chicago surprised all of us by having truly pleasant weather for late July. Crisp, dry and not too hot. That always helps perk everyone up.

The diversity of talks was represented right from the very first session with talks ranging from using experimental evolution to examine questions relating to evolution of regulatory function (Saeed Tavazoie), to very mechanistic analysis of enhancers that regulate different tissues in different ways (from Scott Barolo).  I did some fairly extensive tweeting (@IanDworkin, #genereg) on what was presented. However there were a few highlights (for me and my own work).

 The work from the Barolo lab tearing apart the enhancers of the Patched gene (regulated by the transcription factor Gli  mediating hedgehog signaling pathway) was really neat and I am still mulling around the findings in my head.  We know that some proteins bind to DNA, and some of these proteins (called transcription factors) help to regulate gene expression. Sometimes turning genes on, sometimes turning them off. In this case the transcription factor Gli (or as we call it in Drosophila, cubitus interruptus) can take on two forms, one called an activator, the other as a repressor. It turns out that while both of these forms bind to the same general binding sites (DNA words  that sometimes can be spelled a bit differently analogous to "colour" and "color"), how tightly they bind (depending on the spelling) matters. More importantly it matters that some of the sites allow the protein to bind tightly, and others weakly. Having the weak binding sites matters (and it matters that they are weak).

I also really enjoyed the approach that Nir Yakoby was taking to understand the evolution of developmental mechanisms underlying how the shell of the egg for fruit flies was patterned. His lab is taking both a cool approach to thinking about regions of gene expression (gene X is expressed here, here and here, but not here), and how it relates to evolution of the eggshell.

Justin Fay (one of the organizers), also talked about one of his model systems using yeast, and how variation throughout one particular gene (FZF1) was important for yeast to grow in the presence of sulfite (think wine - yeast - sulfites....). The take home message (for me anyways) was that the whole gene had a signature of natural selection (between species), and that the differences in the DNA in many places across the gene seem to contribute to the effects with respect to how the yeast can grow with sulfite.


 There was far more to the first day to the meeting (check the twitter feed for details) and I can not do it all justice, but hopefully this gives you a small flavour for the meeting. If you attending, post about what got you excited.

Genetics really is hard (to interpret)

I am sure this will not surprise most of you, but genetics research can be really hard. I don't simply mean that doing genetics experiments is hard (which it can be), but interpreting the results from genetic analysis can be difficult.  This post is about an interesting story involving the analysis of a a gene called I'm not dead yet (Indy) in the fruitfly Drosophila (one of the geneticists favorite organisms) and its role in extending lifespan. This story, that has taken place over the past decade has taken a number of interesting twist and turns involving many of the subjects that I like to discuss in this blog and my own work, including trying to make sense of the results from genetic studies, the influence of factors like genetic background and environment on mutational effects, and of course Drosophila itself. While I do not study lifespan (longevity), I have been interested, and following the story for this research over the past 5-6 years because of the implications of the influence of genetic background effects (which I do work on).  I should also mention that other than being a geneticist I do not claim to have any great knowledge of the study of aging, but I will do my best on that.

I hope in this (and future) posts to accomplish a few things, so I thought I would lay them all out first (in case I start to ramble off in strange directions).

1. Describe a cool story about something important to just about everyone (who does not want to find out how to live longer).
2. Discuss the means and logic of how genetic analysis. That is how we (geneticists) go about figuring out whether a particular gene (or variant of a gene) influences something we care about (like how long we live).
3. Context matters a lot for genetic analysis. Factors like the food used to feed your critters (among many others factors), and the genetic background (of the critters) that the mutation is studied in can profoundly change what you see (the results).
4. Scientists, even when making honest efforts to perform good, reproducible research can get different results because of seemingly subtle differences in 2&3. 

Not surprisingly, many scientists are interested in the biology of aging, and in particular in what factors influence longevity. In addition to it being very cool, and of obvious importance to many people on the planet, it is also important for aspects of evolutionary theory. The point being that many scientists are interested, and approach questions of aging from many different perspectives, which is great. It is also not surprising that geneticists (and again the general public) are interested in finding genes that influence the aging process (why do some people live longer than others). So in the year 2000 (you know, when all of our computers did not shut down) when a paper entitled "Extended life-span conferred by cotransporter gene mutations in Drosophila" came out, there was a lot of buzz. The basic results suggested that reducing the function or expression of a particular gene, Indy increased how long fruit flies lived. While we (the people) are not fruit flies, by the year 2000 research had already clearly demonstrated that there were many shared genes in all animals (including people and flies), and many seemed to have pretty similar functions. Thus explaining the excitement and buzz. By the way, Indy is short for "I'm not dead yet", and if you do not get the reference check this out (start at 0:58 if the two minutes is too long), or here if you prefer it in musical form, or here as a cartoon.

So what did they do in this study? The punchline is that using multiple, independently generated mutations they demonstrated that as you reduced Indy expression and function, the fruit flies lived for a longer time (increased longevity) when compared to the fruit flies with normal (wild-type) copies of the Indy gene. Seems straightforward enough, and by using multiple independent mutations they demonstrate (at one level) the repeatability of the results. That is, there results are not some strange one-off random results, but can be reproduced, which provides some degree of generality to these results.

Of course, results are rarely so simple and clear, and with additional investigations layers of complexities are often demonstrated.  Studying longevity can be particularly difficult, and not only because you will have to wait a long time to see when something dies of natural causes.

So does Indy actually influence lifespan? The short answer is that the results from follow up studies have been pretty mixed, so it is perhaps not as clear as hoped from the original study. More on that soon in subsequent posts!

References and links if you want more information from the original studies
Rogina B, et al. (2000) Extended life-span conferred by cotransporter gene mutations in Drosophila. Science 290:2137–2140.

Toivonen, et al. 2007. No influence of Indy on lifespan in Drosophila after correction for genetic and cytoplasmic background effects. PLoS Genetics. 3(6):e95

Wang, et al. 2009. Long-lived Indy and calorie restriction interact to extend life span. PNAS USA. 106(23):9262-7. doi: 10.1073/pnas.0904115106

Toivonen JM, Gems D, Partridge L. Longevity of Indy mutant Drosophila not attributable to Indy mutation. Proc Natl Acad Sci USA. doi: 10.1073/pnas.0902462106.

Helfand, SL. et al., 2009. Reply to Partridge et al.: Longevity of Drosophila Indy mutant is influenced by caloric intake and genetic background. 106(21): E54. doi:  10.1073/pnas.0902947106.

Frankel. S. & B. Rogina. 2012. Indy mutants: live long and prosper. Frontiers in Genetics. 3(13). doi: 10.3389/fgene.2012.00013

Rogina B, Helfand SL. 2013. Indy mutations and Drosophila longevity. Front Genet. 4:47. 
  doi: 10.3389/fgene.2013.00047

From the meeting ASBMB:Evolution and Core Processes in Gene Regulation - Chicago (July 25th-28th)

Well this will be something new for me. I will be both blogging and tweeting from the meeting on the evolution and core processes in Gene Regulation from Chicago over the next few days. I will also be giving a talk there as well.  Understanding how and why genes are turned on (or off) both in space (different parts of your body), and time (when you are growing, or as you age) remains an important question that spans a great deal of biology. It turns out that even when asking seemingly similar questions related to gene regulation, the approaches (and language) that say a biochemist or an evolutionary geneticist use can be completely different (and mutually incomprehensible). So meetings like this are really important to getting us all on the same pages and interacting with one another.

This meeting is being organized by David Arnosti, Justin Fay and Ilya Ruvinsky. The three of them (all great scientists and fun to talk with) are bringing together a very diverse group of scientists from hardcore biochemists to straight up evolutionary biologists. We did a related symposium back in 2011, and also in 2008 (at Michigan State University). It was a lot of fun then. Should be more fun (for me anyways) now, as I am not one of the organizers this time. In any case the program for the meeting can be found here.

David, Justin and Ilya also sent this to all of the speakers (to think about for their talk):

We understand that the DNA-RNA-protein dogma is sculpted by evolutionary forces, but where have experimental insights shown HOW particular features of the central machinery are shaped? Some aspects of the process of gene expression have been tightly linked to evolutionary innovations, such as variable HOX expression and limb morphology. 

In your system, which aspects, if any, of the process studied are known to reflect selective pressure? For example, for central components of gene expression machinery, are there particular features that can be linked to specific conditions or attributes of the species/developmental stage/tissue? Alternative tissue-specific or species-specific components, activities?

If you were founding a new research institute dedicated to understand how evolutionary selection acts at all levels of gene expression, how would you combine nuts-and-bolts mechanistic studies with Big Picture evolution research?

What technical breakthroughs would make the biggest impacts on such efforts? Let's think big; e.g. mosquito blood meals preserved in amber (OK, that was a joke).

Tweets start tomorrow afternoon (Thursday July 25th at around 2PM).  I also hope to be doing a daily summary of the meeting on this blog.  Time to get my talk ready!