Last week, the Open Science Collaboration reported that only 36% of a sample of 100 claims from published psychology studies were succesfully replicated: Estimating the reproducibility of psychological science.
A reproducibility rate of 36% seems bad. But what would be a good value? Is it realistic to expect all studies to replicate? If not, where should we set the bar?
In this post I'll argue that it should be 100%.
First off however, I'll note that no single replication attemp... Read more »
Open Science Collaboration. (2015) Estimating the reproducibility of psychological science. Science (New York, N.Y.), 349(6251). PMID: 26315443
Just like attempts at influencing hairstyles or clothing can backfire, adults who try to guilt middle-schoolers into exercising won’t get them to be any more active. The study found students who don’t feel in control of their exercise choices or who feel pressured by adults to be more active typically aren’t.... Read more »
DISHMAN, R., MCIVER, K., DOWDA, M., SAUNDERS, R., & PATE, R. (2015) Motivation and Behavioral Regulation of Physical Activity in Middle School Students. Medicine , 47(9), 1913-1921. DOI: 10.1249/MSS.0000000000000616
“Gaydar” — the purported ability to infer whether people are gay or straight based on their appearance — seemed to get a scientific boost from a 2008 study that concluded people could accurately guess someone’s sexual orientation based on photographs of their faces. In a new paper researchers at the University of Wisconsin-Madison challenge what they call “the gaydar myth.” William Cox, an assistant scientist in the Department of Psychology and the lead author, says gaydar isn’t accurate and is actually a harmful form of stereotyping.... Read more »
Cox, W., Devine, P., Bischmann, A., & Hyde, J. (2015) Inferences About Sexual Orientation: The Roles of Stereotypes, Faces, and The Gaydar Myth. The Journal of Sex Research, 1-15. DOI: 10.1080/00224499.2015.1015714
Most Psychology findings are not replicable. What can be done? Stanford psychologist Michael Frank has an idea : Cumulative study sets with internal replication. ‘If I had to advocate for a single change to practice, this would be it.’ I took a look whether this makes any difference. A recent paper in the journal Science […]... Read more »
Bem DJ. (2011) Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of personality and social psychology, 100(3), 407-25. PMID: 21280961
Open Science Collaboration. (2015) PSYCHOLOGY. Estimating the reproducibility of psychological science. Science (New York, N.Y.), 349(6251). PMID: 26315443
Ritchie SJ, Wiseman R, & French CC. (2012) Failing the future: three unsuccessful attempts to replicate Bem's 'retroactive facilitation of recall' effect. PloS one, 7(3). PMID: 22432019
Schimmack U. (2012) The ironic effect of significant results on the credibility of multiple-study articles. Psychological methods, 17(4), 551-66. PMID: 22924598
A commonly prescribed antidepressant may alter brain structures in depressed and non-depressed individuals in very different ways, according to new research. The study – conducted in nonhuman primates with brain structures and functions similar to those of humans – found that the antidepressant sertraline, a selective serotonin reuptake inhibitor (SSRI) marketed as Zoloft, significantly increased the volume of one brain region in depressed subjects but decreased the volume of two brain areas in non-depressed subjects.... Read more »
Willard, S., Uberseder, B., Clark, A., Daunais, J., Johnston, W., Neely, D., Massey, A., Williamson, J., Kraft, R., Bourland, J.... (2015) Long term sertraline effects on neural structures in depressed and nondepressed adult female nonhuman primates. Neuropharmacology, 369-378. DOI: 10.1016/j.neuropharm.2015.06.011
People diagnosed with schizophrenia critically rely upon treatment with antipsychotic medications to manage their symptoms and help them function at home and in the workplace. But despite their benefits, antipsychotic medications might also have some negative effects on brain structure or function when taken for long periods of time.... Read more »
Vita, A., De Peri, L., Deste, G., Barlati, S., & Sacchetti, E. (2015) The Effect of Antipsychotic Treatment on Cortical Gray Matter Changes in Schizophrenia: Does the Class Matter? A Meta-analysis and Meta-regression of Longitudinal Magnetic Resonance Imaging Studies. Biological Psychiatry, 78(6), 403-412. DOI: 10.1016/j.biopsych.2015.02.008
Take 97 psychological effects from top journals which are claimed to be robust. How many will replicate? Brian Nosek and his huge team tried it out and the results were sobering, to say the least. How did we get here? The data give some clues. Sometimes the title of a paper just sounds incredible. Estimating […]... Read more »
John LK, Loewenstein G, & Prelec D. (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science, 23(5), 524-32. PMID: 22508865
Lakens, D. (2014) Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701-710. DOI: 10.1002/ejsp.2023
Open Science Collaboration. (2015) Estimating the reproducibility of psychological science. Science (New York, N.Y.), 349(6251). PMID: 26315443
Simmons, J., Nelson, L., & Simonsohn, U. (2011) False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359-1366. DOI: 10.1177/0956797611417632
The world might seem a little grayer than usual when we’re down in the dumps and we often talk about “feeling blue” — new research suggests that the associations we make between emotion and color go beyond mere metaphor. The results of two studies indicate that feeling sadness may actually change how we perceive color. Specifically, researchers found that participants who were induced to feel sad were less accurate in identifying colors on the blue-yellow axis than those who were led to feel amused or emotionally neutral.... Read more »
What makes someone better at switching between different tasks? Looking for the mechanisms behind cognitive flexibility, researchers have used brain scans to shed new light on this question. By studying networks of activity in the brain’s frontal cortex, a region associated with control over thoughts and actions, the researchers have shown that the degree to which these networks reconfigure themselves while switching from task to task predicts people’s cognitive flexibility.... Read more »
Uddin LQ, Supekar KS, Ryali S, & Menon V. (2011) Dynamic reconfiguration of structural and functional connectivity across core neurocognitive brain networks with development. The Journal of neuroscience : the official journal of the Society for Neuroscience, 31(50), 18578-89. PMID: 22171056
People with hostile personality traits who watch more television than their peers may be at a greater risk for injury, potentially because they are more susceptible to the influence of television on violence and risk-taking behaviors, a University of Pittsburgh Graduate School of Public Health analysis discovered.... Read more »
Fabio, A., Chen, C., Dearwater, S., Jacobs, D., Erickson, D., Matthews, K., Iribarren, C., Sidney, S., & Pereira, M. (2015) Television viewing and hostile personality trait increase the risk of injuries. International Journal of Injury Control and Safety Promotion, 1-10. DOI: 10.1080/17457300.2015.1061560
It seems like everyone is running around buying school supplies and books, registering for classes, and fretting about how hard it is going to be to learn another whole year’s worth of stuff. The secret to success, it turns out, may lie in cow dung.A cow pie. Photo taken by Jeff Vanuga at the USDA available at Wikimedia Commons.Recent research has highlighted the important role that microbes living in animal digestive tracts have on host animals’ health and behavior. This influence of our gut microbes on our behavior is called the microbiota-gut-brain axis. Many of these microbes have long-standing populations that reproduce and spend their whole lives in our guts. Because our digestive tracts do not have much oxygen, these species are anaerobic (do not require oxygen to live). However, our gut communities also have more transient aerobic members (species that do require oxygen to live) that come in when they are ingested and die or leave with the droppings. One of these transient aerobic intestinal citizens is Mycobacterium vaccae (or M. vaccae for short), an aerobic bacterium that naturally lives in soil, water, and yes, cow dung.When mice are injected with heat-killed M. vaccae, they develop an immune response that activates their brain serotonin system and reduces signs of stress. Serotonin is a neurotransmitter that is found in the brain and is involved in regulating alertness, mood, learning and memory. In fact, many antidepressant drugs work by increasing the amount of available serotonin in the brain. Interestingly, serotonin is also found in the digestive system, where it plays a role in digestive health. Since M. vaccae can increase serotonin function, and serotonin reduces anxiety and improves learning, researchers Dorothy Matthews and Susan Jenks at The Sage Colleges in New York set out to test whether eating live M. vaccae could reduce anxiety and improve learning in mice. A drawing of the mouse maze used by Dorothy and Susan. This image is from their 2013 Behavioural Processes paper.The researchers developed a Plexiglas mouse-maze with three difficulty levels, where each increase in difficulty was marked by more turns and a longer path. They encouraged the mice to run the maze by placing a tasty treat (a square of peanut butter on Wonder Bread™) at the end of the maze. Half of the mice were given live M. vaccae on the peanut butter and bread treat three weeks and one week before running the maze, and then again on each treat at the end of each maze run. The other half were given peanut butter and bread without the bacterial additive. The mice then ran the maze roughly every other day: four times at level 1, four times at level 2 and four times at level 3. Each maze run was video recorded and the researchers later watched the videos to count stress-related behaviors.The mice that ingested M. vaccae on their peanut butter sandwiches completed the maze twice as fast as those that ate plain peanut butter sandwiches. They also had fewer stress-related behaviors, particularly at the first difficulty level of the maze when everything was new and scary. In general, the fewer stress behaviors a mouse did, the faster its maze-running time was. The mice that ate the M. vaccae also tended to make fewer mistakes.The researchers then wanted to know how long the effects of M. vaccae lasted. They continued to test the mice in the same maze, again with four runs at level 1, four runs at level 2 and four runs at level 3, but for these maze runs no one was given the M. vaccae. The mice that had previously eaten the M. vaccae continued to complete the maze faster and with fewer mistakes and to show fewer stress-related behaviors for about the first week before the M. vaccae effects wore off.What does this all mean? It means eating dirt isn’t all bad (although I don't recommend eating cow poop). Letting yourself get a bit dirty and ingesting some of nature's microbes could even help you learn better, remember more, and stay calm - especially in new situations. Just something to think about as the school year gets started.Want to know more? Check these out:1. Matthews, D., & Jenks, S. (2013). Ingestion of Mycobacterium vaccae decreases anxiety-related behavior and improves learning in mice Behavioural Processes, 96, 27-35 DOI: 10.1016/j.beproc.2013.02.0072. Lowry, C., Hollis, J., de Vries, A., Pan, B., Brunet, L., Hunt, J., Paton, J., van Kampen, E., Knight, D., Evans, A., Rook, G., & Lightman, S. (2007). Identification of an immune-responsive mesolimbocortical serotonergic system: Potential role in regulation of emotional behavior Neuroscience, 146 (2), 756-772 DOI: 10.1016/j.neuroscience.2007.01.067 ... Read more »
Matthews, D., & Jenks, S. (2013) Ingestion of Mycobacterium vaccae decreases anxiety-related behavior and improves learning in mice. Behavioural Processes, 27-35. DOI: 10.1016/j.beproc.2013.02.007
Lowry, C., Hollis, J., de Vries, A., Pan, B., Brunet, L., Hunt, J., Paton, J., van Kampen, E., Knight, D., Evans, A.... (2007) Identification of an immune-responsive mesolimbocortical serotonergic system: Potential role in regulation of emotional behavior. Neuroscience, 146(2), 756-772. DOI: 10.1016/j.neuroscience.2007.01.067
Parents go to great lengths to ensure the health and well-being of their developing offspring. The favor, however, may not always be returned. Dramatic research has shown that during pregnancy, cells of the fetus often migrate through the placenta, taking up residence in many areas of the mother’s body, where their influence may benefit or undermine maternal health.... Read more »
Boddy, A., Fortunato, A., Wilson Sayres, M., & Aktipis, A. (2015) Fetal microchimerism and maternal health: A review and evolutionary analysis of cooperation and conflict beyond the womb. BioEssays. DOI: 10.1002/bies.201500059
To understand how confidence in parenting may predict parenting behaviors in women who were abused as children, psychologists have found that mothers who experienced more types of maltreatment as children are more critical of their ability to parent successfully. Intervention programs for moms at-risk, therefore, should focus on bolstering mothers’ self-confidence–not just teach parenting skills, the researchers said.... Read more »
Michl, L., Handley, E., Rogosch, F., Cicchetti, D., & Toth, S. (2015) Self-Criticism as a Mechanism Linking Childhood Maltreatment and Maternal Efficacy Beliefs in Low-Income Mothers With and Without Depression. Child Maltreatment. DOI: 10.1177/1077559515602095
Diets rich in fish oil versus diets rich in lard produce very different bacteria in the guts of mice, reports a new study. The researchers transferred these microbes into other mice to see how they affected health. The results suggest that gut bacteria share some of the responsibility for the beneficial effects of fish oil and the harmful effects of lard.... Read more »
Caesar, R., Tremaroli, V., Kovatcheva-Datchary, P., Cani, P., & Bäckhed, F. (2015) Crosstalk between Gut Microbiota and Dietary Lipids Aggravates WAT Inflammation through TLR Signaling. Cell Metabolism. DOI: 10.1016/j.cmet.2015.07.026
Researchers have revealed that HIV does not cause AIDS by the virus’s direct effect on the host’s immune cells, but rather through the cells’ lethal influence on one another. HIV can either be spread through free-floating virus that directly infect the host immune cells or an infected cell can pass the virus to an uninfected cell.... Read more »
Nicole L.K. Galloway et. al. (2015) Cell-to-Cell Transmission of HIV-1 Is Required to Trigger Pyroptotic Death of Lymphoid-Tissue-Derived CD4 T Cells. Cell Reports. DOI: http://dx.org/10.1016/j.celrep.2015.08.011
The recent publication of our paper “Current methods for automated filtering of multiple sequence alignments frequently worsen single-gene phylogenetic inference” in Systematic Biology is the conclusion of 5 years of work, most of which was spent in peer-review. I will write a separate post on the issue of pre- vs. post-publication in a later post; for now, I’ll summarise our main results and try to provide an intuition for them.
Does automatic alignment filtering lead to better trees?
One major use of multiple sequence alignments is for tree inference. Because aligners make mistakes, many practitioners like to mask the uncertain parts of the alignment. This is done by hand or using automated tools such as Gblocks, TrimAl, or Guidance.
The aim of our study was to compare different automated filtering methods and assess under which conditions filtering might be beneficial. We compared ten different approaches on several datasets covering hundreds of genomes across different domains of life—(including nearly all of the Ensembl database) as well as simulated data. We used several criteria to assess the impact of filtering on tree inference (comparing the congruence of resulting trees with undisputed species trees and counting the number of gene duplications implied). We sliced the data in many different ways (sequence length, divergence, “gappyness”).
The more we filter alignments, the worse trees become.
In all datasets, tests, and conditions we tried, we could hardly find any situation in which filtering methods lead to better trees; in many instances, the trees got worse:
Overall, the more alignments get filtered (x-axis in figure), the worse the trees become! This holds across different datasets and filtering methods. Furthermore, under default parameters, most methods filter far too many columns.
The results were rather unexpected, and potentially controversial, so we went to great lengths to ensure that they were not spurious. This included many control analyses and replication of the results on different datasets, and using different criteria of tree quality. We also used simulated data, for which the correct tree is known with certainty.
So is filtering an inherently bad idea?
I should however stress that filtering is not all bad. Moderate amounts of filtering
did not seem to have much impact—positive or negative—but can save some
Also, if we consider the accuracy of the alignment themselves, which we did in
simulations (such that we know the true alignment), filtering does decrease
the proportion of erroneous sites in the aligments (though, of course, these
alignments get shorter!). Thus for
applications more sensitive to alignment errors than tree inference, such as detection of
sites under positive selection, it is conceivable that filtering might, in
some circumstances, help. However, the literature on the topic is rather
ambivalent (see here, here, here, and here).
Why it took us so long: a brief chronology of the project
The project started in summer 2010 as a 3-week rotation project by Ge Tan, who was a talented MSc student at ETH Zurich at the time (he is now a PhD student at Imperial College London, in Boris Lenhard’s group). The project took a few months more than originally foreseen to complete, but early-on the results were already apparent. In his report, Ge concluded:
“In summary, the filtering methods do not help much in most cases.”
After a few follow-up analyses to complete the study, we submitted a first manuscript to MBE in Autumn 2011. This first submission was rejected after peer-review due to insufficient controls (e.g. lack of DNA alignments, no control for sequence length, proportion of gaps, etc.). The editor stated:
“Because the work is premature to reach the conclusion, I cannot help rejecting the paper at this stage”.
Meanwhile, having just moved to EMBL-EBI near Cambridge UK, I gave a seminar on the work. Puzzled by my conclusions, Matthieu Muffato and Javier Herrero from the Ensembl Compara team set out to replicate our results on the Ensembl Compara pipeline. They saw the same systematic worsening of their trees after alignment filtering.
We joined forces and combined our results in a revised manuscript, alongside additional controls requested by the reviewers from our original submission. The additional controls necessitated several additional months of computations but all confirmed our initial observations. We resubmitted the manuscript to MBE in late 2012 alongside a 10-page cover letter detailing the improvements.
Once again, the paper was rejected. Basically, the editor and one referee did not believe in the conclusions and no amount of controls were going to convince them of the contrary. We appealed. The editor-in-chief rebutted the appeal but now the reason was rather different:
“[Members of the Board of Editors] were not convinced that the finding that the automated filtering of multiple alignment does not improve the phylogenetic inference on average for a single-gene data set was sufficiently high impact for MBE.”
We moved on and submitted our work to Systematic Biology. Things worked out better there, but it nevertheless took another two years and three resubmissions—addressing a total of 147 major and minor points (total length of rebuttal letters: 43 pages)—before the work got accepted. Two of the four peer-reviewers went so far as to reanalyse our data as part of their report—one conceding that our results were correct and the other one holding out until the bitter end.
Why no preprint?
Some of the problem with this slow publication process could would have been
mitigated if we had submitted the paper as a preprint. In hindsight, it’s
obvious that we should have done so. Initially, however, I did not anticipated
that it would take so long. And with each resubmission, the paper was
strengthening so I thought for the whole time that it was just about to be accepted… Also, I surely
also fell for the Sunk Cost
Other perils of long-term projects
I’ll finish with a few amusing anectodes highlighting the perils of papers requiring many cycles of resubmissions:
More than once, we had to redo analyses with new filtering methods that got published after we started the project.
At some point, one referee asked why we were using such an outdated version of TCoffee (went from version 5 to version 10 during the project!).
The editor-in-chief of MBE changed, and alongside some of the editorial policy and manuscript format (the paper had to be restructured with the method section at the end).
Tan, G., Muffato, M., Ledergerber, C., Herrero, J., Goldman, N., Gil, M., & Dessimoz, C. (2015). Current Methods for Automated Filtering of Multiple Sequence Alignments Frequently Worsen Single-Gene Phylogenetic Inference Systematic Biology, 64 (5), 778-791 DOI: 10.1093/sysbio/syv033
If you enjoyed this post, you might want to check the other entries of our
series “story behind the paper”.
... Read more »
Tan, G., Muffato, M., Ledergerber, C., Herrero, J., Goldman, N., Gil, M., & Dessimoz, C. (2015) Current Methods for Automated Filtering of Multiple Sequence Alignments Frequently Worsen Single-Gene Phylogenetic Inference. Systematic Biology, 64(5), 778-791. DOI: 10.1093/sysbio/syv033
Could the sperm harpoon the egg to facilitate fertilization? That’s the intriguing possibility raised by the University of Virginia School of Medicine’s discovery that a protein within the head of the sperm forms spiky filaments, suggesting that these tiny filaments may lash together the sperm and its target.... Read more »
Zheng, H., Mandal, A., Shumilin, I., Chordia, M., Panneerdoss, S., Herr, J., & Minor, W. (2015) Sperm Lysozyme-Like Protein 1 (SLLP1), an intra-acrosomal oolemmal-binding sperm protein, reveals filamentous organization in protein crystal form. Andrology, 3(4), 756-771. DOI: 10.1111/andr.12057
Murderers who kill intimate partners and family members have a significantly different psychological and forensic profile from murderers who kill people they don’t know, reports a new Northwestern Medicine study that examined the demographics, psychiatric history and neuropsychology of these individuals.... Read more »
West-Smith, M. (2014) Book Review: Survived by one: The life and mind of a family mass murderer. Criminal Justice Review, 39(2), 222-224. DOI: 10.1177/0734016814523119
Scientists who analyzed the genes involved in 10 autoimmune diseases that begin in childhood have discovered 22 genome-wide signals shared by two or more diseases. These shared gene sites may reveal potential new targets for treating many of these diseases, in some cases with existing drugs already available for non-autoimmune disorders.... Read more »
Yun R Li,, Jin Li,, Sihai D Zhao,, Jonathan P Bradfield,, Frank D Mentch,, S Melkorka Maggadottir,, Cuiping Hou,, Debra J Abrams,, Diana Chang,, Feng Gao,.... (2015) Meta-analysis of shared genetic architecture across ten pediatric autoimmune diseases. Nature Medicine. DOI: http://.org/10.1038/nm.3933
It's a question that many people struggle with and has great implications for the study of our species: are men and women more alike than different or more different than alike, and what differences exist between the sexes?... Read more »
Do you write about peer-reviewed research in your blog? Use ResearchBlogging.org to make it easy for your readers — and others from around the world — to find your serious posts about academic research.
If you don't have a blog, you can still use our site to learn about fascinating developments in cutting-edge research from around the world.
Research Blogging is powered by SMG Technology.
To learn more, visit seedmediagroup.com.