Is There Really a Reproducibility Problem in the Biomedical Sciences?


By John McLaughlin

The ability to reproduce experimental findings is a keystone of the scientific method; it is a major part of what makes modern science such a successful social activity. In the past few years, however, there has been growing alarm over what is being called a “reproducibility crisis” in science, particularly the biomedical sciences.


One especially high-profile example was discussed in a Nature commentary two years ago: The biotech company Amgen, before investing resources into a new drug program, attempted to reproduce the findings of what it considered 53 “landmark” papers in the cancer biology field, and failed to do so for all but six of the publications. This raises the question, are resources being misguidedly invested into therapeutics that are based on flawed results? And more importantly, is this problem unique to pre-clinical research or is it more pervasive?


The replication problem is definitely receiving attention, in both the popular and scientific press. Several of the world’s most elite scientific journals, including Nature and Science, have recently published editorials calling for answers. Unsurprisingly, the proposed solutions have varied. Some are pushing for more extreme approaches, such as hiring independent, third party laboratories to reproduce the findings of a paper before it reaches publication. Other suggestions have been more modest; journals should require increased transparency regarding the description of experimental methods, and raw data should be submitted to open-access repositories where they can be scrutinized more closely.


The call for more rigorous standards of reproducibility is already evoking concrete responses. Last year, several organizations, including PLOS One, the Science Exchange, and Mendeley, together started the Reproducibility Initiative, which bills itself as an effort to “reward high quality reproducible research”. Here’s the basic idea: scientists confidentially submit their experiments for replication (for a fee), choosing among a network of labs with expertise in a chosen technique. If the findings are confirmed, they can boast an “Independently Validated” badge upon publication of the results. They have already received a $1.3 million grant to reproduce 50 of the “most impactful” cancer biology studies published during 2010-2012.


But if this practice becomes a norm, it may place further financial burdens on labs that are already struggling for funds. Are there any more modest, practical changes we can begin making in our own labs to combat this problem? Part of the solution can be improved graduate training of scientists; regarding the day-to-day use of statistics, which types of analysis are appropriate for your experiment, what sample sizes are needed and what conclusions can reasonably be drawn? Miscommunication between scientists may be a factor as well. Today’s biological science involves complicated experimental techniques, using highly complex animal and cell culture models; more intimate knowledge of the methods may be needed in order to faithfully replicate the results.


On the flip side, are institutional and cultural issues also playing a role? The frantic competition for academic faculty positions and grant funding may skew incentives, encouraging post-docs and PIs to cut corners and push for publication as quickly as possible, in high-tier journals. Nobel Laureate Randy Schekman called attention to this problem last year, and vowed to boycott publishing in “glamour” journals like Nature, Cell, and Science.


Whether or not you agree there is a replication crisis in biomedical science, it surely can’t hurt to encourage more openness, transparency, and improved training. The next generation of young scientists would benefit from making these practices a cultural norm.





The Global STEM Alliance – Revolutionary or More of the Same?


By Celine Cammarata

A few weeks ago, the New York Academy of Sciences and an impressive slew of public and private partners announced the Global STEM Alliance, a new initiative to attract and retain bright young minds in science, math and engineering. On the surface this new project seems much like many other efforts to “fix” STEM education that have already come and gone, and indeed the Alliance is built on principles that can hardly be considered unprecedented: it seeks to provide better resources for science teachers, give kids exposure to real labs and scientists to pique their interest, and leverage the internet to reach widespread populations. But the unique combination of details comprising the program give the impression that it might actually achieve something more revolutionary.


Networks that Work

The Global STEM Alliance is using an innovative combination of on-the-ground and online components. While the Alliance will work with partner agencies to develop classroom materials, teacher education and other activities, the crux of the operation will be an new online platform combining videoconferencing and state-of-the-art educational tools to create, essentially, a virtual playground for science learning. Although the Alliance has not revealed exactly what will be hosted on this platform, it seems the main idea is that the e-space will foster collaborations, and will provide a central location for Alliance programs to reach diverse audiences.


Locally Conscious with a Global Perspective

True to name, the Global STEM Alliance will be an international endeavor. The Alliance is building from the NYAS’s current educational projects, which already are established or developing in six countries, and will use strategic partnerships to continue expansion from there. At the same time, the group plans to structure on-the-ground components so that they can be tailored to local needs.

This framework hopes to increases diversity, capturing the creative power and scientific spark of students in nations that may not typically be considered science hubs. Furthermore, this global reach replaces the nationalist rhetoric often associated with STEM education with a more collaborative approach, stressing science as something the brings humanity together rather than as a source of competition. The goal is not, in this case, to ensure our STEM workforce continues to dominate that of other nations, but instead to ensure that, collectively, we can meet the scientific challenges the world poses.


Mentors Matter

The program will heavily emphasize mentorship, particularly as a means of fighting attrition of students from STEM fields due to disinterest and discouragement; the hope seems to be that getting students under someone’s wing – preferably someone they can relate to – will help them feel more encouraged to pursue STEM, more excited and inspired, and to find STEM fields more accessible. Furthermore, the online platform will enable previously unlikely mentor-mentee relationships, linking researchers, industry professionals, and others from all over the world with an equally distributed student pool.


An All-Star Team

The Global STEM Alliance is not only built on partnership, but arguably has some of the best possible partners available. The NYAS has buddied up with communications giant Cisco to develop the online component of the Alliance, and has (or will be) recruited Nobel laureates and leaders in industry to be among the collaborative network and to mentor students.


Soft Skills Sell

The Alliance also claims that it will be emphasizing soft skills – management, teamwork, communication – which lie at the heart of the “STEM paradox.” However, it remains unclear how the Group plans to accomplish this, particularly given that most of the proposed activities that have been describes, such as working with a team of other future-scientists to solve a research problem, are not too different form the training already being received by university, graduate and postdoctoral students in STEM, which apparently has not prevented this crisis from occurring.



What can we expect to result from the Global STEM Alliance? While the project’s success of course remains to be seen, it seems likely that this effort will make significant headway in improving retention so students in STEM, inspiring them to consider STEM careers, and possible better preparing them by developing the soft skill side. The impact of this, in turn, depends on your perspective. While the Alliance says that it’s efforts will drastically combat the ongoing STEM crisis, numerous commentators have called into question the existence of many such crisis. But that remains for another day…