Measuring the Value of Science: Keeping Bias out of NIH Grant Review

 

By Rebecca Delker, PhD

Measuring the value of science has always been – and, likely, will always remain – a challenge. However, this task, with regard to federal funding via grants, has become increasingly more daunting as the number of biomedical researchers has grown substantially and the available funds contracted. As a result of this anti-correlation, funding rates for NIH grants, most notably, the R01, have dropped precipitously. The most troubling consequences of the current funding environment are (1) the concentration of government funds in the hands of older, established investigators at the cost of young researchers, (2) a shift in the focus of lab-heads toward securing sufficient funds to conduct research, rather than the research itself and (3) an expectation for substantial output, increasing the demands for preliminary experiments and discouraging the proposal of high-risk, high-reward projects. The federal grant system has a direct impact on how science is conducted and, in its current form, restricts intellectual freedom and creativity, promoting instead guaranteed, but incremental, scientific progress.

 

History has taught us that hindsight is the only reliable means of judging the importance of science. It was sixteen years after the death of Gregor Mendel – and thirty-five years after his seminal publication – before researchers acknowledged his work on genetic inheritance. The rapid advance of HIV research in the 1980s was made possible by years of retroviral research that occurred decades prior. Thus, to know the value of research prior, or even a handful of years after publication, is extremely difficult, if not impossible. Nonetheless, science is an innately forward-thinking endeavor and, as a nation, we must do our best to fairly distribute available government funds to the most promising research endeavors, while ensuring that creativity is not stifled. At the heart of this task lies a much more fundamental question – what is the best way to predict the value of scientific research?

 

In a paper published last month in Cell, Ronald Germain joins the conversation of grant reform and tackles this question by proposing a new NIH funding system that shifts the focus from project-oriented to investigator-oriented grants. He builds his new system on the notion that the track record of a scientist is the best predictor of future success and research value. By switching to a granting mechanism similar to privately funded groups like the HHMI, he asserts, the government can distribute funds more evenly, as well as free up time and space for creativity in research. Under the new plan, funding for new investigators would be directly tied to securing a faculty position by providing universities “block grants,” which are distributed to new hires. In parallel, individual grants for established investigators would be merged into one (or a few) grant(s), covering a wider range of research avenues. For both new and established investigators, the funding cycle would be increased to 5-7 years and – the most significant departure from the current system – grant renewal dependent primarily on a retrospective analysis of work completed during the prior years. The foundation for the proposed granting system relies on the assumption that past performance, with regard to output, predicts future performance. As Germain remarks, most established lab-heads trust a CV over a grant proposal when making funding decisions; but it is exactly this component of the proposal – of our current academic culture – that warrants a more in-depth discussion.

 

Germain is not the first to call into question the reliability of current NIH peer reviews. As he points out, funding decisions for project-oriented grants are greatly influenced by the inclusion of considerable preliminary data, as well as form and structure over content. Others go further and argue that the peer review process is only capable of weeding out bad proposals, but fails at accurately ranking the good. This conclusion is supported by studies, which establish a correlation between prior publication, not peer review score, and research outcome. (It should be noted that a recent study following the outcomes of greater than 100,000 funded R01 grants found that peer review scores are predictive of grant outcome, even when controlling for the effects of institute and investigator. The contradictory results of these two studies cannot yet be explained, though anecdotal evidence falls heavily in support of the former conclusions.)

 

Publication decisions are not without biases. Journals are businesses and, as such, benefit from publishing headline-grabbing science, creating an unintended bias against less trendy, but high quality, work. The more prestigious the journal, the higher its impact factor, the more this pressure seems to come into play. Further, just as there is a necessary skill set associated with successful grant writing that goes beyond the scientific ideas, publication success depends on more factors than the research itself. An element of “story-telling” can make research much more appealing; and human perception of the work during peer review can easily be influenced by name recognition of the investigator and/or institute. I think it is time to ask ourselves if past publication record is truly predictive of future potential, or, if it simply eases the way to additional papers.

 

In our modern academic culture, the quality of research and of scientists is often judged by quantitative measures that, at times, can mask true potential. Productivity, as measured by the number of papers published in a given period of time, is a standard gaining momentum in recent years to serve as a meaningful evaluation of the quality of a scientist. As Germain states, a “highly competent investigator” is unlikely “to fail to produce enough … to warrant a ‘passing grade’.” The interchangeability of competence and output has been taken to such extremes that pioneering physicist and Nobel Prize winner, Peter Higgs, has publicly stated that he would be overlooked in current academia because of the requirement to “keep churning out papers.” The demand for rapid productivity and high impact factor has caused an increase in the publication of poorly validated findings, as well as in retraction rates due to scientific misconduct. The metrics used currently to value science are just as, if not more, dangerous to the progress of science as the restrictions placed on research by current funding mechanisms.

 

I certainly do not have a fail-proof plan to fix the current funding problems; I don’t think anyone does. But, I do think that we need to look at grant reform in the context of the larger issues plaguing biomedical sciences. As a group of people who have chosen a line of work founded in doing/discovering/inventing the impossible, we have taken the easy way out when approached with measuring the value of research. Without the aid of hindsight, this task will never be objective and assigning quantitative measures like impact factor, productivity, and the h-index has proven only to generate greater bias in the system. We must embrace the subjectivity present in our review of scientific ideas while remaining careful not to vandalize scientific progress with bias. Measures to bring greater anonymity to the grant review process and greater emphasis on qualitative and descriptive assessments of past work and future ideas will help lessen the influence of human bias and make funding more fair. As our culture stands, a retrospective review process, as Germain proposes, with a focus on output runs the risk of adopting into the grant review process our flawed, and highly politicized, methods of judging the quality of science. I caution that in parallel to grant reform, we begin to initiate change in the metrics we use to measure the value of science.

 

Though NIH funding-related problems and the other systemic flaws of our culture seem at an all time high right now, the number of publications addressing these issues has also increased, especially in recent years. Now, more than ever, scientists at all stages recognize the immediacy of the problems and are engaging in conversations both in-person and online to brainstorm potential solutions. A new website  serves as a forum for all interested to join the discussion and contribute reform ideas – grant, or otherwise. With enough ideas and pilot experiments from the NIH we can ensure that the best science is funded and conducted. Onward and upward!

 

Squeezed Science – Should We Switch to a Business Mindset?

 

By Jesica Levingston Mac leod, PhD

It is a common conversation topic among researchers, but it was not until the NPR article saw the light, and the dark side, that the public realized the problems that young scientists are facing when pursuing a successful career in Academia. As we raise awareness about these tribulations, my colleagues mentioned how a “postdoc”’s quality life depends on the quality of the lab, the institution, the project, the relationships with colleagues and the Principal investigator or PI (the boss), not forgetting that this is a very self driven career. So, if your hypothesis is very difficult to prove, or you have been hitting your head against the wall with all the negative results that took you years to get, you may eventually come to hating this path and leaving Academia. The same if you have been working in a non “hot field” where the funding sources do not consider interesting enough to support or your PI is not supportive, or you have a very wicked competence inside or outside the lab. All these negative situations can aggravate the perspective of the very little options one may have by pursuing a career in Academia. On the other hand, if you are obtaining excellent results, publishing in top tier journals, made hundreds of good connections and collaborators, have a “great boss” and literally love you job… well, probably you are also doomed…

One solution could be implementing the business approach to the scientific mindset: Why only having one PI per lab? At the end, two minds think more than 1. Perhaps collaborative research centers have a solution were 2 or more PIs can have access to more equipment, grants and professionals, and therefore use the best skills needed for the job, like a company where you have an executive committee and you distribute the stock between the employees, in order to make them be part of the enterprise.

Having a business mindset would mean to have a planed strategy about your career development. Having a backup career plan is one example of this: starting to apply for jobs before needed, or before it is too late. Begin with your preparation to be a leader, and make your PI know, and discuss a good starting point. Look for leadership opportunities in any situations, such as coordinating workshops or conferences.

Sign up to run workshops and career developing series!. Many postdocs can discover a great professional gain if these opportunities would be offer to them. Get training in other expertise to be competitive in, for example, the investing or consulting field. Taking classes about how to give a class is a great example of a course that could be offered to postdocs and graduate students, in order to train them to explain and transfer their empirical knowledge to the next generation.

A month ago, at the Mount Sinai Postdoc symposium, Dr. Bruce Alberts (yes, THE Alberts,  from “The Molecular Biology of the Cell” book) who spoke about “The Future of Biology: Keeping Science Healthy” and illustrated the dramatic changes in the age of the scientist successfully obtaining project grants from NIH. In contrast to 30 years ago, the average age of new investigators with PhD at initial RO1 was 36.8 year old, a large number of grants were awarded to scientist in their early 30s, but this tendency has been decreasing drastically, to the point where now, the mean age for receiving these prestigious grants is 42 years of age. Dr. Alberts, himself, made fun on the fact that he obtained his postdoc position, before been awarded with his PhD. (which actually his thesis was rejected the first time, delaying the whole process) and learned from his failures. He also pointed out that he got his professor position at a very young age, something that is almost impossible nowadays. He advocated for a change in this unfair situation, which cripples the young innovators from getting a start. Also, he encouraged researchers to get out of the lab and talk to the public about science and its importance. First, to attract/engage curious minds to the scientific field, and second to communicate “in simple language” what we do for 9 hours plus per day in the lab.

We must offer to all this new scientific minds the reality about the current situation of science, but we also need to fix it, so it is not going to turn into a snow ball and make disappear all the interest in pursuing a scientific career for the new generations. In a business mind-set we must recognize that the money is not only in the governmental funding, but also in private foundations and other organizations like angels or venture capitals. So go out there and try to pitch your science to investors.

Should Postdocs Jump The Academic Ship?

By Elizabeth Ohneck, PhD

 

A recent series of articles on NPR called “Science Squeeze” painted a rather abysmal picture of the current state of scientific research, from lack of funding, to job shortages for young scientists, to stories of scientists “giving up,” leaving academia for other, though not always better, ventures. The article “Too Few University Jobs for America’s Young Scientists” features interviews with a few postdocs at NYU about their current situations and their prospects for an academic future. Their responses are not altogether negative, but are far from resoundingly positive. The article also hints that PhDs may be better off pursuing careers outside of academia, a path that more and more graduate students and postdocs are beginning to take. To get a broader perspective on how the current scientific research climate is affecting the career trajectories of postdocs, I talked with several postdoctoral scientists at varying stages of their careers about their reactions to the NPR series and how the issues presented affect their outlook for the future.

 

Not all postdocs are ready to jump the proverbial ship when it comes to pursuing an academic career, despite awareness of the hurdles ahead. Dr. Randy Morgenstein, a senior postdoc an Ivy League university, pointed out the limited scope of the NPR series, which focused on only a couple specific universities and individuals whose situations were particularly dire, and felt the articles portrayed the academic environment in an overly gloomy manner without actually addressing the overarching flaws in the system. “The articles make a pity party out of 1 or 2 places or people without making me feel the system isn’t working. So overall, I think they might have presented the state of scientific research in this country in too much of doomsday state… A better approach would have been to make me feel bad for society because good scientists are unable to get grants and do research.” He acknowledges, however, the truth of difficulties in obtaining grants and the competition for an extremely limited number of faculty positions. Despite these factors, he is persistent in pursuing a career in academia. “Academic research gives you the most freedom to pursue the research you are interested in. I like that aspect of it and think it is worth the risk to pursue.” When asked how one might overcome the obstacles in funding and faculty position availability, he responded, “I think anyone becoming a PI has to be self-confident almost to the point of arrogance, and therefore think that it’s the other people who won’t be able to get grants.  I do not think I am doing anything special to overcome these difficulties. Same as everyone else, I am trying to publish the best papers that I can, hopefully on a topic that people think is worth funding in the future.”

 

What about those who have successfully made the transition from postdoc to assistant professor, who might provide hope for those postdocs still set on an academic track? Dr. Francis Alonzo III is one such scientist, having recently obtained an assistant professor position at Loyola University Chicago. He chose to pursue an academic career because of his love of science and education, and credits his success to persistence, passion, drive, and curiosity. In addition, he added, “I really just could not see myself doing anything else. Because of that, I knew what my goals were from the start and worked as hard as I needed to get there.” But he feels that the NPR series accurately portrayed the state of scientific research, and this reality of uncertain funding means securing an assistant professorship doesn’t necessarily relieve his apprehension. “I do still love engaging in the scientific process and being involved in training and educating students,” says Dr. Alonzo. “And I still get a lot of joy coming into the lab everyday. However, I am considerably more apprehensive about what the future holds. In particular because I am just gearing up to submit my first larger grants and I have no idea how my ideas will be perceived.”

 

There are, however, many postdocs struggling to find jobs, and many who are turning away from academia in hopes of finding more opportunities. Dr. Bree Szostek Barker, a junior postdoc at the University of North Carolina, originally planned to pursue an academic career, but has recently been looking into possibilities outside of academia. She feels the NPR series actually understated the severity of the problems with funding and the job outlook in academic research. “The articles’ focus on a few universities, namely Baylor and Virginia, makes it appear that this is an issue isolated to a portion of schools/institutions/researchers that overextended during good times,” she said. “Every university and the vast majority of PI’s are feeling this, with the exception of the select few who are immeasurably successful.” The lack of job security created by limited academic positions and uncertain funding resulting from the current system of the academic research sector has pushed her to explore alternative careers. But securing a job in the private sector or a job that is not research-based has turned up its own set of problems; specifically, PhDs and postdocs seem to be missing relevant experience in the eyes of recruiters for these positions. For this reason, Dr. Szostek Barker disagrees with the assertion made in “Too Few University Jobs for America’s Young Scientists” that there are abundant jobs for PhDs outside of academia. “The fact is the number of jobs seeking a PhD with no experience in their industry is low and to pretend otherwise is offensive. And the jobs that do arise are so heavily competed for that the chances of getting the position is extremely slim,” she said, adding, “Unfortunately academia doesn’t count as ‘experience’ for anything except academia.”

 

It seems that the NPR series may have portrayed academic research in too much of a doom-and-gloom state, but also didn’t delve deep enough into the overarching problems in the structure of the scientific research sector. Funding is difficult to obtain, and faculty positions are few. Yet there are success stories to be found, and there are postdocs maintaining a hopeful outlook in spite of the enormous obstacles they face. But the system in which each PI trains multiple successors is unsustainable, and so to overcome job shortages, many postdocs are looking outside of academia for careers. What is not acknowledged in this series is that these non-academic jobs may be equally as hard to come by. Altogether, the consensus is that the system is flawed. But how do we fix the system? More money alone is likely not the answer. What contributes to one’s success on the academic track? Plenty of bright, passionate, confident, motivated scientists end up leaving academia, unable to secure funding, or worn down by the fierce competition, so what factors, both personal and academic, allow some to flourish while forcing others out? And finally, how can we better prepare PhDs for jobs outside of academia? The NPR series has brought these issues to the public eye. Hopefully this exposure will drive further discussion and a search for solutions to ensure a future full of happy, fulfilled scientists and prolific, productive scientific research.

 

What Are PhDs?

Gaia Vasiliver-Shamis

A couple of weeks ago I came across Google Poems, apparently it’s a Reddit page with some quite hilarious poems, the ones generated by Google Instant, which is the official name for the “auto-suggest/ complete” feature that I’m sure you’re all familiar with when you start typing a search query in Google.  And so, one “poem” really caught my attention:

googlepoem1

 

 

 

I wasn’t sure if I should laugh or cry, it’s clearly too late for me, but I did wonder what if I have googled this BEFORE applying to grad school? Would it change my course of action? Do people today Google this when considering going to grad school?

Now you have to understand, that autosuggest function is based on pulling data from the ginormous database of Google for what people type in the most! Which means the top four results above are based on the most typed in real queries.  And thus I decided to tackle each one individually:

PhDs are useless

Ahm…this is the toughest one. From the one hand, PhD is a prerequisite to stay in the ivy tower, but from the other hand, there aren’t that many prospects in academia these days, and from the third hand (if one existed), most of us don’t stay in academia anyway and fervor off-the bench jobs.  So if it’s not a true scientist job that we want, why spend 5-7yrs training to be one?

I’d argue that the skills you acquire might be worth it. While getting your PhD, you’ll (most likely) become good at:

[unordered_list style=”green-dot”]

  • Problem solving and troubleshooting
  • Learning new topics FAST
  • Presenting, especially to not-necessarily friendly audience
  • Being grilled and not taking it personally

[/unordered_list]

I know it’s hard to see that you actually have those skills, when you are surrounded by PhDs, but when you’re out in the “real world”, you’ll discover that most people are not so good at problem solving, public speaking and being criticized without bursting in tears. I’m not saying you need a PhD to have these skills, but you sure get them when you get your PhD. On a positive note, in 2011, 58% said a PhD is not a waste of time in a poll on The Economist.

Conclusion: Neh… we are NOT useless!

[hr]

PhDs are doctors vs. PhDs are not real doctors

Technically and historically speaking, hell yeh! Of course we’re REAL doctors, “fake” doctors existed before medical doctors did. Unfortunately my thesis wasn’t titled “The Stems of the Public Misconception that only Medical Doctors are Real Doctors?”  (And this sounds like a humanistic PhD thesis topic anyway; and humanistic PhDs are not real doctors), and so I have to put my scientist hat down and admit – I don’t have a definite answer.

Speaking of real vs. fake doctors, I have a confession to make: my (then) 3.5yrs old son complained that his knee hurts and he needs to go to the doctor, I told him: “I’m a doctor and you’ll be fine”. His teacher then told him:  “your mom isn’t a real doctor”, and to that I say (in my passive aggressive way) – do you have a PhD? And if you don’t – don’t tell me I’m not a real doctor! I used to chart my son’s growth chart by myself since I didn’t trust his “real doctor”’s ability to accurately plot data.

Conclusion: it clearly depends what your PhD is in, but we are REAL doctors.

[hr]

PhDs are arrogant

The following paragraph should be read in a British accent:

Arrogant? Pfff…of course we are….not! We just think we’re brilliant. Does that count as arrogant?

But if we can’t figure out how to change the display so we can start our presentation, change a lamp or order wine, please don’t use the fact we have a PhD against us!

Based on Google, the best answer for why PhDs act so arrogant is: “Who knows, they are not real doctors anyways”.!

My sassy response: I bet you’d be arrogant if you got a PhD

My classy response: I’m trained to look at things differently and critically, deal with it!

My Big Bang Theory response: Bazinga!

Conclusion: it depends who you ask

[hr]

But you know, maybe we PhDs shouldn’t feel so bad about ourselves, after all, check out this poem (assuming googlers mean doctors are MDs):

googlepoem2

 

 

 

Final conclusion I: I take useless and arrogant over evil and dangerous any time.

Final conclusion II: Google is out of suggestions when you type in “MDs PhDs are…” – so being Dr2 is a safe career bet. No prejudice there.

 [unordered_list style=”green-dot”]

  • Yes! This screenshot was taken on the day Google honored Dr. Schrodinger, as like Erwin Schrodinger, PhD. So maybe one should Google:  do PhDs have a higher chance of getting a doodle on Google? I leave this one up to you….
  • No! I have nothing against MDs. I swear!

[/unordered_list]

Dr. Gaia Vasiliver-Shamis didn’t become a real doctor as she can’t stand seeing blood. She’s definitely not arrogant (most of the time) and aims to do at least one useful thing a day.

This post was originally published on BioCareers.com

5 Ways to Make the Most Out of Your Grad School Experience

Welcome to grad school, you are on your way to adding 3 magical letters at the end of your name. As we’d like y’all to start well-informed and be prepared, our brilliant contributors share their wisdom and best advice on making the most our of your grad school (and beyond) experience!

That’s our top 5:

  1. Run while you still can! Just kidding….
  2. Learn new things and learn all the time and it will all come together at the end, we promise!
  3. Take a careful look of the PI personality and lab’s dynamics when choosing a lab.
  4. Keep it balanced, as in stay healthy!
  5. Diversify your experience at the bench and beyond it.

Now read on: Continue reading “5 Ways to Make the Most Out of Your Grad School Experience”

Follow the Yellow Brick Road

Neeley Remmers

Have you ever been asked the question, “Oh, you’re getting your PhD, what do you plan on doing with that?” I get asked that question on average 5 times week or more when I’m trying to explain to someone outside of scientific research what it is I do. Explaining the career-path of a scientist is no easy task because quite frankly, there is no defined career-path like for our profession like there is for other professions. For example,  if you’re ambition was to become an M.D. the path is has been cleared for you – you go to medical school, take your boards, do a residency, complete a fellowship, and finally get a position as an attending physician. However, for those of us who either have or are obtaining our PhD’s in science, there is no yellow-brick road for us to follow. Ask any professor or senior scientist how they got to their current position and you’ll find that no two career paths are the same.

One reason for the lack of a clearly defined path may be because there are so Continue reading “Follow the Yellow Brick Road”

Through The Looking Glass

Neeley Remmers

Let me begin this blog entry by first apologizing to those of you who had hoped to maybe learn another snippet of cancer biology or read a summary of a couple interesting scientific papers. Instead, this week’s entry is going to be more a reflection of my graduate career. Forewarning, I think my brain is still recovering from the intense past few weeks so I can only hope that my thoughts seem somewhat connected.

I have to confess, this decision is totally influenced by the fact that I just defended my thesis two days ago and I had the opportunity to interact with students still in their first couple of years of graduate school while attending a conference last weekend. The conversation I had with these students got me thinking about the roller-coaster ride that is graduate school. We all agreed that Continue reading “Through The Looking Glass”