During my first year of graduate school, I took “Biochemistry 507: Protein Structure and Function.” I got a C+ in the class, which was a failing grade by graduate program standards. The following year, I re-enrolled in the same course and was one of two students who got an A. This time, I abandoned the method that had landed me the C+ (rote memorization) and learned to evaluate how things worked to yield the results.
I learned how to read a scientific paper in Biochem 507. Before that, I would read a scientific paper in presentation sequence — abstract, introduction, methods, results and discussion — which seemed logical. I got a lot of information but didn’t think critically, which didn’t help me during the final exam when I had to apply concepts to new situations. Now I approach scientific papers with three questions:
1. What is the hypothesis? What do the authors want to show or prove? This question is usually answered by one sentence in the abstract. I don’t bother with the rest of the abstract at this point.
2. How do the authors go about showing this or proving this? The answer to this question is in the methods section. Sometimes the approach is not optimal for addressing the hypothesis, which will affect the data collected.
3. What are the data? This question is usually answered in the results section, in tables as a series of data points. I don’t bother with the paragraphs that interpret the data or graphical representations of the data; I look at just the data.
I generally avoid the “discussion” section because it usually consists of what my professor called “hand-waving about the data.” I look at how my conclusions compare with the authors’ conclusions, and whether or why there are disparities. This approach to reading a scientific paper takes time to learn. However, this approach forces you to think critically about the science behind the experiment and to understand the rationale behind the process. You will understand how the data were derived.
More questions to ask
The following questions can guide you through the process of evaluating a research paper:
What is the hypothesis? What are the authors aiming to demonstrate or prove in this study? You’ll find this information in the abstract, usually in the beginning or middle of the paragraph.
What is the primary endpoint, and is it appropriate to the hypothesis? What is the main objective of the study? How appropriate is the endpoint to the disease? For anti-infectives used in acute conditions like pneumonia, the primary endpoint is usually clinical cure. For chronic conditions like diabetes, an endpoint can be sustained blood glucose control, for example. In diseases such as cancer, the “gold standard” primary endpoint is overall survival.
What phase is this study? Is this the first human subject investigation designed to look at drug safety (phase I), efficacy (phase II) or comparison with current treatment standards (phase III), or is it a post-marketing phase IV trial? The phase of the trial will influence the sample size (the number of subjects or patients in the study) and therefore the “impact” of a data point.
What is the study design? Is this an experimental study or an observational study? The robustness of empirical evidence differs between experimental studies and observational studies, as well as within the study type. Randomized controlled clinical trials provide stronger clinical evidence than randomized crossover clinical trials. A case report is anecdotal and provides weak clinical evidence compared with a cohort study.
What is the significance of the data? What are the p values of the data points, and does this make the data statistically significant? In other words, can you “trust” these data as a representation of a true effect rather than attributing it to chance? P values less than 0.05 are arbitrarily assigned as statistically significant, while p values exceeding 0.05 may be regarded as “due to chance” or not statistically significant. In addition to statistical significance, are the data clinically significant? You can have highly statistically significant p values for the results of a head-to-head trial of a new drug versus a comparative drug but demonstrate no real clinical significance in the new drug (for more on p values, see the “Clinical Side” column in the July 2004 issue of Pharmaceutical Representative). Physicians consider factors like cost, compliance and side effect profiles when evaluating the clinical significance of a drug.
While it’s true that pharmaceutical sales representatives may not need to examine clinical papers with the same level of complexity as research scientists, learning to ask critical questions about a research paper can help you become more scientifically competent and credible.