Journal Club Lesson: Never rely on the abstract

by Dr. Phil on April 11, 2019

Dear Editor*:

I was happy to see a meta-analysis by Silvia Lopes et al. (DOI: 10.1177/2050312119831116) comparing elastic resistance with ‘conventional’ resistance on muscle strength. In general, I believe the analysis was adequate; however, I noted several serious errors in the manuscript that must be addressed.

It appears that the legends of Figures 2 and 3 were switched. The forest plot of Figure 2 output contains data for lower limb outcomes, while the legend is labeled “upper limbs.” More importantly, the authors have reversed their results in the results section and abstract, providing upper limb SMD for lower limb SMD, and vice-versa. Based on the forest plots, it appears the correct results should be:

Upper Limb SMD = 0.09 (-0.18, 0.35) p = 0.5

Lower Limb SMD = -0.11 (-0.40, 0.19) p = 0.48. 

Unfortunately, the authors have also mistakenly reported SMD as 0.011 in the abstract, rather than -0.11.

The authors used a fixed effect model for meta-analysis because of homogeneity of the included studies “through the I2value”; however, no I2value was reported. Furthermore, the heterogeneity of the study samples (healthy and diseased cohorts included), length of studies, and muscle groups examined would likely warrant the use of a random-effects model to strengthen the validity of the analysis. In fact, the authors go on to note, “Regarding health, the sample varied from physically active individuals and athletes to individuals with coronary heart disease and moderate COPD.”

In the Results section, as well as the abstract, the authors stated that eight studies were included. When examining the characteristics of the studies, the authors only reference studies from their bibliography (1, 2, 3, 4, 6, 7, 8), apparently having dropped reference #5 (Webber et al. 2010) which was not included any analysis of the results through the forest plots in Figures 2 or 3. 


The PRISMA chart in Figure 1 reveals more inconsistencies compared to the text. In addition to showing 7 studies (rather than the stated “eight” in the text) included in the final analysis, Figure 1 indicates that 29 were screened and 22 excluded; however, the text indicates the authors screened 23 and excluded 15.

Under Methodological Quality Of Included Studies, the authors identified the number of studies with certain characteristics; however, the superscripts are not consistent with the text. For example, the authors state that ‘four’ studies scored “7”, but they superscripted only 2 references.

In the Discussionsection, the authors note that references 16-18 compared elastic resistance to a control group; however, only 1 of these review studies focused on elastic resistance (reference 16); the other 2 reviewed all types of resistance training, including elastic resistance.

As a journal editor, I was very disappointed to see these errors published in this pay-for-publish open access journal. These obvious errors should have been noted in peer review, which leads me to question if adequate peer review even occurred.

I assume the authors paid the $1125 fee to have their article published. In today’s world of predatory journals, these types of errors indicate the possibility of poor editorial and peer review process, leading to numerous critical errors that affect the article’s validity. It’s unfair for readers to be presented with such an erroneous publication. Busy clinicians reading these articles are often limited in time, and they must often trust that the publication is free of errors to ensure validity.

Meta-analytical designs are generally rated as the “highest” level of evidence: “level 1” evidence. However, the level of evidence should always be limited by the quality of the paper. Unfortunately, the errors in this paper go beyond quality assessment. 

A quality assessment tool may not have even identified these errors. These errors are generally detected only upon focused review of the paper, which should have been done during the editorial and peer review process. Your readers should not have to bring these errors to your attention. This paper is an example of the importance of reading the full publication rather than relying on the abstract alone (which in this case was also wrong). 

I applaud the authors for completing this research, and I doubt these errors change the outcome of the analysis, as we have noted similar findings of no difference in EMG activity between elastic and isotonic resistance1. However, the authors and journal editor have a responsibility to correct these errors immediately (as well as the Pubmed abstract) and ensure their conclusions remain valid. Furthermore, I suggest the journal editor should review their editorial process to ensure these mistakes do not occur again.

Phil Page, PhD, PT, ATC, LAT, CSCS, FACSM

1. Aboodarda SJ, Page PA, Behm DG. Muscle activation comparisons between elastic and isoinertial resistance: a meta-analysisClinical Biomechanics. 2016. 39:52-61.

*This is a letter to the editor of SAGE Open Medicine, an open-access, pay-for-publish journal. Ironically, my letter has to go through peer review before considering for publication. And as of now, there is no fee for publishing letters to the editor.

{ 0 comments }

Page P. 2017. J Performance Health. 2(2)

Today’s approach to “evidence-based practice” (EBP) should be a process rather than an end-point. In this article published in the Journal of Performance Health, I recommend a 6-step process for a pragmatic approach to EBP….beginning with asking the question and ending with analyzing the results: AAAAAA.

The AAAAA’s of Pragmatic EBP

Want some help to better utilize research in your practice? Need to get ‘up-to-date’ on the latest research? Are you frustrated or confused, or simply scared of the process? If you have a clinical question and need some direction in finding the answer in the scientific literature, just ask the Fonz, and he’ll say “AAAAAA!”: Ask, Acquire, Analyze, Answer, Apply, and Assess 

ASK

What is your question? Be very specific in what you want to know. The more specific, the better. Broad questions will lead to information overload; but being too specific may narrow your results too much. So, striking the right amount of detail in your question is important. And it might take some tweaking along the way, either adding or subtracting details from your question. Specifically, think about keywords related to your question—that’s how most information is categorized now—and provide keywords to these categories:

  • population
  • body part
  • disease or injury
  • intervention
  • outcome

For example, if your question is, “Will kinesiology tape help reduce swelling in patients with lymphedema after breast surgery,” you might consider these keywords to use:

  • population = females
  • body part = upper extremity
  • disease or injury = breast cancer, lymphedema
  • intervention = kinesiology taping
  • outcome = swelling

Remember, you might need to add some details along the way such as an age range for your population, or other outcomes such as range of motion. 

ACQUIRE 

Now that you have your question and keywords, it’s time to search the literature.  The Internet offers so many easily accessible resources today; however, you need to know where to look and what to trust.  Since one source typically doesn’t provide all the information you’re looking for, don’t rely on only one source for all your information; however, you’ll probably want to limit your search to 2 or 3 reputable sources. 

Finding the articles

Remember that different sources have different ‘levels’ of information that can range from original research articles you’ll need to interpret (like raw data), to more “consumerized” articles that may or may not provide sufficient or even credible information.  Obviously, you want to have a good scientific basis and evidence for your answer, but you want to gather and interpret the information in an efficient manner. Here are some good freesources for you instead of simply entering your keywords into Google search bar. Note: there are research databases such as CINAHL and SportDiscus that include journals that might not be indexed in Pubmed; however, these databases often require access through a university library.

  • Pubmed: The “Granddaddy” of online scientific searching contains the most reputable database of research papers. You can enter keywords with Boolean searches (using “AND”, “OR”, “NOT”) to limit your results. You can also filter your results, limiting them to clinical trials or reviews for example. Sometimes, meta-analyses and systematic reviews have already been done to answer your question, so look for those first by filtering for “Reviews” in Pubmed. And if you create a free account, you can save your searches and create alerts of new articles using keywords. Unfortunately, you will only have access to the abstract; however, some full articles are provided for free. And note that Pubmed does not index allrelevant journals! (www.pubmed.com)
  • PEDro: The free Physiotherapy Evidence Database contains over 37,000 randomized trials, systematic reviews, and clinical practice guidelines from around the world. (www.pedro.org.au)
  • Google Scholar: Focus your Google search to research articles by using Google Scholar’s free search. (scholar.google.com)
  • Industry websites: Industry-sponsored websites often contain research specific to their products. While there is a potential for publication bias in these sites (but not in TheraBandAcademy.com!), you may quickly find research to help answer your question. Other sources to find answers may include industry association websites like www.apta.orgor www.acsm.org, which may publish position statements or clinical summaries.

Narrowing the results

When you start to comb through the search results, start by looking at the title. If the title is relevant to your question, then read the abstract. If the abstract is relevant to your question, regardless of the conclusion, save the abstract/article. You can print them, copy and paste, or save within your PubMed account, for example. Next, you’ll want to get access to the full articles.

Unfortunately, many full articles are difficult to attain without university library privileges. Aside from purchasing the full article directly from the publisher website at a hefty price (usually $20-40), there are a few things you can try. 

  • Professional organizations like the APTA sometimes provide free access to their online journal databases for members (www.ptnow.org)
  • Type the title of the article you’re looking for in Google to see if the full article is accessible

Staying ahead of the curve

At this point, you might want to set up email “alerts” on your topic if you want to know when the very latest research comes out. There are several ways to do this:

  • Set up Pubmed alerts using keywords
  • Sign up for relevant journal table of contents alerts for new issues and articles at their home pages
  • Use a service like www.AMEDEO.com to send you new article titles from journals
  • Use an app like QxMD and select the categories and alerts you want

Once you have access to the full article, you can start to review it. You can often find more articles by ‘cross-referencing’ the cited articles in a study as well.  Ideally, you should take time to critically review the full article if possible; never rely on the title or abstract conclusion alone. Without analyzing the full article, you will not know if the study is applicable to your question 

ANALYZE

Once the title or abstract piques your interest, it’s time to further analyze. Let’s start with the anatomy of an abstract or article. (Note: While all research articles should have abstracts, not all abstracts have a research article!). Articles and abstracts generally follow an outline that may or may not include these 5 sections:

  • Background & Purpose
  • Methods
  • Results
  • Discussion
  • Conclusion

Now let’s examine the contents of the different sections.

Background & Purpose: Authors should provide appropriate background information that supports the rationale for the study, leading to the research question and/or hypothesis. The purpose of the study should be clearly stated. Sometimes, researchers use “null hypothesis” testing to establish statistical significance. 

Methods: The study design should be identified here, and should address the research question/purpose (more on that later). The key items to look for in the methods section of clinical trials are known as “PICO”: Population, Intervention, Comparison Group, Outcome. Note that other study designs may not include these specifically. The statistical analysis procedures should be identified at the end of this section. Should follow reporting guidelines at www.equator-network.org

Results: This section provides the specific results of the statistical analysis, quantifying the outcome measures. This might include averages and statistical significance. Look for clinically-relevant outcomes such as effect sizes, confidence intervals, and minimal clinically important differences.

Discussion: The authors should address the results and provide context relative to the purpose. Limitations and threats to internal and external validity (bias) should be addressed here as well. Clinical implications and need for further research should also be mentioned.

Conclusion: The conclusion should answer the research question and be consistent with the findings within the context of the study limitations.

Now that you have a feeling for the anatomy, let’s get down to the analysis. In order, these are the things you should consider when analyzing an article. 

  1. The research design & level of evidence
  2. Quality of the research, including bias
  3. Impact of the evidence (clinical statistics)
  4. The source

Level of Evidence

Research articles come in all shapes and sizes. In other words, there are several study designs that are used to answer a research question. In general, there are 2 types of research designs: experimental and non-experimental. Each type has sub-types with various designs:

These different designs have different “levels” of evidence as well. The Center for Evidence-Based Medicine (www.CEBM.net) provides 5 hierarchies of evidence. The 2011 levels are designed as a “short-cut for busy clinicians, researchers, or patients to find the likely best evidence.” In other words, certain study designs are stronger than others, and are more likely to provide the answer to a clinical question.  In general, here are the 5 levels: Systematic Review/Meta-Analysis, Randomized Controlled Trials, Cohort Studies, Case Series, and Mechanistic Reasoning.

Quality

The quality of the evidence is as important (if not more) than the level of evidence. A poorly designed or reported randomized control trial may be of little benefit in clinical practice. Unfortunately, this requires clinicians to understand how to identify potential bias within studies that may influence the results, or bias that may influence the clinical implementation of the study. This is known as threats to internal and external validity, respectively.

Unfortunately, there is an inherent weakness in clinical research: “sterility.” While clinical populations are often heterogeneous, researchers must create narrow and focused parameters of subject populations (homogenous) to control for potential bias. This effort to maximize internal validity may compromise external validity.  On the other end of the spectrum, some researchers lump patients with the same clinical diagnosis together, regardless of etiology, which may influence outcomes. For example, in a population of patients diagnosed with “anterior knee pain,” a hip strengthening intervention may be more effective in patients with hip weakness, compared to patients with quadriceps weakness…even with the same diagnosis.

Busy clinicians can assume studies possess internal validity if the full article is published in a highly credible journal source. While some amount of bias is expected in research, it’s important that the authors do their best to minimize within the design, identify the potential for bias in the study, and discuss how bias may influence the study results or implementation.

While overall quality is associated with internal validity, there are several scales and tools used to assess other aspects of research bias in different study designs. One of the most commonly used in clinical rehabilitation trials is the “PEDro” scale. With PEDro, studies are graded on 11 criteria such as blinding and randomization.  Other scales combine the quantity, quality, and level of studies to make overall recommendations, such as the GRADE scale.

Impact of the Evidence

Remember that scientific research and statistical significance is based on probability, and that statistical significance really doesn’t have much value for clinical interpretation. The P-value only gives us the probability that the researcher made the right decision in accepting or rejecting the null hypothesis. We must make certain assumptions to apply the results of a study to other individuals in a population (such as a representative sample). Yet, as clinicians, we know that every patient and situation is different. 100% of patients don’t have successful treatment in real life or in studies, so there’s no guarantee that the results of one study will be seen in your clinical population. 

Since statistical significance is of little value in a clinical setting, clinically relevant statistics are now recommended for reporting clinical study results. A statistically significant outcome may not have clinical relevance. For example, a pain study may report that one group significantly reduced their pain over another group; however, the difference was only 1 point out of 10 on the Visual Analog Scale. Depending on the diagnosis, a difference of 2 may be more beneficial for patients. This number is known as the “minimal clinically important difference” or MCID, and these are specific to the outcome measure and the population.

Another disadvantage of statistical significance is that the P-value does not give us information on the size of the difference between groups (magnitude) or the direction (harm vs. benefit). Effect sizes can quantify the magnitude of the difference between groups and provide direction on harm or benefit. A large effect size is often seen as clinically useful, while a smaller effect is seen as “trivial.” 

Finally, confidence intervals provide more clinically-relevant information, including an estimated point and range of the true value of a parameter within a population. Confidence intervals can be used to estimate effect sizes or outcome values. That’s the beauty of confidence intervals. We can predict with a high level of confidence (usually 95% confidence) that the “true” value of an outcome lies within a certain range.

Unfortunately, some studies do not provide effect sizes or confidence intervals; however, there are several online tools to help you estimate these values if the study provides the necessary data. Using clinically-relevant statistics can help improve your interpretation of study results.

Source of Evidence

Finally, consider the source. Is there a full article available with sufficient detail provided for analysis and replication? Research abstracts such as those presented at national meetings should have less influence on answering your question than full length articles. Also consider the publication. High impact, indexed peer-reviewed journals with a blinded review process are considered the best source. Secondary sources such as textbooks and trade magazines offer less quality sources.   

Do not rely on the journal name to establish credibility; there are many ‘predatory journals’ with fancy titles that offer a ‘pay to publish’ scheme, often for poor quality studies. And, even indexing in the most prestigious Pubmed does not guarantee the quality of a journal or article!

In summary, try to answer these “R” questions when analyzing an article to answer a clinical question:

Being an informed research consumer

Researchers sometimes hide bias in their study, which may not be obvious to the typical clinician. More often, the media, manufacturers, and so-called “gurus” exhibit bias in their reporting though the subtle art of “pseudoscience.” Unfortunately, some gurus and others use the words “research” and “science” inappropriately to support their message. Because of this, clinicians should be skeptical of claims without being cynical, both in and outside of research. Here are some common signs of pseudoscience :

pseudoscience

Today, many companies are using ‘research’ to support their marketing efforts. Don’t just take their word for it: you need to ask companies to back up their claims with solid evidence. According to the Federal Register, the FDA offers guidance on the research that companies can base their medical device claims, using “adequate and well-controlled” studies:

  • Clear statement of objective and methods
  • Valid design (only level 1-3) with control (blinded and randomized)
  • Subjects have disease or condition
  • Methods are well-defined and reliable
  • Appropriate statistical analysis for effects

ANSWER

Now you’ve gathered and analyzed the evidence…. What’s the answer to your question? It may not be as obvious as you think. There are 2 questions to consider when you read a research paper: is the answer to the research question in each study appropriate; and, what’s the answer to your clinical question. Sometimes, your answer will already be in a meta-analysis, which generally pools data from various quality studies. But meta-analyses have limitations: date of publication, inclusion/exclusion criteria limitations, etc. As with anything, your clinical decision should not lie with only one study, since evidence-based practice combines patient presentation and clinician experience, but be sure you have the best available evidence first!

What if you have 10 studies on your topic that have different conclusions? If you are faced with multiple studies on an intervention with different outcomes, look for reasons why they may be different; specifically, PICO. And of course, look for overall bias in each study and attempts to minimize it.

  • Population: Compare inclusion/exclusion criteria and recruitment.
  • Intervention: Were treatment parameters clearly reported and similar
  • Comparison: Was the comparison treatment (control) truly a control? 
  • Outcome Measures: Were similar tools used to measure the same thing, such as balance? 

APPLY

This section is easy: put your answer into practice. Using the parameters of evidence based practice, if you’ve analyzed the best available evidence, combine it with your clinical experience and the patient presentation.  If it’s a clear answer (ie, Yes or No), then great. If not (ie, inconclusive or conflicting), use your clinical judgement. 

There are a few factors to consider when deciding to apply your findings:

  • Harm vs. Benefit: Does the benefit of the outcome outweigh potential harms, potentially in comparison to the standard treatment?
  • Cost-Benefit: Does the benefit of the outcome outweigh the cost of implementing the treatment?
  • Feasibility / reproducibility: Can the treatment be implemented in your setting?

Don’t forget to share your findings with your colleagues. Having a journal club or other research meeting regularly helps facilitate evidence-based practice to support clinical decision making.

ASSESS

Lastly, you should assess the impact of your clinical decision. Was your answer correct for your situation? How beneficial was your decision? Researchers have developed a system for quantifying that: RE-AIM (www.re-aim.org).  Specifically, RE-AIM quantifies:

  • Reach: how many benefited from the intervention in the target group?
  • Effectiveness: how effective was the intervention (in measurable outcomes)?
  • Adoption: how many individuals went on to continue using the intervention regularly?
  • Implementation: how consistently is the intervention performed, and how costly?
  • Maintenance: How did the intervention affect individuals over time?

For example, if your clinic decides to use a new kinesiology taping technique in patients with lymphedema after breast cancer surgery, use the RE-AIM model to assess how well the intervention worked in patients, as well as in your clinic. RE-AIM can assess how well the intervention was adopted and implemented within clinicians as well as patients. By quantifying its success, you not only support the use of an evidence-based intervention, but you may also have data specific to your clinic to support marketing efforts in the community!

Staying up-to-date with the literature can be very time-consuming for the busy clinician. The “6A” approach helps provide an efficient framework to facilitate the process of answer clinical questions with research to support evidence-based practice. 

Do you have any questions on how to improve your clinical decision making with research? Post them below or contact me!

{ 0 comments }

Safer Pain Relief Summits: Delivering the Latest Techniques for Pain Relief and Movement

February 26, 2018

Identifying, rehabilitating and preventing musculoskeletal dysfunctions is not only a primary component of pain management, but is fundamental to facilitate and reinforce proper movement patterns. As you and your fellow hands-on healthcare professionals seek to become the experts in pain management, obtaining continuing education credits (CEUs) that encompass the observation of whole movements, regional independence, […]

The post Safer Pain Relief Summits: Delivering the Latest Techniques for Pain Relief and Movement appeared first on Performance Health Academy.

Read the full article →

Pelvic Asymmetry 101: How to Mediate the Misaligned

February 19, 2018

For us desk jockeys sitting for eight hours a day at work, proper pelvic alignment and overall workplace ergonomics are constantly preached so that we may be kind to our bodies and maintain the integrity of our musculoskeletal system. For those who are permanently wheelchair-bound, on the other hand, proper pelvic alignment is even more […]

The post Pelvic Asymmetry 101: How to Mediate the Misaligned appeared first on Performance Health Academy.

Read the full article →

5 Important Pieces of Advice for the New Physical Therapists

February 15, 2018

Congratulations, you made it through the hard part— school, licensing, job; check, check, and check. You’re in this field because you understand the importance of preparation and you have a passion for helping others, but now you’re ready to excel in clinical practice. It’s important to seek wise counsel when mapping out a strategy to […]

The post 5 Important Pieces of Advice for the New Physical Therapists appeared first on Performance Health Academy.

Read the full article →

Take Advantage of Two Free Wheelchair Seating and Positioning Webinars

February 5, 2018

Successfully managing seating and positioning issues can improve patient outcomes and function with these 45 minute educational webinars. Join Dr. Kirsten Davin, a recognized authority in Seating and Positioning for free educational webinars and learn from an expert! Pelvic Asymmetry Webinar February 27, 7pm CT – The pelvis is the key point of control, and adjustments […]

The post Take Advantage of Two Free Wheelchair Seating and Positioning Webinars appeared first on Performance Health Academy.

Read the full article →

108 More Research Studies That You Need to Read… Now

December 6, 2017

If you missed the first 55 research studies and second 93 research studies published this year, catch up! Just because we’re getting into the swing of the holiday season doesn’t mean the research is stopping anytime soon. This quarter, there were 86 new original research articles added to the Academy database and 22 abstracts from the annual […]

The post 108 More Research Studies That You Need to Read… Now appeared first on Performance Health Academy.

Read the full article →

Evidence-Based Musculoskeletal Pain Relief is a Spray Away

November 22, 2017

As you’re reading this sentence, millions of people are struggling with some form of musculoskeletal pain. This notion isn’t new to you, though; you treat these patients day in and day out in your clinic. Thankfully, more clinicians are wising up to the detriments of opioid prescriptions to treat musculoskeletal pain, and these pain sufferers […]

The post Evidence-Based Musculoskeletal Pain Relief is a Spray Away appeared first on Performance Health Academy.

Read the full article →

Dynamic vs. Isometric Resistance Training for Knee Osteoarthritis

November 15, 2017

Osteoarthritis is a progressive condition that can significantly hinder a person’s ability to get around and negatively impacts their overall quality of life. Osteoarthritis is the most common joint disorder in the US and the second most common disability in older adults. When osteoarthritis impacts the knee, it’s especially limiting and can eventually prevent the affected […]

The post Dynamic vs. Isometric Resistance Training for Knee Osteoarthritis appeared first on Performance Health Academy.

Read the full article →

Don’t Rely on Your Ankle Brace in The Off-Season

November 8, 2017

Ankle braces are great for preventing injuries in athletes who participate in sports with high risk of ankle injury. The question remains; should athletes wear ankle braces even during the off-season? A study recently released by the Journal of Performance Health suggests that performing neuromuscular ankle training during the off-season without wearing an ankle brace […]

The post Don’t Rely on Your Ankle Brace in The Off-Season appeared first on Performance Health Academy.

Read the full article →