United States: Guest Post − Peering Behind The Curtain: Taking A Closer Look At Peer Review And Predatory Journals

Last Updated: January 4 2019
Article by James Beck

This guest post is from long-time friend of the blog Bill Childs, from Bowman & Brooke, who also wishes to thank Elizabeth Haley for research assistance. It's a reworking of a piece on bogus scholarly literature that Bill previously published here. We thought it was both good and relevant enough that we approached Bill with a request to re-run it as a guest post on the Blog, and he graciously accepted. As always, our guest bloggers are 100% responsible for the content of their posts (and here that disclaimer also extends to B&B and its clients), and deserve all the credit (and any blame).

**********

The Daubert court, in interpreting Rule 702 of the Federal Rules of Evidence, laid out various non-exclusive criteria for consideration in evaluating proposed scientific evidence, one of them peer review. As the Court put it: "The fact of publication (or lack thereof) in a peer reviewed journal...will be a relevant, though not dispositive, consideration in assessing the scientific validity of a particular technique or methodology on which an opinion is premised." Daubert v. Merrell Dow Pharms., 509 U.S. 579, 594 (1993). Peer review, or the absence thereof, was mentioned repeatedly by the New Jersey Supreme Court in endorsing Daubert in the recent decision in In re: Accutane Litigation, 191 A.3d 560, 586, 592, 594 (N.J. 2018). Among other things, the Court noted that the plaintiffs' expert had not submitted "his ideas...for peer review or publication," considering that failure to be a strike against his methodology. Id. at 572.

Compared to other Daubert factors (or those described in the subsequent comments to Rule 702), the presence or absence of peer review may seem more binary than other factors - i.e., easier for a court to evaluate - it's either there or it's not, it seems. Not so, either in the traditional sense of peer review or the changing world of things that now get called peer review. Given this perceived simplicity, though, it frequently gets less attention than it deserves. Litigants should think about peer review as being more complex than it appears, and in some specific contexts, additional exploration - whether through discovery into your adversaries' experts, or early investigation of your own potential experts - may make sense.

Daubert vs. Predator

One fascinating consequence of this consideration of peer review in the Daubert context is the potential for experts publishing litigation-related work in what are called "predatory journals" (sometimes also called "vanity publications)." See Kouassi v. W. Illinois Univ., 2015 WL 2406947, at *10-11 (C.D. Ill. May 19, 2015); Jeffrey Beall, "Predatory Publishing Is Just One of the Consequences of Gold Open Access," 26 Learned Pub'g 79-84 (2013); John Bohannon, "Who's Afraid of Peer Review?" 342 Science 60-65 (Oct 4, 2013).

Predatory journals, like the eponymous Predator in the 1987 film and its 2018 reboot, camouflage themselves. They make themselves look not like the Central American jungle background, but like legitimate medical or scientific journals. Their publishers' websites generally look like legitimate publishers' websites (if sloppy at times), their PDFs look like "real articles," and their submission process might even look normal. They'll even claim to have peer review and editorial boards and all the rest of what you expect from journals. Like the Predator, they even try to manipulate their editorial voices to sound like real journals.

These journals are, however, just aping the façades of real journals. They typically do not have legitimate peer review processes - or possibly any review processes at all. Frequently, if an author pays the exorbitant fees, the submitted article will get published.

Myriad examples exist revealing such journals as frauds. My favorite is probably the publication of a case report of "uromysitisis" an entirely fictional condition - first referenced in Seinfeld as a condition from which Jerry claims to suffer after being arrested for public urination - by the purported journal Urology & Nephrology Open Access Journal. The author of the intentionally nonsensical article - not a urologist, nor a medical doctor at all - wrote about his experience here. After that article's exposure as an obvious fake, and something that even the most casual of reviewers should have rejected, the article was removed, but the "journal" is still up and publishing on the MedCrave site, described, a bit awkwardly, as "an internationally peer-reviewed open access journal with a strong motto to promote information regarding the improvements and advances in the fields of urology, nephrology and research." A few years earlier, a computer scientist published an article consisting solely of the phrase "Get me off your [obscenity] mailing list," with related graphs, repeated for eight pages. That journal remains in existence as well.

Such journals are largely set up to entrap new (and naïve) scholars who are under tremendous pressure to publish for promotion and tenure purposes - but they also can provide an opportunity for dubious expert witnesses to get something published they can cite as "peer reviewed," especially as courts more and more often note the presence or absence of peer review. It isn't news to many litigation experts that having peer review for some of their more outlandish assertions can increase the odds of their testimony being admitted. If an expert in fact has published in a predatory journal (and it can be shown that the expert knew or should have known about that fact), that fact should count against the admissibility of the testimony.

Given the camouflage, it is fortunate that there are resources and strategies that can help identify such publications. Retraction Watch, published by the Center for Scientific Integrity and headed by science writer Adam Marcus and physician and writer Ivan Oransky (full disclosure: Ivan and I are friends, based in large part on our shared love for power pop like Fountains of Wayne and western Massachusetts bands like Gentle Hen. He should not be blamed for my Predator references) while not focused solely (or even largely) on predatory journals, is an accessible look at the world of retractions "as a window into the scientific process." They keep an eye out for interesting developments in the world of predatory journals, and scientific publications generally, and their coverage is what made me suspicious when, in one of my cases, an adversary's expert's article was published by a MedCrave journal (home to the Seinfeld article). Retraction Watch's coverage of that article led to what I assume will be the only time in my career I had the chance to ask a Ph.D./M.D. if he was familiar with Seinfeld and if the show is, in fact, fiction, based on him publishing - and in fact being listed as an editor of - another MedCrave journal.

There is also a list of suspected predatory journals archived at Beall's List. The appearance of a journal on that list is not conclusive evidence that it is predatory, but it is enough to raise questions. The removal of a journal from the Directory of Open Access Journals for "editorial misconduct" or "not adhering to best practices" (see list, here) is another giveaway. The Loyola Law School's "Journal Evaluation Tool" can also provide a useful rubric, accessible to non-scientifically-trained lawyers, for evaluating whether a journal is likely legitimate or not. And your own experts can likely provide feedback to you about journals.

Most experts will not have published in predatory journals. But it is still worth the time to explore the question, especially about pivotal articles on which the experts are relying - whether the expert is your adversary's or your own. Even if the publication offer was innocently accepted (i.e., even if the author did not realize she was publishing in a predatory journal), the lack of rigor in evaluating the article by the publisher should at a minimum eliminate any weight given to the peer review factor. And if an author has intentionally published in such a journal, that should be the equivalent of an intentionally false statement in a C.V.

Not All Peer Review Is the Same

Of course, these relatively new faux journals are not the only way experts get published. Consider the most traditional form of peer review, where editors of a journal have outside reviewers, usually with their identities screened from the authors, evaluate the quality and originality of the work, confirming that the methodologies presented appear legitimate and that the conclusions reached are reasonable based on what's described. Given that those goals line up nicely with the goals of a Daubert analysis, it is sensible indeed for a court to look at that as a potential indicator of reliability - indeed, that's why peer review is a factor in the first place.

But even if a proffered expert testifies to having followed a methodology that matches something in a peer-reviewed publication, it is often worth at least a few deposition questions about the review process and a line in your subpoena duces tecum requesting copies of any materials the author has received relating to the review, or to attempt some third party discovery on the journals in question - though some courts may limit or refuse that discovery. See, e.g., In re Bextra & Celebrex Mktg. Sales Practices & Prod. Liab. Litig., 2008 WL 859207 (D. Mass. March 31, 2008) (granting protective order for non-party medical journal publisher, expressing concerns about a chilling effect). The propriety of allowing such discovery is beyond the scope of this article, but I addressed it in more detail in The Overlapping Magisteria of Law and Science: When Litigation and Science Collide, 85 Neb. L. Rev. 643 (2007).

If you get peer review notes, it's possible you'll find that a reviewer recommended the removal of a conclusion that the expert is now presenting, or that the reviewer warned against a particular inference from what is in the article. Making it even easier, some journals, traditional and, more often, "open access," are now posting their reviewers' comments online. Even if you do not find anything relevant, most experts will readily concede that peer review reflects at most an "approval" of the overall approach and is not a guarantee of correctness as to conclusions. And sometimes you'll be able to establish that the study in question was based on flawed data or that the work done for litigation did not, in fact, use the same methodology as that in the publication. See, e.g., In re Mirena IUS Levonorgestrel-Related Prods. Liab. Litig., ___ F. Supp.3d ___, 2018 WL 5276431, at *11-13, *28, *34, 37-38, *50-51 (S.D.N.Y. Oct. 24, 2018) (rejecting expert's reliance on "repudiated" open access journal article by author that did not disclose retention as a plaintiff's litigation expert); In re Viagra Prods. Liab. Litig., 658 F. Supp. 2d 936, 945 (D. Minn. 2009) (reversing an initial denial of defendants' Daubert motion after learning of flaws in underlying data and processing, noting that "Peer review and publication mean little if a study is not based on accurate underlying data."); Palazzolo v. Hoffman La Roche, Inc., No. A-3789-07T3, 2010 WL 363834, at *5 (N.J. Super. App. Div. Feb. 3, 2010) (finding no abuse of discretion in excluding an expert's conclusion based on conclusion that the expert did not in fact use the methodology claimed to have used in the underlying peer-reviewed study).

Sometimes, even in a more traditional context, the peer review that was performed was not what was likely pictured by the Daubert court, particularly when the work at issue is outside the so-called "hard sciences." In a publicized example, the review of a history-oriented book about the lead and vinyl chloride industries, authored by frequent plaintiffs' experts and published by the University of California, involved reviewers known to - and in some cases recommended by - at least one of the authors . See 85 Neb. L.R. at 660-63 (describing this situation; original book website was removed). Whether or not that review was adequate for the academic purpose, it was materially different from, say, the reviewers of a double-blind clinical trial, and the facts surrounding it seem plainly relevant to how much weight a court should give it under Rule 702 and Daubert. Without that discovery, the court may well not have learned about what "peer review" meant in that context.

Consider also the scenario where an expert says that their methodology has gone through peer review but the article has not yet been published. Again, it may be worth pursuing more details, especially if the expert seems likely to cite to that review in defending their position. If it has not yet been accepted for publication, consider requesting a copy of the comments the expert received from the reviewers. If those comments are provided, they may be helpful; if their production is refused, the fact of that review should be rejected as a basis for admissibility.

What To Watch Out For

Fundamentally, the important thing is to look through your and your adversaries' experts' C.V.s with care, especially as to articles that are directly on point with the issue you're addressing. It is not enough to think about what the articles say, and it also is not enough to think to yourself, "Well, that sounds like a legitimate journal." Look at the publishers' site; look for hints in the article itself; and do some searches. Ask a few questions of the expert about author fees and what the peer review entailed and throw in a document request to see if there is something worth exploring further. And if you are dealing with a situation with what you think is a predatory journal, be ready to teach a judge about what that means; as of this writing, no court has referenced "predatory journals" in a reported Daubert decision.

This article is presented for informational purposes only and is not intended to constitute legal advice.

To print this article, all you need is to be registered on Mondaq.com.

Click to Login as an existing user or Register so you can print this article.

Authors
James Beck
 
In association with
Related Topics
 
Related Articles
 
Related Video
Up-coming Events Search
Tools
Print
Font Size:
Translation
Channels
Mondaq on Twitter
 
Mondaq Sign Up
Gain free access to lawyers expertise from more than 250 countries.
 
Email Address
Company Name
Password
Confirm Password
Position
Industry
Mondaq Newsalert
Select Topics
Select Regions
Registration (you must scroll down to set your data preferences)

Mondaq Ltd requires you to register and provide information that personally identifies you, including your content preferences, for three primary purposes (full details of Mondaq’s use of your personal data can be found in our Privacy and Cookies Notice):

  • To allow you to personalize the Mondaq websites you are visiting to show content ("Content") relevant to your interests.
  • To enable features such as password reminder, news alerts, email a colleague, and linking from Mondaq (and its affiliate sites) to your website.
  • To produce demographic feedback for our content providers ("Contributors") who contribute Content for free for your use.

Mondaq hopes that our registered users will support us in maintaining our free to view business model by consenting to our use of your personal data as described below.

Mondaq has a "free to view" business model. Our services are paid for by Contributors in exchange for Mondaq providing them with access to information about who accesses their content. Once personal data is transferred to our Contributors they become a data controller of this personal data. They use it to measure the response that their articles are receiving, as a form of market research. They may also use it to provide Mondaq users with information about their products and services.

Details of each Contributor to which your personal data will be transferred is clearly stated within the Content that you access. For full details of how this Contributor will use your personal data, you should review the Contributor’s own Privacy Notice.

Please indicate your preference below:

Yes, I am happy to support Mondaq in maintaining its free to view business model by agreeing to allow Mondaq to share my personal data with Contributors whose Content I access
No, I do not want Mondaq to share my personal data with Contributors

Also please let us know whether you are happy to receive communications promoting products and services offered by Mondaq:

Yes, I am happy to received promotional communications from Mondaq
No, please do not send me promotional communications from Mondaq
Terms & Conditions

Mondaq.com (the Website) is owned and managed by Mondaq Ltd (Mondaq). Mondaq grants you a non-exclusive, revocable licence to access the Website and associated services, such as the Mondaq News Alerts (Services), subject to and in consideration of your compliance with the following terms and conditions of use (Terms). Your use of the Website and/or Services constitutes your agreement to the Terms. Mondaq may terminate your use of the Website and Services if you are in breach of these Terms or if Mondaq decides to terminate the licence granted hereunder for any reason whatsoever.

Use of www.mondaq.com

To Use Mondaq.com you must be: eighteen (18) years old or over; legally capable of entering into binding contracts; and not in any way prohibited by the applicable law to enter into these Terms in the jurisdiction which you are currently located.

You may use the Website as an unregistered user, however, you are required to register as a user if you wish to read the full text of the Content or to receive the Services.

You may not modify, publish, transmit, transfer or sell, reproduce, create derivative works from, distribute, perform, link, display, or in any way exploit any of the Content, in whole or in part, except as expressly permitted in these Terms or with the prior written consent of Mondaq. You may not use electronic or other means to extract details or information from the Content. Nor shall you extract information about users or Contributors in order to offer them any services or products.

In your use of the Website and/or Services you shall: comply with all applicable laws, regulations, directives and legislations which apply to your Use of the Website and/or Services in whatever country you are physically located including without limitation any and all consumer law, export control laws and regulations; provide to us true, correct and accurate information and promptly inform us in the event that any information that you have provided to us changes or becomes inaccurate; notify Mondaq immediately of any circumstances where you have reason to believe that any Intellectual Property Rights or any other rights of any third party may have been infringed; co-operate with reasonable security or other checks or requests for information made by Mondaq from time to time; and at all times be fully liable for the breach of any of these Terms by a third party using your login details to access the Website and/or Services

however, you shall not: do anything likely to impair, interfere with or damage or cause harm or distress to any persons, or the network; do anything that will infringe any Intellectual Property Rights or other rights of Mondaq or any third party; or use the Website, Services and/or Content otherwise than in accordance with these Terms; use any trade marks or service marks of Mondaq or the Contributors, or do anything which may be seen to take unfair advantage of the reputation and goodwill of Mondaq or the Contributors, or the Website, Services and/or Content.

Mondaq reserves the right, in its sole discretion, to take any action that it deems necessary and appropriate in the event it considers that there is a breach or threatened breach of the Terms.

Mondaq’s Rights and Obligations

Unless otherwise expressly set out to the contrary, nothing in these Terms shall serve to transfer from Mondaq to you, any Intellectual Property Rights owned by and/or licensed to Mondaq and all rights, title and interest in and to such Intellectual Property Rights will remain exclusively with Mondaq and/or its licensors.

Mondaq shall use its reasonable endeavours to make the Website and Services available to you at all times, but we cannot guarantee an uninterrupted and fault free service.

Mondaq reserves the right to make changes to the services and/or the Website or part thereof, from time to time, and we may add, remove, modify and/or vary any elements of features and functionalities of the Website or the services.

Mondaq also reserves the right from time to time to monitor your Use of the Website and/or services.

Disclaimer

The Content is general information only. It is not intended to constitute legal advice or seek to be the complete and comprehensive statement of the law, nor is it intended to address your specific requirements or provide advice on which reliance should be placed. Mondaq and/or its Contributors and other suppliers make no representations about the suitability of the information contained in the Content for any purpose. All Content provided "as is" without warranty of any kind. Mondaq and/or its Contributors and other suppliers hereby exclude and disclaim all representations, warranties or guarantees with regard to the Content, including all implied warranties and conditions of merchantability, fitness for a particular purpose, title and non-infringement. To the maximum extent permitted by law, Mondaq expressly excludes all representations, warranties, obligations, and liabilities arising out of or in connection with all Content. In no event shall Mondaq and/or its respective suppliers be liable for any special, indirect or consequential damages or any damages whatsoever resulting from loss of use, data or profits, whether in an action of contract, negligence or other tortious action, arising out of or in connection with the use of the Content or performance of Mondaq’s Services.

General

Mondaq may alter or amend these Terms by amending them on the Website. By continuing to Use the Services and/or the Website after such amendment, you will be deemed to have accepted any amendment to these Terms.

These Terms shall be governed by and construed in accordance with the laws of England and Wales and you irrevocably submit to the exclusive jurisdiction of the courts of England and Wales to settle any dispute which may arise out of or in connection with these Terms. If you live outside the United Kingdom, English law shall apply only to the extent that English law shall not deprive you of any legal protection accorded in accordance with the law of the place where you are habitually resident ("Local Law"). In the event English law deprives you of any legal protection which is accorded to you under Local Law, then these terms shall be governed by Local Law and any dispute or claim arising out of or in connection with these Terms shall be subject to the non-exclusive jurisdiction of the courts where you are habitually resident.

You may print and keep a copy of these Terms, which form the entire agreement between you and Mondaq and supersede any other communications or advertising in respect of the Service and/or the Website.

No delay in exercising or non-exercise by you and/or Mondaq of any of its rights under or in connection with these Terms shall operate as a waiver or release of each of your or Mondaq’s right. Rather, any such waiver or release must be specifically granted in writing signed by the party granting it.

If any part of these Terms is held unenforceable, that part shall be enforced to the maximum extent permissible so as to give effect to the intent of the parties, and the Terms shall continue in full force and effect.

Mondaq shall not incur any liability to you on account of any loss or damage resulting from any delay or failure to perform all or any part of these Terms if such delay or failure is caused, in whole or in part, by events, occurrences, or causes beyond the control of Mondaq. Such events, occurrences or causes will include, without limitation, acts of God, strikes, lockouts, server and network failure, riots, acts of war, earthquakes, fire and explosions.

By clicking Register you state you have read and agree to our Terms and Conditions