It started with shock and disappointment. A damning critique of the 2018 Cochrane review of the HPV vaccine was published in BMJ Evidence-Based Medicine, and a news report in the BMJ with this alarming headline: HPV vaccine safety: Cochrane launches urgent investigation into review after criticisms. How could Cochrane have gotten such an important review so catastrophically wrong?
Well, turns out they didn't. Once I dug into it, I decided it was a manufactured controversy - a hatchet job by people with such strong intellectual and ideological biases, that they didn't even pick up that some of their most implausible allegations were based on their own analytical and investigative slip-ups. You can see how I came to that conclusion in this "forensic" post:
The HPV Vaccine: A Critique of a Critique of a Meta-Analysis.
The conversation that followed that post made me realize there were so many misconceptions out there about when we would know if the vaccine was working out the way the trials predicted - and so many blanks in my own knowledge on these things. So I went digging again, coming up very optimistic about the potential for the vaccine, with a second post:
The HPV Vaccine Should be Preventing Cervical Cancer: Can We Tell Whether It Actually Is?
That post tackles the basics you need if the debate turns to the argument - "But it's never been shown to prevent cervical cancer!"
Meanwhile, there's that saga with the Cochrane review and Cochrane's investigation of the BMJ EBM critique. So I started this post to keep the threads together.
Cochrane's response to the critique was published on 3 September (UK time). As my post predicted, there was very little efficacy data - only 5 eligible trials, with only 1 including the primary outcome (CIN2+). Incorporating the data wouldn't change the main conclusion. I added a coda to my original post.
Previously on 3 September, I tweeted to the BMJ EBM, noting serious error in what they published and asking if they would correct, and on the 4th, to its editor-in-chief. (There was no reply.)
I also submitted a comment on the Cochrane review. (In case that's behind a paywall, a copy is below this post.)
I'll also be discussing and analyzing this more as this situation evolves. But in the meantime, a quick summary of its escalation since I posted this (on 4 September):
- On 5 September, the Nordic Cochrane Centre tweeted that the Cochrane response "does not address our most serious concerns", and they would be responding.
- On 12 September, the editors of BMJ EBM, publishers of the Copenhagen critique, published a response.
- On 15 September, a link to a statement circulated on Twitter - it's an open letter dated 14 September from Peter Gøtzsche on Nordic Cochrane Centre letterhead. Gøtzsche is one of the authors of what I've been calling the critique. He reports that he was expelled from the Cochrane Collaboration and that he and 4 others resigned from the organization's board in protest. It's a declaration of civil war, weaving a wide range of internal Cochrane political issues and controversies into a particular narrative, and calling for "dissolution of the present board, new elections and a broad-based participatory debate about the future strategy and governance of the organization".
- A statement from Cochrane's co-chairs dated 15 September was posted on the organization's website. It specifies what changes occurred to the board, and reported that the situation with Gøtzsche is "ongoing".
- The 4 board members who resigned issued a statement of their own on 15 September too.
That's a lot to unpack, and we'll get to it. Cochrane's annual international conference (called the Colloquium) starts in Edinburgh on 16 September - and that's no coincidence. There's a lead-up to the Colloquium of Cochrane sector and governance meetings, and it is a traditional time for factional disputes and lobbying, with the occasional major schism. The organization is a registered charity - here's its constitution/articles of association.
At the AGM on 17 September local time, the Cochrane board issued this detailed statement. And around the same time, BMJ EBM published the Copenhagen group's response to the Cochrane editors' response. Here's my take on developments:
I'm not going to try to keep on top of all the posting and documents, but I will post some.
- On 19 September, one of the Nordic Cochrane Centre co-authors of the Copenhagen critique posted his take.
- The Cochrane board's statement had included a statement criticizing the statement made by the resigning board members. On the 18th, the Canadian Cochrane Center director (who was not a board member), Holger Schünemann, called on Twitter for them to delete that criticism. In my blog post on the 18th, I had singled this out as an example of the type of common situation addressed by Sandman in his book on crises and outrage management: it was an unnecessary defensive and provocative action, and taking Schünemann's advice was an opportunity to demonstrate responsiveness. On 19 September, the Cochrane Collaboration removed the sentence.
- On 20 September Europe/America time (I saw it on the 21st in Australian time), someone messaged me that the link in my blog post to the resigning board member's statement was broken. I have asked if the statement has been withdrawn. If it has been as a similarly conciliatory gesture, that would be heartening.
- On 20 September, the BMJ published a report on Gøtzsche saying the board's process was "flawed" and came after a "show trial". They published selected excerpts Gøtzsche gave them of the lawyer's report to the board, and discussed events from some board meetings. I re-read the Cochrane Board's statement (original and revised) and the resigning board's statement carefully. I can see nothing in them that is contradicted by Gøtzsche's statement. For example, he cites a part with advice about historical incidents. The board statement, however, referred to the lawyer's finding in the current situation, where Gøtzsche made a serious allegation about a senior member of Cochrane's staff.
Gøtzsche was still both a member of the Cochrane Collaboration and its board at the time he publicly released his statement and gave this material and these anecdotes to the BMJ for publication. Whether or not you think he damaged the Cochrane Collaboration's reputation before all this, I think few would disagree that he is doing that now.
- Sometime before the 23rd, the Austrian Cochrane Centre removed the statement of the resigning board members to further aid de-escalation. (I confirmed this with Gerald Gartlehner, who is both director of the Austrian Cochrane Centre and one of the resigning board members.) Great to see such a conciliatory move.
On the 24th, I followed up on some issues being debated, in a much shorter post at Absolutely Maybe:
Scientific Advocacy and Biases of the Ideological and Industry Kinds
- Also on the 24th, the authors of the Copenhagen critique (Lars Jørgensen, Peter Gøtzsche, and Tom Jefferson) were joined by Peter Doshi, in an editorial piece in the BMJ titled, "Challenges of independent assessment of potential harms of HPV vaccine".
- On 26 September, the Cochrane board announced that it had "voted unanimously on 25th September to terminate Professor Peter Gøtzsche’s membership of the organization, and his present position as a Member of the Governing Board and Director of the Nordic Cochrane Centre". Gøtzsche annotated it with his responses, and posted that on his website.
- Cochrane posted a video recording of the Annual General Meeting in Edinburgh. The section on this issue begins at 36 minutes in.
- On 28 September, Gerd Antes, the director of the German Cochrane Centre issued a statement (original German, here in machine-translated English).
- STAT News published a report by Ivan Oransky and Adam Marcus, including comments from a board co-chair and Gøtzsche. This links to Gøtzsche's personal website, where he has been posting confidential documents, including the lawyer's report to the board. (I'm not posting direct links, because these materials also contain information for people inside and outside Cochrane with a right to privacy, in my opinion.)
- On 29 September, Gøtzsche tweeted that the Nordic Cochrane Centre (NCC) had withdrawn from the Cochrane Collaboration: his website stated "I withdrew the Nordic Cochrane Centre from the Cochrane Collaboration". He expressed concern about the Cochrane Collaboration having made changes to the NCC website. However, Gøtzsche is no longer director of "the NCC", and although his statement shows he regards his research center as independent of the Collaboration, it is unclear what impact this severance may have for its funders.
Some background information: Cochrane Centre status is conferred by the Cochrane Collaboration, and a Cochrane Centre can transfer location and leadership. The Nordic Cochrane Centre serves several countries, any one of which could theoretically become the Nordic Cochrane Centre if Denmark were no longer interested in doing so. The Cochrane Collaboration name and logo are registered trademarks in the EU. The research centre Gøtzsche directs will presumably need to change its name. He asserts its independence in a statement dated 26 September, and has protested the Collaboration making changes to the website. However, that website is an offshoot of Cochrane.org (nordic.cochrane.org).
I could find no financial disclosures on the NCC website when I searched at the time of writing my "Boilover" blog post, or now. Their annual reports disclose funders' names only. The Cochrane Collaboration was no longer listed as a funder for the last few years, after having contributed significant funding in some years at least for NCC information management work and leadership for the organization internationally.
According to Cochrane's current financial disclosures, the charity, the Cochrane Collaboration, "owns three subsidiaries, each with its own board of directors", one of which is Cochrane IKMD Denmark ApS, "a Danish company set up to support the work of Cochrane's Informatics & Knowledge Management Department based in Denmark". Executive leadership for the IKMD transferred to Germany in January 2017. Danish employees are listed in that team.
The NCC's main direct funder is the Danish Ministry of Health. According to a report in Danish media, the Ministry informed the NCC that their funding was under review in August, for the first time in 25 years. The report says the government budget for 2019 allocates 18 million krone (over US $2.8 million) to the NCC. A symposium to celebrate the 25th anniversary of the NCC was scheduled for 12 October. The NCC is, administratively, at the Rigshospitalet, a teaching hospital where the health faculty of the University of Copenhagen is located.
- On 29 September, The Lancet published a report, including interviews with Gøtzsche and Gerald Gartlehner, director of the Austrian Cochrane Centre and one of the Cochrane board members who resigned: "He stressed that his resignation was entirely related to the process. 'This is not about defending behaviour or endorsing Peter's scientific opinions', said Gartlehner".
Hilda Bastian
Originally posted on 4 September 2018. Last updated 30 September 2018.
(For background: I was "first generation Cochrane Collaboration", part of the group Iain Chalmers assembled to found an international organization out of the roots of the Cochrane Centre he had inspired into life recently in Oxford. I served on the board of the international organization from its founding in 1993 to 2001. I'm no longer a member of the organization. A disclosure of interests at the foot of each of my posts on this matter at Absolutely Maybe.)
UPDATES
3 updates on 16 September 2018 (the first in Australian time, 15 September European time)
17 September 2018: Deleted a paragraph on 2 comments awaiting moderation at the Absolutely Maybe blog, after I released the comments and replied to them.
2 updates on 19 September 2018 (Australian time): Added link to new blog post, then a link to Tom Jefferson's post.
On 21 September, added bullet points for events from 19 to September.
On 22 September, added the update about the removal of the resigning board members' statement.
On 25 September, added link to my new blog post, and to BMJ editorial on harms, both from the 24th.
On 26 September, added link to the Cochrane board's announcement.
On 27 September, added link to the video recording of the Cochrane AGM.
On 29 September, added bullet points for events of the 28th.
On 30 September, added information about NCC status and Lancet report. Added Gøtzsche's response to the Cochrane board's statement terminating his membership to the bullet point for 26 September. (I hadn't seen it on the 26th.) Edited section on the NCC's status to reflect the content of that statement.
Text of my comment at Cochrane:
Support for the rigor of this review; concern about flaws and bias in recent critiques
This review has attracted a critical comment here by Riva and colleagues [1], and another in BMJ Evidence-Based Medicine by Jørgensen and colleagues [2]. Neither establish that there are significant flaws in the Cochrane HPV vaccine review by Arbyn and colleagues; both critiques have fundamental flaws, and especially in the case of Jørgensen et al, significant bias. I have posted a detailed critique of the Jørgensen piece at my blog [3].
The Riva comment calls on Cochrane to change the review’s title by deleting reference to cervical cancer, because the trials were designed to detect changes in precursors, but not cervical cancer: “No RCTs ever assessed the impact of HPV vaccines on cervical cancer incidence and its associated mortality”.
Firstly, Cochrane reviews should address key questions, whether or not there are trials that can provide data on them. Secondly, cervical cancer was an outcome in trials, even though they were not powered, or initially long enough, to detect a difference. And thirdly, long-term follow-up studies for trials may indeed provide useful data on cervical cancer. For example, such a study is perhaps the first to find a statistically significant difference in HPV-related cancer [4].
Riva go on to say that any conclusion based on cancer lesions is “purely speculative”. Moderate to severe cervical lesions are a well-founded surrogate for cervical cancer. For example, a modeling study in the UK found that the incidence of invasive cervical cancer after CIN2/CIN3 was up to 15.4% [5].
Observational studies in vaccinated communities are showing similar reductions to that in the Cochrane review – see for example the systematic review by Garland and colleagues [6] – and for countries with extensive national vaccination and screening programs, such as Australia and the UK, data on whether the drop in severe lesions in successive cohorts are followed by concomitant drops in cancer and associated mortality will be available in the next few years (see my second related post [7]).
Indeed, despite concerns expressed by Riva and others that the cervical cancer rate might not budge because other HPV strains cause further disease, in some ways the reverse might be happening: the drop may even be higher because of herd immunity in communities with high vaccination levels, particularly as vaccination is no longer limited to girls and vaccines against a broader group of HPV strains are now available.
Both the Riva and Jørgensen comments take issue with the Arbyn et al’s assessment of risk of bias. Those judgments are very subjective, and disagreement on this score isn’t really surprising. What is striking to me, though, is the amount of bias about the HPV vaccine evident in the comments about, as Riva for example say, “the supposed benefits of these vaccines”. Both groups criticize the review because of the conflicts of interests of the authors of the first version of this review’s protocol – despite the review reporting that Cochrane’s action in this regard, including a new protocol after the exit of conflicted reviewers. A bigger issue to me appears to be the intellectual and ideological conflicts of these critics.
SOURCES:
[1] Riva C, Tinari S, Spinosa J-P (2018). Concerns over methodological flaws in the assessment of vaccine efficacy. Feedback section for Arbyn et al (2018) doi: doi/10.1002/14651858.CD009069.pub3 https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD009069.pub3/detailed-comment/en?messageId=154255807
[2] Jørgensen L, Gøtzsche PC, Jefferson T (2018). The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias. BMJ Evidence-Based Medicine Published Online First: 27 July 2018. doi:10.1136/bmjebm-2018-111012 https://ebm.bmj.com/content/early/2018/07/27/bmjebm-2018-111012
[3] Bastian H (2018). The HPV vaccine: a critique of a critique of a meta-analysis. Absolutely Maybe at PLOS Blogs: https://blogs.plos.org/absolutely-maybe/2018/08/25/the-hpv-vaccine-a-critique-of-a-critique-of-a-meta-analysis/
[4] Luostarinen T et al (2018). Vaccination protects against invasive HPV-associated cancers. International Journal of Cancer 142(10): 2186-2187. https://onlinelibrary.wiley.com/doi/abs/10.1002/ijc.31231 Registered at ClinicalTrials.gov in 2011, NCT01393470.
[5] Raffle AE et al (2003). Outcomes of screening to prevent cancer: analysis of cumulative incidence of cervical abnormalities and modelling of cases and deaths prevented. BMJ 326:901 https://www.bmj.com/content/326/7395/901.short
[6] Garland SM et al (2016). Impact and effectiveness of the quadrivalent Human Papillomavirus vaccine: a systematic review of 10 years of real-world experience. Clinical Infectious Diseases 63(4): 519-527. https://academic.oup.com/cid/article/63/4/519/2566619
[7] Bastian H (2018). The HPV vaccine should be preventing cervical cancer: can we tell whether it actually is? Absolutely Maybe at PLOS Blogs https://blogs.plos.org/absolutely-maybe/2018/08/31/the-hpv-vaccine-should-be-preventing-cervical-cancer-can-we-tell-whether-it-actually-is/