Tamiflu: it's a bit more complicated than that

Tamiflu is in the news again today. You will no doubt have read about how the government wasted vast sums of money on stockpiling Tamiflu, which today's new research shows is completely ineffective.

Well, that's what the press release said, anyway. And journalists are pretty good at regurgitating press releases. Here are some examples (though to be fair, they do at least all include a very brief mention that Roche disagrees, if you read that far). The reality, however, is a bit more complicated than that.

The Tamiflu story is one I've written about more than once before. It's a strange story. It's as much about politics as about science. For reasons which are not entirely clear to me, Tamiflu has become the poster child of a campaign by the Cochrane investigators and the BMJ for clinical trial results to be made more widely available. That campaign is of course entirely worthy: transparent disclosure of clinical trial results is a thoroughly good thing. However, by mixing up a campaign for greater clinical trial disclosure with a specific question about the efficacy of Tamiflu, the waters have become severely muddied.

In an ideal world, systematic reviewers assessing the efficacy of Tamiflu would simply look at the evidence for Tamiflu. But in this case, the review that's published today is partially a systematic review, and partially a campaigning tool. Very murky.

Because of the way this story has unfolded, the Cochrane reviewers have a serious conflict of interest. I don't mean a financial conflict: as far as I know, the reviewers have nothing financially to gain from downplaying the efficacy of Tamiflu. However, the reviewers have been involved in a protracted campaign against Roche, the makers of Tamiflu. Intellectually, it would be hard for them to turn round at this stage and say "actually, the data on Tamiflu are fine". It's also worth remembering that the lead author, Tom Jefferson, has been found collaborating with anti-vaccine campaigners, which suggests a certain hostility to the pharmaceutical industry.

There is some evidence of an anti-industry mindset within the Cochrane Collaboration. A review published in 2012 looked at the effect of industry sponsorship on clinical trial publications. It found that bias was less likely in industry sponsored research, but concluded that it was more likely. Granted, that review was by different authors to the Tamiflu review, but it does suggest that there may be a culture of hostility to the pharmaceutical industry within the Cochrane Collaboration.

And there is certainly evidence that the BMJ treat all this as a political campaign, rather than a scholarly and impartial search for the truth.

Another very strange thing about this story is that in the previous Cochrane review, published in 2012, the authors said that it was impossible to analyse data on many of the outcomes, because the study reports did not report data on the ITT population. Oddly, this wasn't explicitly stated in the review, but was explained when the Cochrane reviewers responded to my blogpost.

And yet the current review does include data on the ITT population. Where did it come from? And if it was in the study reports all along, why wasn't it analysed in the 2012 review? I did ask the Cochrane reviewers to share the study reports with me so that I could figure this out for myself, but they refused. This is also a little odd, when they had previously stated that they couldn't agree to receiving the reports under conditions of confidentiality because it was important that they should be able to share the data.

So what do the Cochrane reviewers claim they found, and what did they actually find?

They question the government's decision to stockpile Tamiflu, on the grounds that there is no evidence that it reduces admissions to hospital or complications of influenza.

Well, that's not strictly true. While the Cochrane review did not show a significant effect on admissions to hospital, it did show a substantial risk reduction (approximately a halving of the risk) of developing pneumonia (see analysis 1.17 on page 201 of the review). Pneumonia is a serious complication of flu, and it's odd that the Cochrane reviewers seem to have dismissed this as unimportant.

There is another problem with looking at the risk of complications. Complications are thankfully rare, so you need enormous randomised studies if you are going to detect an effect on complications. The Tamiflu studies, even when combined in a meta-analysis, lack that power. For example, the Cochrane review analysed data on just 70 hospitalisations.

So it's useful to supplement what we know from randomised trials with what we know from observational studies. Although there are obviously problems with observational studies, they can be very helpful when looking at rare outcomes such as death. Perhaps the most useful publication here is a systematic review of studies that looked at mortality and other complications, which found very clearly that Tamiflu (if given early) reduces mortality and other severe complications.

Now, obviously that's not as good as data from randomised trials would have been. Inferring causality from observational data is always tricky. But since we know from randomised trials that Tamiflu reduces the duration of flu symptoms (not by much, admittedly, but it's a statistically significant effect which clearly shows that the drug has some activity against the virus), and that it also reduced the risk of pneumonia, there is obvious biological plausibility in the hypothesis that the lower rate of complications in Tamiflu-treated patients is a direct result of that treatment.

You might argue that the refusal to acknowledge the contribution to the evidence base from non-randomised trials is merely a case of dogmatic methodolatry, though I do wonder if it is simply because the evidence doesn't fit the reviewers' agenda.

It is interesting to note in passing that the Cochrane review included only studies for which regulatory reports were available. This is rather odd: most systematic reviews include all available evidence, yet this review excluded several published randomised studies (mostly those done by academic researchers independently of Roche). This one, for example, which was not included in the review, found a significantly reduced risk of complications in the Tamiflu group. Would including it have changed the results of the meta-analysis, one wonders?

The spin from the Cochrane group also highlights the risk of psychiatric adverse events with Tamiflu. But did they really find an increased risk of psychiatric events? Not really. In their primary analysis, looking at treatment trials, the relative risk of psychiatric events was 0.93, which means that the risk of psychiatric adverse events was actually slightly lower in the Tamiflu patients (see analysis 1.32 on page 216 of the review). There was, however, an increased risk in prophylactic trials, though this was not statistically significant.

However, as all statisticians know, if you torture the data enough, it will confess. In this case, the reviewers managed to find a statistically significant difference by including events that happened in the trials when patients were not actually taking Tamiflu. It's hard to argue a good case for that based on biological plausibility, and yet this is the analysis that the reviewers focus on in their press release.

And again, the trials not included in the report might have change the results. This prophylactic trial, which was not included in the review, found a higher rate of psychiatric events in the placebo group than the Tamiflu group. When the analysis is based on such small numbers (even when including events that happened when patients weren't being treated, only 63 psychiatric events were included in the review), it wouldn't take many events in the other direction to nullify the statistically significant effect. Not that the significant effect should be taken seriously anyway when it's based on such obvious data-dredging.

Now, of course it's always prudent to be mindful of increased risk of adverse events, even if they are not confirmed statistically. I'm not saying non-significant increased risks should be ignored: it's important to be aware of them as a possibility. However, when the review's conclusions categorically state "The use of oseltamivir increases the risk of adverse effects, such as ... psychiatric effects", they are clearly going beyond the evidence. Perhaps they focus on psychiatric events because they sound scary.

It's clear that Tamiflu is far from being a wonder-drug. Its effects on influenza, while real, are undeniably modest. It's also true that Roche have hardly covered themselves in glory in the way they have been so slow to release the data on Tamiflu. But what also seems clear to me is that this latest Cochrane review has conclusions that do not simply follow from a scholarly review of the data, but have been coloured by political considerations.

This is not how systematic reviews are supposed to be.

So did the government waste money on stockpiling Tamiflu?

With hindsight, the answer is obviously yes, because there was no flu pandemic. Hindsight, of course, is a wonderful thing. The more important question is was their action reasonable at the time, given that when they made the decision a pandemic seemed a reasonable possibility.

In my mind, the government clearly made the right decision. Sure, Tamiflu was never going to completely nullify the effects of a flu pandemic. Its efficacy is modest. Some people would have taken Tamiflu and died anyway. But the evidence shows, to those who are prepared to look at it with an open mind, that it is more likely than not that it has a measurable effect on reducing the risk of severe complications of flu. You might ask how the government could have known that in 2009, given that the Tamiflu reports were not in the public domain at that time. Well, remember that the regulators, who are part of the machinery of government, did have access to them. And of course, they also had access to published papers, such as this one, which the Cochrane reviewers ignored.

But they spent half a billion pounds, I hear you cry! Isn't that an enormous sum of money? It may be to you or me, but in the context of government expenditure, it's a rounding error. It's roughly equivalent to what the government spends on paying the interest on the national debt every 4 days, or what it spends on pensions every single day.

It's worth considering what the consequences might have been if the government had not stockpiled Tamiflu and if there had been a serious flu pandemic. There would have been a great many deaths, some of which could have been prevented.

What do you think the headlines in the papers would have said in that case?