Does Tamiflu prevent complications of flu?

There was a very strange story surrounding yesterday's publication of a systematic review of the role of drugs such as Tamiflu in treating flu. This made the lead story on yesterday's Channel 4 News. The story involved the Cochrane Collaboration, the British Medical Journal, and Roche (makers of Tamiflu), and I have to say I don't think any of them has emerged from the story with much credit.

A previous Cochrane review had concluded that Tamiflu was effective in preventing the complications of flu, based on a meta-analysis of 10 studies that found a 59% reduction in hospitalisations. However, that meta-analysis, despite having been published in the prestigious Archives of Internal Medicine, contained a schoolboy error in its statistical analysis. This is surprising, given that Roche has some very smart statisticians and that one would hope that a journal such as the Archives would have some good peer-reviewers. However, it's not that surprising. Mistakes like that get through more often than we'd like to think.

So what was the error? This is a bit technical, so if you don't like stats, feel free to skip this paragraph. What they had done was add up all the hospitalisations in the Tamiflu and placebo groups in all 10 trials, and treat the totals as if they had come from a single trial. That is not a good thing to do, because the trials had different inclusion criteria and therefore different risks of hospitalisation, and not all trials had equal numbers of Tamiflu and placebo patients. The effect of the drug is therefore confounded by the type of trial. A correct way to do the analysis would either be by logistic regression, controlling for the trial, or by a meta-analysis of the results of all trials. But not by pooling the data and ignoring which trial they came from, which is what they did.

So, when the Cochrane reviewers came to update their meta-analysis, they realised that they needed to get more details on the 10 trials in the meta-analysis. Sadly, only 2 of them had been published. Now I don't want to be too hard on Roche for that: the trials completed about 10 years ago, and at the time it was quite common for many trials to remain unpublished. So the sensible thing for the Cochrane reviewers to do would be to ask Roche to supply the data.

However, rather than asking Roche directly for the data, they went to Channel 4 News, who then approached Roche to ask for the data. That seems pretty extraordinary to me. There has been no explanation of why they went down that route. As Roche said in their response on the BMJ website, this was "a move that questioned whether the motives for inquiries were truly for clarity and scientific validation". Hard to argue with that. I'd love to know why the Cochrane reviewers and the BMJ thought it appropriate to involve a TV company in a systematic review.

However, although Roche had a great opportunity at this point to occupy the moral high ground, they have, IMHO, squandered that opportunity by not making the data available in full. They were prepared to supply the data to the Cochrane reviewers if they signed a confidentiality agreement, but the reviewers were not prepared to sign such an agreement. This makes both sides look pretty bad to me. I don't see why Roche can't make the data available in full, and I don't see why the Cochrane reviewers should refuse to sign a confidentiality agreement. Roche obviously believed that there were some valid reasons to keep the data confidential. It's really hard to imagine what those reasons could be, let alone how they could trump the absolutely pressing PR reasons for making all the data available. It's also hard to imagine why the Cochrane reviewers felt unable to sign the agreement, even if they had every right to feel a bit miffed at being asked to do so.

Nonetheless, some data were supplied, and although it wasn't sufficiently detailed to answer all the Cochrane reviewers' questions, progress was being made, dialogue had been established, and it might be reasonable to think that ongoing dialogue would result in the necessary data being supplied before too long. However, the Cochrane reviewers were too impatient for this. They decided to go ahead and publish their review anyway.

This seems pretty extraordinary to me. The whole point of Cochrane reviews is that they are supposed to be systematic, in other words to include all the available data. To knowingly publish a review that excludes 8 relevant studies because they weren't willing to wait until they had got hold of the data seems extremely irresponsible.

They could have waited until they got the data.

Roche could have published the study reports, in full, on their website.

However, as it is, neither side did the things that they could easily have done to give us a reliable answer to the question of whether Tamiflu prevents complications of flu. So the rest of us still don't really know whether or not it does.

All in all, a bad day for science.