In 2012 a cost-effectiveness analysis of the PACE trial was published by PLOS ONE. One of the conditions of publication was an agreement by the authors ‘to make freely available any materials and data described in their publication that may be reasonably requested for the purpose of academic, non-commercial research’.

A number of requests have been made to the authors for the patient-level data. These requests have been rejected.

PLOS ONE has issued an Expression of Concern, but has not obliged the authors to make the data available. In response, the authors state they are ‘surprised by and question the decision by the journal to issue this Expression of Concern’. They continue to reject all requests for patient-level data. In their Statement, they make it clear why:

‘During negotiations with the journal over these matters, we have sought further guidance from the PACE trial R[esearch] E[thics] C[ommittee]. They have advised that public release, even of anonymised data, is not appropriate.’

The PACE trial REC is a public body and its advice to the authors was a public document covered by the FOIA, so I made a request asking for a copy (available here).

The REC does indeed advise against public release of the data, though for what many would consider a spurious reason and one beyond its remit, namely because of the controversy surrounding the illness (‘a contentious matter upon which there are divergent views’).

However, the REC also made clear that the data should be shared with an independent researcher or institution who/which would then ‘receive the anonymised data and produce a report’. In its response to my request the REC’s summarized its position as follows (email available here):

‘…from a transparency perspective the sharing of data would be the HRA’s normal recommended course of action. However as suggested in the attached email by the REC Chair, and from our discussions with other parties involved we understand the data may be made accessible via a secure site to other independent researchers wishing to analyse the data.’

The REC’s view then is that there should not be public release of the data but that the data should be shared with independent researchers.

So why does PLOS ONE not enforce its conditions of publication and oblige the authors to provide the data to the researchers who have requested them or retract the paper?

Is PLOS ONE saying the researchers who have requested the data, such as Professor Coyne (see also here and here), are not ‘independent’?

If those researchers are not deemed sufficiently ‘independent’, then why doesn’t PLOS ONE nominate an independent scientist and instruct the authors to comply with the regulations and the wishes of the REC and provide that scientist with the patient-level data or retract the paper?

How does this refusal by PLOS ONE meet the claim by its editor to be committed to ‘full transparency‘?

Where does PLOS ONE’s failure to abide by its regulations leave all the academics who work for it as unpaid peer-reviewers, bloggers and editors because they support open science?

And why do the PACE authors continue to refuse access to the patient-level data even when the REC says they should provide it to independent researchers?

 

The PACE trial and why the data need to be shared

PACE was the ‘definitive’ trial of two interventions for myalgic encephalomyelitis (ME), sometimes known as chronic fatigue syndrome (CFS): cognitive behavioural therapy (CBT) and graded exercise therapy (GET). The main paper was heavily promoted in the media as providing evidence that CBT and GET could help patients ‘get back to normal’.

A number of criticisms have been made about the design of the trial, in particular: First, the trial used a very broad criteria for diagnosis of the illness which could include patients with general fatigue and/or mental health problems. Such patients are known to respond to the interventions. Second, the investigators failed to guard against the powerful distorting effect of researcher bias. Third, there was a heavy reliance on self-report measures in an unblinded trial (where both patients and therapists knew who was and who was not receiving the interventions). It has long been known that the use of subjective measures in an ‘open-label trial’ gives false results.

There has also been considerable criticism of the decision, which was taken after the trial had been completed, to deviate from the protocol and make a major change in the thresholds in the outcome measures. The investigators altered how they measured whether an intervention was effective or not. According to the measures they eventually used, a participant could actually get worse in the trial — their score could go down during the period when they were receiving treatment — and yet they could still count as evidence for the effectiveness of the treatment. 13% of patients on this new scoring were already ‘recovered’ on a measure before they had even entered the trial. In other words, they were both ill enough to qualify to participate in the trial and ‘recovered’.

The investigators, the responsible authorities and the main funder, the MRC, have always said they were prepared to provide access to the data to ‘bona fide researchers’. Patient-level data were shared with a few researchers, who are known to support the interventions, for a Cochrane systematic review (though note criticism of Cochrane reviews of interventins for this illness here, here and here, comments by among others Bob Courtney and Tom Kindlon, and reanalysis by Mark Vink and Alexandra Vink-Niese).

When researchers, patient organizations and patients have asked for the data in order to conduct independent analyses using the measures set out in the protocol, these requests have been rejected. These rejections led the First Tier Tribunal (FTT) in Matthees to express concern that the researchers may have been cherry-picking who analysed their data to only sympathetic researchers in an attempt to suppress criticism and proper scrutiny of their trial.

An open letter signed by many independent statisticians and scientists, along with MPs, lawyers and patient organizations, has been sent to The Lancet calling for the release of the data for independent analysis. There has been no response.

After a Freedom Of Information Act (FOIA) request by Alem Matthees, the FTT did order the release of some data. In the unanimous part of its ruling, the FTT was clear: there is no legal or ethical consideration which prevents release of the trial data; there is a strong public interest in release; making data available advances legitimate scientific debate; and where possible data should be released.

These data were analysed using the outcomes specified by the investigators in the trial protocol. It was found:

‘When recovery was defined according to the original protocol, recovery rates in the GET and CBT groups were low and not significantly higher than in the control group.‘

And:

‘These findings raise serious concerns about the robustness of the claims made about the efficacy of CBT and GET. The modest treatment effects obtained on self-report measures in the PACE trial do not exceed what could be reasonably accounted for by participant reporting biases.’

Since the decision in Matthees, the responsible authority, QMUL, has announced a trial data-sharing policy but has rejected a number of FOI requests for access to more data and continues to deny access to anyone critical of the trial, including highly qualified academics.