> I have just watched a video of a fascinating presentation by
> Prof. John Ioannidis at NIH entitled: "Great Teachers:
> Translation, Replication and Credibility of Research Findings".
> It is available on the web at:
> http://videocast.nih.gov/PastEventDetail.asp?13691
> I think this is available to all, but I'm not certain because I'm
> watching it from inside the NIH firewall.
> Ioannidis has done a series of studies of medical publishing
> extending over quite a few years. His method is to do an
> extensive search of the literature on a specific topic, identify
> all studies on that topic, isolate the findings of each study,
> and perform statistical analysis to find out how often and with
> what degree of correlation initial results are confirmed by
> follow on studies.
> I'm not a statistician but I hope I'm presenting a reasonable
> precis of what he said.
> What he said was that the vast majority of "statistically
> significant" results reported in the scientific literature (89%
> if I remember correctly) turn out not to be statistically
> significant at all after later studies are done to attempt to
> replicate the results. He believed that the most common source
> of "significance" in results turns out to be bias of one of
> several kinds. Typical sources of bias are:
> Sample size bias:
> Many studies are reporting effects that are small enough
> that the sample sizes are insufficient to justify the
> reported effects.
> Selection bias:
> The authors analyze and report findings selectively,
> leaving out data that they regarded as "insignificant",
> but which in fact often is significant.
> Publication bias:
> Journals look for articles that show highly statistically
> significant results, therefore biasing publishing in the
> direction of those articles that contain lots of outliers
> in their findings.
> Fraud:
> He said that there were reported cases of this and seemed
> to think that more of the bias that authors introduce is
> not as inadvertent as we would hope that it is.
> One thing that I found particularly interesting in his
> presentation was that he derived numerical values for bias. For
> example, in one specific field, he compared early reports with
> a huge, well conducted study of thousands of patients, and
> found that there was a built in bias of 1.3 in the results of the
> early reports. If I understand that correctly, he is saying
> that, in that field, unless a particular effect is at least 30%
> greater than the controls, it is indistinguishable from the
> built-in bias in that field.
> When a very bright Chinese student came to work in his lab, he
> did a parallel study on Chinese publishing in the same field and
> found the bias there was 3.0 (compared to 1.3 in European and
> American journals.) In that field, the Americans and Europeans
> were doing a bad job, but the Chinese were atrocious.
> What all this means to me is that I'm going to add a lot more
> grains of salt to all of the latest research findings that we see
> reported daily in the scientific press.
> Alan
|
