t is an emotional cycle familiar to most AIDS-vaccine researchers: the high of finally making measurable headway against HIV, followed by the crushing low of discovering that the virus has once again found a way to elude them.
It happened again on Saturday when researchers learned that the first ever successful AIDS vaccine turned out not to be the triumph they had originally hoped. In September, scientists from the National Institutes of Health (NIH) and the U.S. Army announced the results of an AIDS-vaccine study in Thailand involving more than 16,000 volunteers. The data showed that the new vaccine had protected 31% of inoculated participants from becoming infected with HIV. But a closer look at a subset of the study's volunteers now reveals that the vaccine in fact protected only 26% of the people who received it. (See pictures of Africa's AIDS crisis.)
The difference is small but critical because the new success rate of 26% falls below the threshold for statistical significance. That means that the odds of being protected from infection by the AIDS vaccine may be no better than chance.
At issue is a matter of head count. If the entire group of volunteers who were enrolled in the study were included in the data, then the results would suggest a 31% effectiveness rate, with 51 in the vaccine arm and 74 in the control group becoming infected with HIV. These are the results that were announced in September. But because this particular vaccine was given in six doses over a six-month period — in what is referred to as a prime and boost regimen, in which the early shots prime the immune system to fend off HIV and the follow-up shots boost the body's immunity — some volunteers became infected with HIV before receiving all six shots, rendering them ineligible to complete the study. Since researchers are aiming to study how well the complete set of inoculations protects against infection, the final data should exclude the one-third of participants who became infected before the study concluded. (Watch an audio slideshow about aging AIDS patients.)
The resulting tally, then, includes a much smaller pool of participants, which automatically lessens the significance of whatever effect, if any, the vaccine appeared to have, says Dr. Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases. "When you lose statistical power, something that would have been significant in the [original, larger] population, could now fall below significance merely by the lower numbers. That's what happened with this trial," Fauci says.
According to statisticians, however, scientists generally include all data from the complete, original population, since these numbers more accurately reflect what might happen in the real world. In other words, it's a more rigorous analysis of how effective a vaccine might be in a population of people who, realistically, could be exposed to HIV before they finish the full six doses of a vaccine.
As for why investigators did not reveal both sets of data in their initial announcement, the Army researchers posted this update on the website of the U.S. Military HIV Research Program: "Explaining the differences between them is complex and the appropriate venue for this technical discussion of statistics is at an open scientific conference and in the scientific publication now under review at a major journal." (See the most common hospital mishaps.)
So does this mean the vaccine was not effective? That depends on whom you ask. Some experts argue that the vaccine's effect, if it exists, is so tiny that it's not worth pursuing in a significant way. "Would I invest in it? The answer is no," says Dr. David Ho, director of the Aaron Diamond AIDS Research Center in New York City. "There are other things that are more likely to work, that are better, easier and more straightforward than this vaccine."
(Submitted by Anne Aldrich)
Subscribe to:
Post Comments (Atom)
As a statistician, I believe that it is incredibly important to recognize the importance of the "effect size" of the vaccine as opposed to the significance of the test. Too often, researchers and statisticians focus so greatly on the significance of a test that they forget to acknowledge the effect size of the treatment itself. For example, if you increase the sample size (let's denote it n) greatly, there will always be a statistical significance that would be detected. Let's say we take all the students at BU and all the students at BC and try to determine if there is a difference in the heights. We can increase the sample size from each population so that a significance is detected through the use of a two sample z-test. As sample size increases, p-value decreases which ensures that there will be a difference; however, what if we found that the effect size, the point value of difference, was 0.1 cm? Do we really care? The more people that we sample, the more pronounced small differences become. This links back to the experts argument that the vaccine's effect, if it exists, is to tiny that it's not worth pursuing in a significant way.
ReplyDeleteIn this frame of mind, the dismay caused by the 31% effectiveness rate to a 26% effectiveness rate is well deserved. At a constant number of people, significance is an important aspect of statistical analysis. While adding individuals to reduce Standard Error and increase the chance to show a difference is generally acceptable; but in this case a statistical fallacy. Although this indicates empirically that there is no significance while there might be [a Type II error], this would simply do a weak test [small 1 - beta "beta is chance for Type II error]. Most likely, the test that was preformed on the people was a two sample-proportion z-test.
Overall, from a statistical standpoint, the vaccine should not be pursued for the effect size on the individuals was negligible. It would be immoral to do so, as there is much more good that can be accomplished with the resources. Furthermore, this would not violate Kant's deontological ideals, as there is not basis for the vaccine being able to save lives. From both a utilitarian and a deontilogical standpoint, continuing on this particular vaccine would be meaningless. As stated by Dr. Ho, "There are other things that are more likely to work, that are better, easier and more straightforward than this vaccine."
The issues with including all of the relevant data in a medical study that tests new vaccines very complicated and there are many of issues that occur. In the case of HIV, there are many drug companies and researchers that are interested in developing both better treatment options for people that are already infected with HIV and vaccines that would prevent people from contracting the virus at all. There have been several recent breakthroughs in terms of possible vaccines, including the one mentioned above.
ReplyDeleteThere are several reasons for a desire to exclude those that contracted the virus during the course of the treatment. One is that they may not be statistically significant. The only way to determine whether or not that is true would be to make the sample size larger and find out if the participants in question are truly significant on a large scale. Another reason to explain why the researchers might have excluded the participants is if they admitted to knowingly exposing themselves to the virus, but the problem with that line of thinking is that in a real life situation, the people that need to be vaccinated the most would be those that are being exposed to the virus in daily life. The data seems to show that a different vaccination method would be more effective. The problem with the study that is spoken about in the beginning of this thread is that there is no definitive way of knowing if the participants contracted the virus from someone they encountered in daily life or if the virus was contracted because of some error in the attempt to attenuate the virus in order to make the vaccine. For this reason there can be no results left out of the final release of the study.
It is unethical to have included the subjects that were infected with the virus before completing the vaccination regimen in the statistical analysis of the study. It is an established ethical norm among researchers to tell the the truth, i.e., not to falsify or misresprent the data resulting from research. As the article highlights, the statistical analysis of the study's data is not immaterial, but rather critical to determining what, if any, significance the results have; in this case, whether a AIDS vaccine is viable for widespread prevention or not. Manipulation of the statistics, then, when they result in a wholly different conclusion surely constitutes a gross misrepresentation of the data and is thus unethical. Notably, the article presents a defense that the subjects of note merely represent a undeniable reality in their early contraction of the virus prior to complete vaccination, and that their reflection of the real world merits their inclusion of the study. However, I agree with Richard that it is not clear whether their infection was the result of their normal daily lives or, rather, was in some way a consequence of their newly adopted vaccine regimen. Their infections and the ambiguity of their origins removes these subjects both from the context of the real world and from the confines of the study, and thus their inclusion was still misrepresentative and an unethical decision. Furthermore, in this case, a potentially dangerous one.
ReplyDeleteNot only does this study's questionable statistics constitute misrepresentation and a violation of ethical research norms. It appears further unethical in its possible antipathy toward society's health as a whole. The inclusion of the aberrant subjects allowed the previously statistically insignificant results to change and thus lifted a vaccine that was otherwise deemed only as effective as chance to a level of providing significant protection from the virus. If this vaccine were marketed, people at risk for infection, having taken the vaccine, might live their lives under the assumption that they are protected, when in reality they are not, and could consequentely contract and spread the virus unbeknownst to anyone. This possiblility is in direct opposition to many ethical standards, including to do no harm and to promote maximum well-being to a maximum number of people. Allowing the original statistical interpretation of the data to hold with the infected subjects included would have increased the likelihood, albeit to a unknown degree, of this hypothetical promotion of infection, a certainly unethical act. These possible consequences of fudging the statistics should have discouraged the inclusion of the subjects.
While the data regarding these subjects may in fact be noteworthy in another research capacity and should, in the end, be reported incidentally, they cannot ethically be included in the statistical interpretation of the effectiveness of the AIDS vaccine.
-Patrick Duggan
Regardless of including all of the participants or not, this vaccine is still one of the first and only actions that has been taken to show this great of a success in keeping people from being infected with HIV. I think it shows great promise and this is one of the first of many great leaps in preventing and curing HIV. To hear that any medical person would not be excited or willing to put money into research that is so new and hopeful. Beyond that I think that it was irresponsible of the researchers to not publish the results of the complete study. Even if not in the original they should have published an addendum or additional report. No matter whether the results were statistically significant, I think that people who live in environments that have high susceptibility to becoming infected with HIV would jump at the thought of having a vaccine that is even 26% effective if not 31%. I also am astounded that this report got as far as it did with out this being mentioned. How did so many outlets of media get a hold of this story without the full information of the report. I think blame lies much farther beyond just the people who conducted the study. Beyond the clear disregard of standards of medical ethics of not publishing the full results. Beyond all of the statistics more and more attention should be brought to the study. With this small glimpse into further understanding a disease that affects so many people around the world, researchers everywhere should take this information and pour time effort knowledge and money into this new hope. I am shocked by the amount of skepticism and negative attention towards this research. I feel like the scientific world should be jumping for joy at this research. Hopefully people will take this information and do wonderful things to fight this awful virus.
ReplyDeleteAlthough it appears we are still a great distance from curing and preventing the spread of Aids, I look at this new research optimistically. Statistically this data may be considered irrelevant, but it could correlate with a modest benefit. It’s the first evidence that we could have a safe and effective preventive vaccine. Even though this study does not show fantastic results, we do not need a vaccine that will help one hundred percent of all those involved to begin with, since even a marginally helpful vaccine could have a big impact. UNAIDS estimates that every day, 7,500 people worldwide are newly infected with HIV; 2 million died of AIDS in 2007. Therefore, even a small percentage of those lives could be saved with this vaccine, if further studies can prove that the success rate is related to the vaccine and not an outcome based purely on chance, even though the success rate is only around thirty percent. This study is also beneficial in the sense that it is not the end of the road, but a solid foundation for further studies and analysis. In addition, the possibility of improving this result and developing a more effective AIDS vaccine, is something that can be accomplished from utilizing the data from this study. I do not believe we are close to solving the huge global problem of AIDs, but I do believe that researchers are heading in the right direction and will learn a great deal from this study and will hopefully be able to generate a vaccine with much greater effectiveness and perhaps one day cure and prevent the spread of AIDs throughout the world.
ReplyDelete-KEENAN