Validating the robustness of a trial

Poulin-Costello provided a detailed and very useful checklist of the key elements that statisticians look for when reading an article. A statistician would first verify the trial’s registration on the www.clinicaltrials.gov website and its publication in a reputable journal with high standards.

To validate the reliability of a trial, statisticians would then consider the following trial design questions:

• What was the sample size and what were the statistical analysis methods?
• Was the study randomized?
• Was it placebo controlled?
• Was it blinded--either single or double blinded versus an open label?
• How were the treatment groups defined?
• Was the choice of control logical and clinically meaningful?
• Was a safety/efficacy monitoring board employed?

P-Value and Statistical Power Explored

A notable mistake that people often make is to declare a greater efficacy of their product in a study with a p-value of 0.0000001 versus a p-value of 0.01. Both p-values confirm statistical significance. P-values do not provide a measure of the degree of efficacy. Poulin-Costello pointed out that every time a statistician calculates the p-value, there is always a chance of error, either a false negative or a false positive. As such statistical power helps researchers control for both of these types of error. "Power is the probability that one will get the right answer," she explained. The FDA requires a minimum of 80% power in a phase III trial. It is therefore vital to determine what power the authors were using to adequately deal with false negatives and false positives.

Dr. Sehdev, medical oncologist at William Osler Health Centre, identified methods for presenting the results of a clinical study as well as key elements a specialist may look for when assessing the value of a trial.

Sandy Sehdev, MD, William Osler Health Centre

In his opinion, the most popular sources of medical information for specialists include congresses, advisory boards, and local dinners. Although much of the data presented at congresses can be found online, Dr. Sehdev believes physically attending the congresses, whether international, national or local, to be so much more beneficial. He is an advocate of advisory boards, which he finds quite valuable as they allow for physician input, debate and discussion. Even local CHE events with a reputable and trustworthy KOL can be very beneficial and constructive for both academic and community based specialists.

Regarding the approach he takes for assessing medical articles, Dr. Sehdev acknowledges that he reads the abstract first. If it is of interest to him, he reads the conclusion then the discussion from which he will interpret the relevance of the results for his own practice. With respect to the statistical analysis, he relies on the journal editorialists to comment on the statistical power, interpret the data and confirm the validity of the results.

The following is a mini checklist of questions Dr. Sehdev uses to confirm the robustness of a study and the value it brings to him.

• Who wrote the article and was it published in a reputable journal?
• What was the objective of the study? Was the objective relevant to his practice as well as interesting and possibly controversial?
• Where was the study carried out? Was it a multi-centre study or were the patients recruited from only one country in which the population does not match his own patients’ population.
• Why was the study conducted? Was the goal to answer scientific questions for future research or to change current practice?
• How was the study designed? Was it statistically sound?

As a note of caution, he warned to never over-reach the conclusions of a study. It will cause distrust and possibly damage what may have been a good relationship with physicians. On the flip side, a trusting relationship can develop when industry approaches specialists in a balanced manner with information that is relevant to the physician’s practice.

Marcia Bos, Pharmacist-Consultant for The Synapse Group, opened her presentation by pointing out that the appeal for scientific evidence dates as far back as the 11^th century, when a 14-volume medical encyclopaedia listing the efficacy of various products was inscribed.

Marcia Bos, The Synapse Group

There is an enormous lag time from the period of trial design to publication. It is not uncommon for a study to take 6-8 years from the time of protocol design, through recruitment, to its final journal publication.

As so much time lapses between the start and end of this process, it is often difficult to remember the reason why certain decisions were made at the protocol development stage. Bos recommends researchers draft a companion report to document why key decisions were made at the time of trial design. Armed with this information, they will be better prepared to confront any challenges, and equipped to provide clear explanations of these decisions when asked at time of publication.

This huge lag time between trial design and journal publication presents further challenges to researchers. It is quite likely that many environmental elements will have changed throughout the course of trial progression: practice guidelines may be updated, new information from post marketing surveillance with additional safety data may be available, new agents may have come on the market, new technologies for diagnosing or monitoring disease may be available, or new biomarkers could have been brought to market for diagnosis and prognosis.

It is quite possible that at the time of publication the results are not as relevant as they may have been years earlier. A strong communication plan needs to be initiated while the study is still ongoing, to manage the challenges that may arise and to tailor communication materials accordingly.

A cross-functional approach to the communication plan is recommended to fully understand the therapeutic area and how current practice behaviour may have changed over the course of the trial. Key opinion leaders should be engaged in the scenario planning process to help interpret the results and the brand implications, considering positive, negative and neutral outcomes.

In closing, Bos emphasizes how integral clinical studies are to the development of a brand. Results from trials could lead to updates in product monographs or modifications for product labelling, and may generate new messages for promotional materials. Fundamental planning up front will in-turn lead to effective communication of the results.

About the Author:
Marla Weingarten is a consultant at Pangaea Group (www.pangaea-consultants.com), a professional services firm that has created a network of experts in Canadian pharma. Marla can be reached at mweingarten@pangaea-consultants.com