A. Nalin1, A. Grippin1, R. Kouzy2, J. A. Jaoude3, A. M. Miller4, and E. B. Ludmir5; 1Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, 2MD Anderson Cancer Center, Houston, TX, 3Department of Radiation Oncology, Stanford University School of Medicine, Stanford, CA, 4Department of Radiation Oncology, Division of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, 5Department of Gastrointestinal Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX
Purpose/Objective(s): Optimism bias is the overestimation of treatment effectiveness relative to the control arm in clinical trials. The objective of this study was to determine the contribution of control arm overperformance (relative to a pre-trial estimate) to optimism bias in oncology clinical trials. We hypothesized that control arm overperformance would be common and would be associated with trial failure. Materials/
Methods: Phase 3, superiority-designed randomized controlled trials (RCTs) in the field of oncology were identified. Trial data were obtained from ClinicalTrials.gov. We previously identified 385 RCTs that reported both observed and expected hazard ratios. From these studies, the published manuscripts were reviewed to identify studies that stated a pre-trial estimate of control arm performance with respect to the primary endpoint. The actual performance of the control arm was reported, and a numerical ratio of control arm performance (ROCAP) relative to the estimate was calculated. Mann-Whitney U tests were used to compare the median ROCAP (mROCAP) among groups of trials. Results: Two hundred seventy-eight studies met inclusion criteria for analysis of control arm overperformance. Among all 278 trials, the mROCAP was 1.05 (interquartile range [IQR] 0.90-1.27), consistent with statistically significant control arm overperformance relative to a hypothetical mROCAP of 1.0 (one sample Wilcoxon test, p < 0.0001). Of these studies, 149 were successful trials (statistically significant improvement in primary endpoint consistent with trial hypothesis) and 129 were unsuccessful. Among unsuccessful trials that reported an expected control arm outcome, the absolute percentage with control arm overperformance was 67%. Compared to successful trials (mROCAP 1.01; IQR 0.85-1.18), unsuccessful trials (mROCAP 1.12; IQR 0.95-1.39) had significantly greater control arm overperformance (p < 0.001). There was no significant difference in control arm overperformance when comparing industry-sponsored vs non-industry sponsored studies, cooperative group sponsored vs. non-cooperative group sponsored studies, metastatic vs non-metastatic studies, specific disease sites, or types of systemic therapies received. Conclusion: Control arm overperformance is prevalent across phase 3 superiority-designed RCTs. The presence of control arm overperformance is significantly associated with study failure. Factors such as trial sponsorship, disease site, or inclusion of patients with metastatic disease were not predictive of control arm overperformance, suggesting that the propensity for control arm overperformance is widespread. Control arm overperformance represents a significant contributor to optimism bias that may result in underpowered studies and clinical trial failure. Future efforts should identify how to better assess control arm performance to improve trial design.
