Validation of the Economic and Health Outcomes Model of Type 2 Diabetes Mellitus (ECHO-T2DM)
Willis M, Johansen P, Nilsson A and Asseburg C
Type 2 diabetes mellitus (T2DM) is chronic and progressive disease and economic evaluation requires that the benefits and costs of treatment interventions be captured over the long run using economic simulation modeling techniques. From a modeling standpoint, T2DM ranks among the most challenging disease areas due to its impact on multiple inter-related organ systems and multiple treatment goals (including blood glucose, blood pressure, and blood lipids).
The usefulness of a model depends, naturally, on its ability to accurately predict health and economic outcomes of patients in real-life treatment settings and users of model results need to be assured of the soundness of model predictions. External validation is defined in applicable guidelines as replicating clinical trials or data registries with a model and comparing the model predictions with the actual observed results (ISPOR/SMDM). Models of T2DM have been at the forefront of model validation, in part because modeling the complex multi-organ pathophysiology usually requires advanced programming (often using compiled code) resulting in reduced transparency.
ECHO-T2DM (v. 2.3.0) is a stochastic, micro-simulation (i.e., patient-level) model, suitable for estimating long-term cost-effectiveness of T2DM interventions. Recent model upgrades created the need for a new model validation. We followed the principles espoused by ISPOR/SMDM. Specifically, study characteristics were entered into ECHO-T2DM to replicate 12 clinical studies (including 17 patient populations). Model predictions for 202 study endpoints were then compared to observed values using established statistical techniques. Sub-group analyses were conducted separately endpoints from studies used to construct the model and those independent of the model and by type of endpoint (microvascular, macrovascular, or mortality).
Using established statistical techniques, model predictions agreed generally well with observed clinical endpoints. Plotting predictions versus observed values, for example, found that most data points lie close to the 45-degree line, which corresponds to perfect match, and there was no clear pattern for under- vs. over-prediction. The R2 value was about 0.90 and an F test of perfect match could not be rejected, indicating good fit. Good fit was found for both dependent and independent endpoints, separately. Despite smaller sample sizes, fit was also reasonably good for individual types of endpoints as well, albeit with a tendency to over-predict mortality and rejection of the F test for macrovascular endpoints. ECHO-T2DM continues to match health outcomes in clinical trials in T2DM, with prediction accuracy similar to other leading models of T2DM.
PharmacoEconomics 2017; 35(3): 375-96