Editorial - Circulation: Cardiovascular Quality and Outcomes

Editorial Realizing the Potential of Clinical Risk Prediction Models Where Are We Now and What Needs to Change to Better Personalize Delivery of Care? Adam C. Salisbury, MD, MSc; John A. Spertus, MD, MPH

A Downloaded from http://circoutcomes.ahajournals.org/ by guest on May 24, 2018

t the turn of the last century, physicians were largely guided by lessons passed down in training, their own personal experience, and the experiences of their colleagues. Although this approach produced thoughtful clinicians, a key limitation remained—even the busiest, most experienced providers could see only so many patients, experience only a limited number of outcomes, and often struggled to ascertain the accuracy of diagnoses or the effectiveness of treatment. These challenges to delivery of safe and effective patient care were subsequently addressed by a growing focus on progressively larger and better designed cohort studies and randomized clinical trials and later by distilling these insights into clinical practice guidelines and appropriate use criteria to help summarize the rapidly evolving medical literature.

In this issue, Wessler et al5 report findings from a rigorous review of clinical prediction models in cardiovascular disease. Examining over 20 years of contemporary literature, they found ≈800 models focusing on conditions across the entire spectrum of cardiovascular disease. In fact, they found that the number of new clinical prediction models has doubled each decade. The growth of prediction models is encouraging, as the entire profession seeks to better understand the outcomes of their patients and how best to optimize these outcomes. Importantly, however, they highlight many important challenges, including the design, development, and testing of prediction models. With respect to the design of clinical prediction models, it is critically important that the results of the model would alter a clinical decision. If a physician will not treat a low- or high-risk patient differently, then what value is there in using a model? With respect to development, the methods currently published are clearly wanting. Only a modest number of publications reported adequate quantification of the model’s capacity to discriminate the outcome of interest (just 63% reported a c-statistic) and even fewer reported a measure of calibration to help readers understand how the model’s predictions are compared with observed event rates (only 36% reported a Hosmer–Lemeshow statistic or a calibration plot)—the latter being critically important in prospectively using models to tailor treatment to risk. Moreover, only a minority of studies reported either an internal or external validation of the model. Another challenge, as the field strives to settle on the best models for clinical care, is that few models reported—just 3%—compared alternative prediction models. Perhaps most distressing is the lack of clinical application of many of these models. Almost no studies have focused on the effective implementation of these models in clinical care—the principal goal of developing such tools. For example, although few cardiologists complete a day without using CHADS2 or CHADS2-Vasc models to predict stroke risk in patients with atrial fibrillation, there are few studies examining whether the use of these models improves the outcomes of patients with atrial fibrillation. One of the few examples of prospectively using a risk prediction model in clinical care, that we are aware of, is the use of the American College of Cardiology’s prediction model for peri–percutaneous coronary intervention bleeding, which was associated with a 44% reduction in the odds of bleeding.6,7 Implementing the best prediction models and demonstrating improvements in care are clearly a high priority for the profession and an important step toward precision medicine. Although the article by Wessler et al5 in this issue, and the recent Prognosis Research Strategy (PROGRESS)8 and

Article see p 368 Despite this exponential growth in well-conducted clinical research, a barrier in applying these studies into clinical care is that an individual patient may not obtain the average benefit observed in a clinical trial. It is well recognized that the heterogeneity of treatment effect across a population can be obscured by focusing only on the mean treatment effect in a population.1–3 Depending on a patient’s age, sex, comorbidities, and other characteristics, that patient may benefit greatly from the same treatment that poses a significant risk for another.4 Providers, after all, are concerned with delivering the safest and most effective treatment for a particular individual, rather than a population of patients. Accordingly, a growing focus has been placed on developing and implementing tools to identify which patients are likely to benefit from a particular treatment or strategy, those who may be harmed, and those for whom balanced risks and benefits exist that should prompt discussions between patients and providers about that patient’s goals and preferences for care. To address this need, there has been a proliferation of published clinical prediction models to help tailor treatment to risk. The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association. From the Department of Cardiovascular Diseases, Saint Luke’s Mid America Heart Institute, Kansas City, MO (A.C.S., J.A.S.); and University of Missouri-Kansas City (A.C.S.). Correspondence to Adam C. Salisbury, MD, MSc, Saint Lukes Mid America Heart Institute and the University of Missouri-Kansas City, 4401 Wornall Rd, Kansas City, MO 64111. E-mail [email protected] (Circ Cardiovasc Qual Outcomes. 2015;8:00-00. DOI: 10.1161/CIRCOUTCOMES.115.002038.) © 2015 American Heart Association, Inc. Circ Cardiovasc Qual Outcomes is available at http://circoutcomes.ahajournals.org DOI: 10.1161/CIRCOUTCOMES.115.002038

1

2   Circ Cardiovasc Qual Outcomes   July 2015

Downloaded from http://circoutcomes.ahajournals.org/ by guest on May 24, 2018

Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis (TRIPOD)9 statements underscores the need to elevate the methodological rigor of model development, there are many additional issues that need to be considered to better understand how prediction models to sort out heterogeneity of treatment benefit can be used to elevate the value of healthcare. First, models that have been shown to be effective deserve a more prominent focus in clinical practice guidelines. In their current form, guidelines focus largely on summarizing the mean treatment effect of trials and many times avoid endorsement of the most effective clinical prediction models to help personalize care. Although they often do include single-patient characteristics to stratify risk and recommend more effective therapies, the specificity of guidelines could be elevated by identifying valid models and recommending treatments for different thresholds of risk. This was recently adopted in the updated cholesterol guidelines,10 but its reach could be far greater and result in easier translation of the guidelines into clinical practice. Once clinical prediction models have been shown to effectively stratify risk and use of the model has been shown to improve outcomes, understanding how to incentivize the model’s use is another challenge. For example, the Society of Thoracic Surgeons risk scores are easily accessible in the form of a web-based calculator. However, these risk scores were rarely used at the bedside until the introduction of transcutaneous aortic valve replacement demanded their use in trial protocols and clinical guidelines.11 The other Society of Thoracic Surgeons models are still rarely used prospectively in routine clinical care at many institutions. Perhaps the most powerful incentive to better deploy risk models in practice is for payers to reimburse providers for using them—something that will also benefit the payers by allowing treatment for their patients to be preferentially provided to those who most benefit. Implementing risk models is complex, time consuming, and expensive, requiring iterative feedback to ensure optimal deployment and supporting the education of providers to know how best to interpret the results and share them with their patients. One potential avenue to support the use of risk models would be to link prospectively established risk, as advocated for by Diamond and Kaul.12 Such payments need not only be for physicians. For example, applying lower copays to patients who benefit substantially more from potent antiplatelet agents,4 such as ticagrelor or prasugrel, could support the use and adherence to such treatments. New and innovative strategies are needed, and research will be needed to identify additional techniques to enhance adoption of models at the point of care and support ongoing use. Another largely unrealized opportunity for use of clinical prediction models is their use to support shared decision making between patients and physicians. Previous studies have found that a majority of patients prefer to play an active role in decision making at the time of acute myocardial infarction.13 Patients’ goals and preferences for care can vary widely,14 and the execution of risk models for individual patients could enable these patients to understand the benefits and risks of treatment for people like them. For example, a patient with a low SYNTAX score who is most concerned about avoiding stroke may prefer percutaneous coronary intervention to

manage severe left main coronary disease, whereas another whose goal is to avoid returning to the hospital for additional procedures may strongly prefer bypass surgery.15 Using risk models to support these decisions represents the pinnacle of patient-centered care. All of these potential applications of clinical prediction models demand continued creation and testing of these models. How will these efforts be incentivized and sustained? Although it is possible that some pharmaceutical and device manufacturers may fund these efforts so that they can create tools to support the use of their interventions at the time they publish the results of clinical trials, this has not yet been done. Professional societies, such as the American College of Cardiology and Society of Thoracic Surgeons, have built models, but these have been self-funded by-products of creating risk-standardized methods for comparing hospitals as part of a larger goal of improving quality. Governmental grants are unlikely to support such efforts, and the grant-writing effort often exceeds the workload in developing the models themselves. Although it is possible that models could be licensed and used to generate the resources needed to support their creation, this will prevent widespread access to potentially valuable models. Identifying funding mechanisms that support this important work is necessary for continued progress. The future of clinical risk prediction models as a driver of more personalized care is promising. To realize this potential, it is critical that the outcomes research community continue to shift its focus from merely describing outcomes to creating models that can support the preferential use of treatments in patients who most benefit, while avoiding use in patients who do not benefit or might even be harmed. This will substantially increase the value of healthcare and is a priority for our field. Although the work of Wessler et al5 to assemble and critically evaluate existing models is an important first step, defining how best to disseminate, to implement and sustain the best models is a top priority.

Disclosures Dr Spertus owns several patents on the ePRISM technology used to deliver clinical risk prediction models at the point of care and has an ownership interest in Health Outcomes Sciences, a company that distributes and supports the ePRISM software to hospitals. Dr Salisbury reports no conflicts.

References 1. Kent DM, Hayward RA. Limitations of applying summary results of clinical trials to individual patients: the need for risk stratification. JAMA. 2007;298:1209–1212. doi: 10.1001/jama.298.10.1209. 2. Kent DM, Alsheikh-Ali A, Hayward RA. Competing risk and heterogeneity of treatment effect in clinical trials. Trials. 2008;9:30. doi: 10.1186/1745-6215-9-30. 3. Kent DM, Rothwell PM, Ioannidis JP, Altman DG, Hayward RA. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials. 2010;11:85. doi: 10.1186/1745-6215-11-85. 4. Salisbury AC, Wang K, Cohen DJ, Li Y, Jones PG, Spertus JA. Selecting antiplatelet therapy at the time of percutaneous intervention for an acute coronary syndrome: weighing the benefits and risks of prasugrel versus clopidogrel. Circ Cardiovasc Qual Outcomes. 2013;6:27–34. doi: 10.1161/CIRCOUTCOMES.112.965624. 5. Wessler BS, Lai L, Kramer W, Cangelosi M, Raman G, Lutz JS, Kent DM. Clinical prediction models in cardiovascular disease: the Tufts PACE CPM database. Circ Cardiovasc Qual Outcomes. 2015;8:XXX–XXX. doi: 10.1161/CIRCOUTCOMES.115.001693.

Salisbury and Spertus   Clinical Risk Prediction Models Editorial   3

Downloaded from http://circoutcomes.ahajournals.org/ by guest on May 24, 2018

6. Rao SV, McCoy LA, Spertus JA, Krone RJ, Singh M, Fitzgerald S, Peterson ED. An updated bleeding model to predict the risk of post-procedure bleeding among patients undergoing percutaneous coronary intervention: a report using an expanded bleeding definition from the National Cardiovascular Data Registry CathPCI Registry. JACC Cardiovasc Interv. 2013;6:897–904. doi: 10.1016/j.jcin.2013.04.016. 7. Spertus JA, Decker C, Gialde E, Jones PG, McNulty EJ, Bach R, Chhatriwalla AK. Precision medicine to improve use of bleeding avoidance strategies and reduce bleeding in patients undergoing percutaneous coronary intervention: prospective cohort study before and after implementation of personalized bleeding risks. BMJ. 2015;350:h1302. 8. Steyerberg EW, Moons KG, van der Windt DA, Hayden JA, Riley SS, Hemingway H, Altman DG. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10:e1001381. 9. Collins GS, Reisma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). Ann Intern Med. 2015;162:735–736. 10. Stone NJ, Robinson JG, Lichtenstein AH, Bairey Merz CN, Blum CB, Eckel RH, Goldberg AC, Gordon D, Levy D, Lloyd-Jones DM, McBride P, Schwartz JS, Shero ST, Smith SC Jr, Watson K, Wilson PW, Eddleman KM, Jarrett NM, LaBresh K, Nevo L, Wnek J, Anderson JL, Halperin JL, Albert NM, Bozkurt B, Brindis RG, Curtis LH, DeMets D, Hochman JS, Kovacs RJ, Ohman EM, Pressler SJ, Sellke FW, Shen WK, Smith SC Jr, Tomaselli GF; American College of Cardiology/American Heart Association Task Force on Practice Guidelines. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129(25 suppl 2):S1–45. doi: 10.1161/01. cir.0000437738.63853.7a.

11. Nishimura RA, Otto CM, Bonow RO, Carabello BA, Erwin JP 3rd, Guyton RA, O’Gara PT, Ruiz CE, Skubas NJ, Sorajja P, Sundt TM 3rd, Thomas JD; ACC/AHA Task Force Members. 2014 AHA/ACC Guideline for the Management of Patients With Valvular Heart Disease: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129:e521–e643. doi: 10.1161/CIR.0000000000000031. 12. Diamond GA, Kaul S. Evidence-based financial incentives for healthcare reform: putting it together. Circ Cardiovasc Qual Outcomes. 2009;2:134– 140. doi: 10.1161/CIRCOUTCOMES.108.825695. 13. Krumholz HM, Barreto-Filho JA, Jones PG, Li Y, Spertus JA. Decision-making preferences among patients with an acute myocardial infarction. JAMA Intern Med. 2013;173:1252–1257. doi: 10.1001/ jamainternmed.2013.6057. 14. Alonso-Coello P, Montori VM, Diaz MG, Devereaux PJ, Mas G, Diez AI, Sola I, Roura M, Souto JC, Oliver S, Ruiz R, Coll-Vinent B, Gich I, Schunemann HJ, Guyatt G. Values and preferences for oral antithrombotic therapy in patients with atrial fibrillation: physician and patient perspectives. Health Expect. 2014. doi: 10.1111/hex.12201. Available at http:// www.biomedcentral.com/1472-6963/8/221. Accessed June 30, 2015. 15. Farooq V, van Klaveren D, Steyerberg EW, Meliga E, Vergouwe Y, Chieffo A, Kappetein AP, Colombo A, Holmes DR Jr, Mack M, Feldman T, Morice MC, Ståhle E, Onuma Y, Morel MA, Garcia-Garcia HM, van Es GA, Dawkins KD, Mohr FW, Serruys PW. Anatomical and clinical characteristics to guide decision making between coronary artery bypass surgery and percutaneous coronary intervention for individual patients: development and validation of SYNTAX score II. Lancet. 2013;381:639– 650. doi: 10.1016/S0140-6736(13)60108-7. Key Words: Editorials ◼ decision support techniques ◼ patient care ◼ physicians ◼ risk assessment

Realizing the Potential of Clinical Risk Prediction Models: Where Are We Now and What Needs to Change to Better Personalize Delivery of Care? Adam C. Salisbury and John A. Spertus Downloaded from http://circoutcomes.ahajournals.org/ by guest on May 24, 2018

Circ Cardiovasc Qual Outcomes. published online July 7, 2015; Circulation: Cardiovascular Quality and Outcomes is published by the American Heart Association, 7272 Greenville Avenue, Dallas, TX 75231 Copyright © 2015 American Heart Association, Inc. All rights reserved. Print ISSN: 1941-7705. Online ISSN: 1941-7713

The online version of this article, along with updated information and services, is located on the World Wide Web at: http://circoutcomes.ahajournals.org/content/early/2015/07/07/CIRCOUTCOMES.115.002038.citation

Permissions: Requests for permissions to reproduce figures, tables, or portions of articles originally published in Circulation: Cardiovascular Quality and Outcomes can be obtained via RightsLink, a service of the Copyright Clearance Center, not the Editorial Office. Once the online version of the published article for which permission is being requested is located, click Request Permissions in the middle column of the Web page under Services. Further information about this process is available in the Permissions and Rights Question and Answer document. Reprints: Information about reprints can be found online at: http://www.lww.com/reprints Subscriptions: Information about subscribing to Circulation: Cardiovascular Quality and Outcomes is online at: http://circoutcomes.ahajournals.org//subscriptions/