Services

Prediction Model Toolkit

QHS researchers provide insight on the creation, implementation and best practices for prediction models.

The resources below were curated by and based on the work of Dr. Mike Kattan, a pioneer in modern statistical prediction methods for medical decision making and department chair of QHS for 20 years.

Prediction Model Development: A guide to understanding, creating and using prediction models.

• Requirements for a statistical prediction model endorsed by the American Joint Commission on cancer.
• Processing data from Epic for research
• Checklist for presenting prediction models in a paper
• Creating a prediction model when there is a time-varying covariate
• Why propensity scores do not improve the accuracy of statistical prediction models
• Code to make binary, ordinal and survival outcome nomograms
• How to make a competing risks regression nomogram, including detailed R code
• What machine learning approaches lose compared to:
• logistic regression
• Cox regression
• Cox regression – additional example
• Machine learning is RSF
• Machine learning is RSF – second example

Prediction Model Assessment: How to compare, test and implement prediction models.

1. Comparison of two rival prediction tools that predict on an ordinal scale.
2. How to make a calibration plot for a prediction model in the presence of competing risks.
3. How to estimate a time-dependent concordance index.
4. Why two prediction models can't be tested on separate datasets.
5. A guide to the many metrics for assessing prediction models.
6. How to determine the area under the ROC curve for a binary diagnostic test.
7. Use the Index of Predictive Accuracy (IPA) instead of concordance index.
8. Framework for reviewers evaluating statistical prediction modeling manuscripts.
9. How to fix 8 common statistical prediction modeling mistakes.

Prediction Communication and Interpretation: Recommendations for patient education, model implementation and diagnosis.

1. How to prove treatment success in cancer victims
2. How patients should be counseled on prediction models, including a table of tailored predictions of benefits and harms crossed by treatment options.
3. Why confidence intervals should not be put around a predicted probability
4. Why a statistical prediction model must be applied to achieve informed consent.
5. What is a real nomogram
6. Definition of comparative effectiveness.
7. How to recognize risk on a continuum.
8. What to consider when judging a new marker
9. Why cancer staging systems need to phase out
10. Exploring "I'm a patient, not a statistic"
11. How to make online risk calculators.
12. How patients respond to risk calculator decision aid
13. How simplifying a regression model with friendly integers loses accuracy.
14. The alternative to reporting a p-value with biomarker studies

Prediction Communication and Interpretation: Recommendations for patient education, model implementation and diagnosis.

1. How averaging individual predictions improves accuracy over the individuals themselves.
2. How cognitive biases can impact predicted probabilities developed by doctors
1. not terribly accurate
2. highly variable
3. occasionally worse than a coin toss, also here.
4. and get worse when doctors choose to ignore what a prediction model says

Decision Analysis and Utility Assessment

1. How the method used to measure utilities affects the decision analytic recommendation.
2. How to measure individual patient utilities to run a decision analysis
3. Stop multiplying health state utilities to get the utility of the combined health state.
4. Why utilities are more helpful than traditional health-related quality of life measures with respect to medical decision making.
5. Why the layout of the time trade-off is problematic.
6. How to measure standard gamble on paper.

Novel Uses of Prediction Models

1. An example of how to make a synthetic control arm for a single arm study, using a prediction model, and how to calculate the p-value.
2. Why to apply a nomogram rather a decision analysis at the bedside

General Biostatistical Issues

1. Statistical reporting guidelines.
2. Guidelines for figures and tables.
3. Problems with p-values and ways to avoid using them.
4. The problem of causal language in observational studies.
5. Common problems with statistical prediction modeling papers.

Risk Calculators

This information is provided by Cleveland Clinic as a convenience service only to physicians and is not intended to replace the physicians' medical advice. Please remember that this information, in the absence of a visit with a physician's patient, must always be considered as an educational service only and is not designed to replace a physician's independent judgment about the appropriateness or risks of a procedure or recommendations for a given patient.

Cleveland Clinic makes no representation or warranty concerning the accuracy or reliability of this information and does not warrant the results of using this tool. In no event shall Cleveland Clinic be liable for any damages, direct, indirect, consequential or otherwise, relating to the use of this information or this tool.

Please continue to the risk calculators on the riskcalc.org website.  

For questions, contact [email protected]

Grant Development

QHS provides free grant development and study design support to help increase the probability of funding.

QHS has been successful in obtaining both National Institutes of Health and industry-sponsored funding. Our success stories include:

• $3.4 million from NHLBI for Pulmonary Vascular Disease Phenomics Program.
• $4.7 million from NINDS for Multi-Omic Biomarkers for Neuropathic Pain Secondary to Chemotherapy.
• $3.5 million from NIA for Digital Twin Neighborhoods for Research on Place-Based Health Inequalities in Mid-Life.
• $4.9 million from Industry Partners for multiple projects and studies. Partner Industries, Bayer Pharmaceuticals, Boehringer Ingleham Pharmaceuticals, International Business Machine, Merck & Co. Inc., Novo Nordisk, Siemens Medical Solutions Inc.

Educational Offerings

Classes & Services

QHS-instructed courses at the Cleveland Clinic:

• Clinical Research Scholars Program
• LRI Molecular Medicine PhD program (MMed501)
• CME credit courses for Cleveland Clinic trainees and staff:
  • Medical Biostatistics Part 1
  • Medical Biostatistics Part 2

QHS-instructed courses at the Cleveland Clinic Lerner College of Medicine (CCLCM):

• CMED402: Medical Biostatistics
• CMED450: Clinical Trials
• CMED458: Statistical Modeling with Applications in Clinical Research

We also:

• Review student research proposals for the Clinical Research Block
• Facilitate journal clubs
• Maintain education databases

---

Cleveland Clinic Statistical Dataset Repository

Download educational datasets for analysis in csv and sas7bdat formats. Descriptions and data dictionaries are available in the table below.

Download datasets