Quality assessment of primary studies is a crucial component of any systematic review process. It is important that reviewers and users of systematic reviews are able to assess the potential impact of sources of bias in the included studies upon the results and conclusions of the review. Systematic review of prediction studies, in particular prediction modelling studies, is a relatively new and evolving area. In comparison to systematic reviews of treatments and diagnostic tests critical appraisal tools and review methods in this area are not yet well developed.
The PROBAST project brings together experts in the field of prediction research, experienced systematic reviewers and methodologists with experience of developing quality assessment tools, with the aim of developing a prediction study risk of bias tool, which will focus on prediction modelling studies.
We are using a Delphi process to develop PROBAST. Forty experts in the fields of prediction research and systematic review methodology are participating in the Delphi process. Early rounds of the Delphi process resulted in agreement that PROBAST would address both risk of bias and applicability. Risk of bias addresses the extent to which reported estimates of the predictive performance/accuracy (e.g. coefficients, discrimination, calibration and (re)classification estimates) of the prediction model are potentially biased and applicability refers to the extent to which the reported prediction model matches the review question. A domain-based approach, similar to that implemented in QUADAS-2, with domains being rated as high, low and unclear risk of bias, was also agreed.
The PROBAST structure currently includes the following four domains: participant selection; predictors; outcome; analysis. Signalling questions for each domain will be defined in current and future rounds and the domain structure may also be refined. We are currently preparing two journal articles on the tool, one introducing the tool and another one with detailed explanation and elaboration.
Progress in the development of PROBAST was presented at recent Cochrane Colloquia, Guidelines International Conference and at the University of Birmingham symposium, Methods for Evaluating Medical tests and Biomarkers.
Members of the PROBAST steering group
- Gary Collins
- Jos Kleijnen
- Susan Mallett
- Carl Moons
- Hans Reitsma
- Richard Riley
- Marie Westwood
- Penny Whiting
- Robert Wolff
- Bouwmeester W, Zuithoff NP, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KG. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1-12.
- Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011 Oct 18;155(8):529-36.
- Hayden JA, van der Windt DA, Cartwright JL, Côté P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013 Feb 19;158(4):280-6.
- Altman DG, McShane LM, Sauerbrei W, Taube SE. Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK): explanation and elaboration. PLoS Med. 2012;9(5):e1001216.
- Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015;350:g7594
- Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, Vickers AJ, Ransohoff DF, Collins GS. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-W73. doi:10.7326/M14-0698