Högskolan i Skövde

In data-driven decision support, having access to reliable confidence measures for individual predictions is crucial. Machine learning algorithms can provide probabilistic predictions, but these are often poorly calibrated, resulting in misleading decision support. This study empirically evaluates a set of readily available state-of-the-art calibration techniques, including both scaling and binning approaches. Using four different underlying models, and in total 40 publicly available datasets, the results analyzed using rigorous statistical testing show that calibration is generally successful. Specifically, applying a post-hoc calibration will reduce both log losses and calibration errors, without significantly lowering the predictive accuracy. However, the choice of calibration technique should depend on both the underlying model and the size of the dataset, resulting in the following guidelines for calibration: (i) We recommend Venn-Abers for decision trees and naïve Bayes (ii) Beta calibration for Extreme Gradient Boosting (XGB), (iii) Platt scaling for small datasets and Venn-Abers for larger ones when using random forests.