Uncertainty quantification

Methods that make a model report how sure it is in a form you can act on, including the option of saying it does not know. Most deployed models return a number whether or not the case in front of them resembles anything in their training data, and that number carries no warning when the case is unfamiliar. Uncertainty quantification supplies the warning: routing low-confidence cases to a human, abstaining instead of guessing, or widening the range around a prediction. It overlaps with calibration and is broader than it, because calibration asks whether stated confidence is honest while uncertainty quantification is about producing and using that signal at all. For a clinical tool the practical test is whether it ever declines to answer, and what happens to the case when it does.

Two tools return the same risk score on a case unlike anything in their training data. One returns it flat. The other returns it with a wide interval and routes the case to a human. The second is more useful and will look worse in a benchmark table, because abstentions cost you coverage. When you evaluate one, ask what fraction of cases it declines, where those cases go, and who is staffed to take them.

Terms like this come up in real clinical scenarios across the HelloAI courses: bite-sized modules with verifiable certificates. An account takes one minute, no password needed.