Abstract: As we begin deploying machine learning models in consequential settings like medical diagnostics or self-driving vehicles, we need ways of knowing when the model may make a consequential error (for example, that the car doesn't hit a human). I'll be discussing how to generate rigorous, finite-sample confidence intervals for any prediction task, any model, and any dataset, for little computational cost. This will be a chalk talk. I will primarily discuss conformal prediction and related methods, which work for a large class of prediction problems including those with high-dimensional, structured outputs (e.g. instance segmentation, multiclass or hierarchical classification, protein folding, and so on).