AI.info
Evaluation
Turning a number into evidence a decision can rest on: baselines, populations, harms, and what a benchmark cannot answer.
- Evaluation as Decision Evidence
- Intended Use, Units, Populations, and Harm
- Baselines, Comparators, and Counterfactual Thinking
- Dataset Partitions and Test-Set Governance
- Grouped, Temporal, Spatial, and Hierarchical Splits
- Cross-Validation, Nested Resampling, and Variance
- Bias, Variance, and Learning-Curve Diagnostics
- Benchmark Design, Contamination, and Saturation
- Metric Portfolios, Acceptance Criteria, and Release Gates
- Confusion Matrices and Error Taxonomy
- Accuracy, Balanced Accuracy, and Averaging
- Precision, Recall, Specificity, and F-Scores
- Thresholds, Costs, Utilities, and Capacity
- ROC, Precision–Recall Curves, AUC, and Discrimination
- Calibration, Log Loss, Brier Score, and Reliability
- Selective Prediction, Abstention, and Risk–Coverage
- Conformal Prediction: Coverage and Efficiency
- Regression Metrics and Residual Diagnostics
- Quantiles, Prediction Intervals, and Probabilistic Regression
- Forecasting Evaluation Across Time, Horizons, and Hierarchies
- Ranking, Retrieval, and Top-K Metrics
- Recommender Evaluation Beyond Relevance
- Clustering and Unsupervised Evaluation
- Anomaly Detection and Alerting Evaluation
- Computer Vision Metrics Across Tasks
- NLP Metrics for Extraction, QA, Translation, and Summarization
- Generative AI Evals: Groundedness, Tools, and Safety
- Human Evaluation, Rubrics, and Annotation Reliability
- Slice Analysis and Subgroup Performance
- Fairness Metrics, Tradeoffs, and Decision Context
- Robustness, Stress Tests, and Out-of-Distribution Evaluation
- Uncertainty, Bootstrap, and Confidence Intervals
- Significance, Effect Size, Multiplicity, and the Winner’s Curse
- Online Controlled Experiments and A/B Testing
- Post-Deployment Evaluation, Drift, and Delayed Labels
- Evaluation Harnesses, Reporting, and Release Governance
- Capstone: Design an Evaluation Program That Can Survive Contact With Reality