AI.info

Training and optimization

Objectives, losses and the feedback loop that fits a model — and what a training curve does not tell you.

1. Training and Optimization as Feedback Engineering
2. From Product Goals to Trainable Objectives
3. Loss Families and Output Contracts
4. Reduction, Masking, Weighting, and Multi-Output Objectives
5. Computational Graphs and the Gradient Signal
6. Stochastic Gradient Descent: Direction Under Noise
7. Mini-Batches, Gradient Variance, and Effective Batch Size
8. Learning Rate: Stability, Speed, and Scale
9. Warmup, Decay, Restarts, and Schedule Design
10. Momentum and Nesterov Dynamics
11. AdaGrad, RMSProp, Adam, and Adaptive Updates
12. AdamW, Weight Decay, and Parameter Groups
13. Curvature, Conditioning, and Preconditioning
14. Initialization and Signal Propagation
15. Normalization Layers and Training Behavior
16. Residual Paths and Deep Network Optimization
17. Gradient Clipping, Exploding Updates, and Trustworthy Norms
18. Numerical Precision, Mixed Precision, and Loss Scaling
19. Regularization as a Portfolio
20. Data Augmentation and Learned Invariances
21. Imbalance, Rare Cases, and Cost-Sensitive Training
22. Sampling Policies, Curriculum, and Hard Examples
23. Learning Curves and Capacity Diagnostics
24. Validation, Early Stopping, and Checkpoint Choice
25. Hyperparameter Search and Multi-Fidelity Experiments
26. Randomness, Seeds, and Reproducible Comparisons
27. Debugging a Model That Will Not Learn
28. NaNs, Divergence, Saturation, and Silent Instability
29. Transfer Learning and Fine-Tuning Schedules
30. Multi-Task and Multi-Objective Optimization
31. Distributed Data-Parallel Training
32. Large-Batch Training, Gradient Accumulation, and Scaling Limits
33. Memory, Throughput, and Compute-Efficient Training
34. Weight Averaging, EMA, SWA, and Checkpoint Ensembling
35. Sharpness-Aware Methods and Advanced Optimizer Choices
36. Monitoring Training Runs and Making Stop/Continue Decisions
37. Capstone: Design, Run, Diagnose, and Defend a Training Program