Data & AI·5 min read
The model is the small box in the middle
A decade before the current AI cycle, Google's engineers documented exactly why ML systems are cheap to prototype and expensive to keep alive.
Source
Hidden Technical Debt in Machine Learning Systems
D. Sculley et al. (Google) · NeurIPS (Advances in Neural Information Processing Systems 28) · 2015
Read the originalAslanWay is not the author of this work. The summary below describes the source; the commentary that follows is ours and is not endorsed by the author.
What it argues
The paper's argument is that machine learning makes it unusually easy to accumulate technical debt, because ML systems have all the maintenance problems of normal software plus a set of their own.
Its most reproduced image shows the ML code as a small box surrounded by much larger blocks: configuration, data collection, feature extraction, data verification, serving infrastructure, resource management, process management tooling, analysis tools and monitoring. The modelling is a minority of the system.
The ML-specific failure modes
- Entanglement — changing any input distribution changes the meaning of every other feature, so nothing is truly isolated.
- Hidden feedback loops — the model's own outputs influence the data it will later be trained on.
- Undeclared consumers — other teams quietly depend on your model's output, and you find out when you change it.
- Data dependencies — harder to track than code dependencies, and there is no compiler to catch them.
- Configuration debt — the volume of tuning options grows until nobody can say why a setting is what it is.
The paper predates transformers and generative models, and its analysis has aged remarkably well. The current wave has changed what the small box contains, not how much of the system it represents.
What we do with it
- When someone estimates an AI project by estimating the modelling work, they have costed the small box and nothing else. The surrounding infrastructure is where the schedule goes.
- Undeclared consumers are the reason to publish an explicit contract for model outputs from day one, even when there is only one consumer.
- Gartner forecast that 30%+ of generative AI projects would be abandoned after proof of concept by the end of 2025, citing data quality, risk controls, cost and unclear value. Every one of those is on Sculley's list from 2015.