Table of Contents
- The Problem MLOps Exists to Solve
- The Lifecycle, Honestly Described
- The Five Capabilities You Need
- Four Levels of Maturity
- The Feature Store Question
- Drift: The Failure Mode Nobody Sees
- Evaluation Is the Whole Game
- How LLMs Changed the Picture
- Who Actually Does MLOps
- What This Costs
- Where to Start If You Have Nothing
- Seven Mistakes That Kill Models in Production
- Frequently Asked Questions
- Final Thoughts
Most machine learning projects do not fail during training. They fail eight months after deployment, when a model that was 94% accurate at launch is quietly making worse decisions than the rule it replaced, and nobody has looked at it since.
MLOps is the discipline of preventing that. It is unglamorous, it is mostly engineering rather than data science, and it is the difference between a model that creates value and one that creates a liability.
The Problem MLOps Exists to Solve
Traditional software is deterministic. You write code, test it, deploy it, and it behaves the same way tomorrow as today unless someone changes it.
Machine learning breaks that in three ways.
The behaviour depends on data, not just code. Two artefacts determine what your system does – the code and the data it learned from – and only one of them is in version control at most companies.
It degrades without anyone touching it. The world moves. Customer behaviour shifts, a competitor changes pricing, a supplier changes their data format. The model is unchanged and is now wrong.
Failure is silent. A broken API returns a 500. A degraded model returns a confident, well-formatted, incorrect answer. Nothing alerts.
MLOps is the set of practices that makes machine learning behave more like software you can operate: reproducible, versioned, monitored, and recoverable.
The Lifecycle, Honestly Described
The textbook version is a neat circle. The real version has a lot more backtracking.
1. Data preparation. Sourcing, cleaning, labelling, feature engineering. Consistently the largest time sink, and the stage most likely to introduce errors nobody notices for months.
2. Experimentation. Many runs, varying features, architectures and hyperparameters. Without tracking, this becomes an untraceable mess by week three.
3. Validation. Not just accuracy – performance across the segments that matter, behaviour on edge cases, and comparison against the baseline you are actually replacing.
4. Deployment. Packaging the model with its dependencies and exposing it somewhere. Sounds simple, and is where the environment mismatches surface.
5. Monitoring. Watching prediction quality, input distributions and latency. This is the stage teams skip and then regret.
6. Retraining. When performance degrades, back to step one – ideally automatically, with the same pipeline that produced the original.
The honest bit: most organisations do steps one to four well and stop. MLOps is largely about the discipline to keep going.
The Five MLOps Capabilities You Need
Whatever tools you use, five things must exist somewhere – our comparison of MLOps tools by layer covers what to use for each.
Experiment tracking. Tools like MLflow and Weights & Biases record every training run with its parameters, data version, code version and results. Without this you cannot answer “what produced the model currently in production”, which is a question you will eventually be asked urgently.
Versioning of data and models. DVC is the common answer here. Code versioning is solved. Data and model versioning frequently is not, and reproducibility requires all three.
Pipeline orchestration. Training as a repeatable, scheduled pipeline rather than a notebook someone runs manually. The overlap with data engineering is total – the same tools in our guide to data orchestration apply here.
Model serving. Exposing predictions reliably, with versioning so you can roll back, and ideally shadow deployment so you can compare a new model against the live one before switching.
Monitoring. Both operational (latency, errors) and statistical (are inputs and outputs still distributed as expected). The second is the one that catches the failures that matter.
Four Levels of MLOps Maturity
Useful for locating yourself honestly rather than aspirationally.
| Level | What it looks like | Typical failure |
|---|---|---|
| 0 — Manual | Notebooks, models handed over as files | Nobody can reproduce anything |
| 1 — Reproducible | Training is a script, experiments tracked | Deployment still manual and slow |
| 2 — Automated | Pipelines run on schedule, models registered | Degradation still found by users |
| 3 — Continuous | Monitoring triggers retraining automatically | Complexity outpaces the team |
Most teams sit between 0 and 1 and describe themselves as level 2. The useful move is not jumping to level 3 – it is getting reliably to the next level up.
Level 3 is genuinely not the goal for everyone. Automatic retraining on a model nobody monitors carefully is a way to automate the propagation of a mistake.
The Feature Store Question
Worth addressing because it is the piece of MLOps infrastructure teams most often build too early.
A feature store is a central place where the computed inputs to your models live, serving both training and inference from the same definitions. The problem it solves is training-serving skew – the situation where a feature is calculated one way in your training pipeline and slightly differently in production, and your model quietly performs worse than it tested.
That is a real and expensive problem. It is also not one most teams have yet.
You probably need one if several models share features, features are computed in real time, or you have already been bitten by skew. You probably do not if you have two models, batch predictions and one person writing the pipelines.
The lighter alternative that covers most of the benefit: define your features once in shared code, and use the same function in both paths. It is not a feature store, it prevents the specific failure a feature store prevents, and it takes an afternoon rather than a quarter.
This is a recurring pattern in MLOps – the sophisticated tool solves a real problem, and the simple discipline solves eighty percent of it for a fraction of the effort. Reach for the tool when the discipline stops scaling, not before.
Drift: The Failure Mode Nobody Sees
If you take one thing from this article, take this section.
Data drift is when your inputs change distribution. Your customer base shifts younger, a new region launches, an upstream system starts sending nulls where it used to send zeros. The model still runs. It is now being asked about a world it was not trained on.
Concept drift is worse and subtler: the relationship between inputs and outcome changes. What predicted churn last year no longer does, because your product changed. Your inputs look completely normal.
Neither produces an error. Both degrade decisions quietly, and the loss compounds for as long as nobody checks.
The practical defence is monitoring the input distributions, not just the accuracy – because ground truth often arrives weeks late, and by the time accuracy drops you have been wrong for a month. Tools like Evidently, Arize and Fiddler exist specifically for this.
Evaluation Is the Whole Game
If MLOps has a single centre of gravity, it is this, and it is the part teams invest least in.
Every other capability exists to serve one question: is this model better than what it replaced, and is it still better today? Without a credible answer, experiment tracking records runs you cannot compare, monitoring alerts on changes you cannot interpret, and automated retraining ships models nobody has validated.
Three things make an evaluation set credible.
It reflects production, not your training data. Sampled from real traffic, including the awkward cases and the segments that matter commercially rather than statistically.
It is stable over time. If the benchmark changes every time you evaluate, you cannot compare across versions. Version it like code and change it deliberately.
It measures what the business cares about. Overall accuracy hides the failures that matter. A fraud model at 99% accuracy that misses the expensive cases is worse than one at 95% that catches them.
Teams with a strong evaluation set and modest MLOps tooling consistently outperform teams with sophisticated infrastructure and a vague sense that the model seems fine. It is the least automatable part of the discipline, which is probably why it gets skipped.
How LLMs Changed the Picture
Much of what teams now deploy is not a trained model at all – it is a prompt, a retrieval system and an API call. The MLOps questions change shape but do not disappear.
Versioning becomes prompt versioning. A prompt is production configuration that changes behaviour. It belongs in version control with a change history, and almost nobody does this at first.
Evaluation gets harder. There is no accuracy metric for “was this summary good”. You need an evaluation set with expected behaviour, and a way to compare versions – which is the same discipline described in our guide to retrieval-augmented generation.
The dependency is external. A hosted model can change underneath you without notice. Pin versions where you can, and keep an evaluation suite that would catch a silent regression.
Cost becomes an operational metric. Token spend per request is a production number worth monitoring alongside latency.
The category name for this is drifting toward LLMOps, but the underlying discipline is identical: version everything, evaluate continuously, monitor in production.
Who Actually Does MLOps
The organisational question causes more failed initiatives than the technical one.
Three models exist in practice. Data scientists own it end to end, which works at small scale and breaks down when the infrastructure becomes a job in itself. A platform team owns the infrastructure and data scientists deploy onto it, which scales well and risks building a platform nobody wanted. A dedicated MLOps engineer bridges both, which is effective and hard to hire for.
What matters more than the structure is that someone is accountable for a model after it ships. The most common organisational failure in MLOps is a clean handover that never happened – the data scientist moved to the next project, the engineering team considers it a black box, and the model runs unattended for a year.
A useful test: pick a model in production and ask who would be paged if it started returning nonsense. If the answer takes more than a few seconds, you have an ownership problem rather than a tooling problem.
What This Costs
Three components, and the balance surprises people.
Tooling is usually the smallest. The core MLOps stack has strong open-source options at every layer. You can build a genuinely capable setup on free software.
Compute is lumpy. Training is expensive and periodic; serving is cheaper and constant. Teams frequently over-provision serving infrastructure for a model handling a handful of requests per minute.
People are the real cost. Building and maintaining pipelines, investigating degradation, keeping evaluation sets current. This does not go away as tooling improves – it shifts from setup to operation.
The calculation worth doing before starting: what does a wrong prediction cost, multiplied by how many you would make before noticing? For a recommendation engine that number is small. For anything touching pricing, credit or safety it is not, and it justifies considerably more MLOps investment than teams typically make.
Where to Start If You Have Nothing
An MLOps rollout in order, where each step is worth doing before the next.
- Track experiments. One tool, every run logged. This alone eliminates most reproducibility pain and takes an afternoon to set up.
- Get training out of notebooks. A script with parameters, runnable by someone who did not write it.
- Register your models. A central place recording which model is in production, what produced it, and how to roll back.
- Monitor inputs. Before sophisticated drift detection, simply log what your model receives and alert on distribution changes.
- Automate retraining last. Only once you trust your evaluation. Automating a pipeline you do not trust just makes mistakes faster.
Teams that follow this order end up with something reliable. Teams that start by building an automated retraining pipeline usually end up with an impressive system nobody trusts.
Seven MLOps Mistakes That Kill Models in Production
1. No baseline. If you cannot say what the model beats – a rule, a human, the previous version – you cannot say whether it is working.
2. Training-serving skew. Features computed one way in training and another in production. The most common cause of a model that tested well and performs badly.
3. No ground truth pipeline. If you never capture what actually happened, you can never measure whether predictions were right.
4. Deploying without rollback. Every model deployment needs a one-command path back to the previous version.
5. Monitoring only accuracy. It arrives too late. Watch inputs, which change first.
6. One person understands it. Machine learning systems are unusually prone to this, and unusually painful when that person leaves.
7. No decommissioning plan. Models outlive their usefulness. Somebody should be asking annually whether each one still earns its place.
Frequently Asked Questions
What is the difference between MLOps and DevOps?
DevOps versions code. MLOps versions code, data and models, and adds monitoring for statistical degradation that has no equivalent in traditional software.
Do small teams need this?
Some of it. Experiment tracking and model registration pay for themselves immediately at any size. Automated retraining pipelines usually do not.
How often should models be retrained?
When monitoring says so, not on a calendar. Scheduled retraining without monitoring is guessing, and it can make things worse.
Is MLOps a job or a practice?
Both, increasingly. Larger organisations hire for it; smaller ones distribute it between data scientists and platform engineers.
Does this apply to LLM applications?
Yes, with different emphasis. Less retraining, far more evaluation and prompt versioning – the discipline transfers even when the artefacts change.
What should I not automate?
The decision to promote a model to production. Every automated pipeline should end at a gate a human passes, at least until you have months of evidence.
Final Thoughts
The uncomfortable truth about MLOps is that the hard parts are not the interesting ones. Experiment tracking, versioning, monitoring input distributions – none of this is intellectually exciting, and all of it determines whether a model creates value or quietly stops working.
The teams that succeed with machine learning in production are rarely the ones with the most sophisticated models. They are the ones who can answer, without hesitation, what is deployed, what produced it, how it is performing today, and how to roll it back.
Start with experiment tracking. Add a model registry. Monitor your inputs. That covers most of the value, and it is achievable in a fortnight.
[…] without monitoring. Scheduled retraining on drifted data can make forecasts worse. Our guide to MLOps covers the monitoring […]