MLOps and Model Deployment Fundamentals
Take a machine learning model from notebook to production: track experiments, version data and models, package and serve them, automate CI/CD, monitor drift and document for governance.
What you will learn
- Describe the ML lifecycle and the three MLOps maturity levels, and explain why models fail in production.
- Track experiments and make a training run reproducible with logged parameters, metrics, code version and pinned environment.
- Version data and models, and use a model registry with aliases and lineage to promote a model safely.
- Package a model with a signature and a pinned environment, and choose between batch and real-time serving and a rollout strategy.
- Design a CI/CD pipeline for ML with data checks, model tests, quality gates and automated rollback.
- Monitor a deployed model for drift and performance, and decide when and how to retrain, using a feature store to avoid skew.
- Document models and operate them under governance: model cards, audit trail, access control, incident runbooks and regulatory duties.
Modules
- From notebook to production
- Experiment tracking and reproducibility
- Versioning data and models
- Packaging and serving models
- CI/CD and pipelines for ML
- Monitoring, drift and retraining
- Governance, documentation and readiness
Course facts
- Estimated study time
- about 60 minutes
- Final assessment
- 42 questions · Pass mark 80%
- Available in
- English, Français, العربية
- Price
- Free. Optional verifiable certificate: $19
- Last updated
- Published by
- AMAADOR ACADEMY
Who this course is for
Data scientists, ML and software engineers, data engineers, DevOps staff and technical team leads who need to run models in production. Also useful for analysts moving into an MLOps role.
Prerequisites
Basic Python, Git and command-line skills, and a first idea of how a model is trained and evaluated. No cloud account is needed to follow the course.
Course content
1. From notebook to production
A model in a notebook is a prototype. A model in production is a service: it must run every day, on fresh data, for real users, and someone must answer when it breaks. MLOps (machine learning operations) is the set of practices that make this reliable by applying DevOps ideas to code, data and models together.
The ML lifecycle is a loop, not a line: monitoring feeds the next round of training.
In their well-known paper on hidden technical debt in ML systems, Sculley and colleagues (Google, NeurIPS 2015) observed that the model code is only a small box inside a much larger system: data collection, feature extraction, configuration, serving, monitoring. Most production problems come from that surrounding system, not from the algorithm.
Typical reasons models fail after launch:
- Data changes. Customers, sensors or prices change, and the input no longer looks like the training set.
- Training-serving skew. Features are computed one way in training and another way in production.
- No reproducibility. Nobody can rebuild the model that is live.
- No monitoring. Errors are found by customers, weeks later.
2. Experiment tracking and reproducibility
Data science is trial and error. Without a record, you cannot say which run was best or why. An experiment tracker stores, for each run, what you changed and what happened. MLflow Tracking is one of the most widely used open-source options; Weights & Biases, Neptune and cloud platforms offer similar features.
MLflow organises work into experiments (a project or question) and runs (one training attempt). In Python the core calls are:
- mlflow.set_experiment("churn") — choose where runs are stored.
- mlflow.start_run() — open a run, usually in a with block.
- mlflow.log_param("max_depth", 6) — record an input setting.
- mlflow.log_metric("auc", 0.91) — record a result, optionally per step.
- mlflow.log_artifact("confusion.png") — attach a file such as a plot or report.
Calling mlflow.autolog() before training logs parameters, metrics and the model automatically for popular libraries such as scikit-learn, XGBoost and PyTorch Lightning. Start the UI with mlflow ui to sort and compare runs side by side. In a team, run a shared tracking server with a database and artifact store rather than local folders.
| Log this | Why |
|---|---|
| Parameters and config file | To repeat the run |
| Git commit hash and dataset version tag | To find the exact code and data |
| Metrics on train, validation and test | To compare runs and spot overfitting |
| Plots, feature list, model file | To inspect results without re-running |
| Run tags: owner, purpose | To search later |
3. Versioning data and models
Git is built for small text files. Datasets and model files are large and binary, so Git alone cannot track them well. DVC (Data Version Control) solves this: it keeps a small text pointer in Git and stores the real data in a remote such as S3, Azure Blob, Google Cloud Storage, SSH or a shared drive.
- dvc init — enable DVC inside a Git repository.
- dvc add data/raw.csv — DVC creates data/raw.csv.dvc (a pointer with a content hash) and adds the data file to .gitignore.
- git add data/raw.csv.dvc .gitignore then commit — the data version is now tied to a code commit.
- dvc remote add -d storage s3://bucket/path then dvc push — upload the data.
- dvc pull — on another machine, after git checkout, fetch the matching data.
DVC can also describe a pipeline in a dvc.yaml file: each stage lists its command, dependencies (code, data, parameters) and outputs. The command dvc repro reruns only the stages whose inputs changed, and dvc exp run manages experiments. Parameters live in params.yaml, and dvc metrics diff compares results between commits.
4. Packaging and serving models
A trained model object is not yet deployable. A package bundles the model with everything needed to run it: the serialised weights, the preprocessing, the expected inputs and outputs, and the software environment. MLflow Models does this with a model flavor (scikit-learn, PyTorch, XGBoost, or the generic python_function flavor), a signature and a requirements list, all stored with the model.
The signature declares the column names and types the model expects and returns. You create it with infer_signature(X_train, predictions) and pass it when logging the model. Serving can then reject a request with a missing column before it reaches the model.
| Format | Notes |
|---|---|
| pickle / joblib | Easy for Python, but loading a file from an untrusted source can execute code. Load only files you produced |
| ONNX | Framework-neutral; runs with ONNX Runtime in many languages |
| safetensors | Safe, fast weight storage for deep learning models |
| MLflow Model | Folder with model, signature, environment files |
Package the preprocessing with the model, for example in a scikit-learn Pipeline. Otherwise the serving code will reimplement it slightly differently and create skew.
5. CI/CD and pipelines for ML
In normal software, continuous integration (CI) runs tests on every change. In ML, a change can be new code, new data or a new model, so CI must test all three. A CI service such as GitHub Actions, GitLab CI or Jenkins runs the checks automatically on each pull request, and the merge is blocked if they fail.
| What | Example checks | Tools |
|---|---|---|
| Code | Unit tests for feature functions, linting, type checks | pytest, ruff, mypy |
| Data | Schema, types, value ranges, null rates, duplicates, row counts | Great Expectations, pandera, TensorFlow Data Validation |
| Model | Metric at least as good as the baseline, no slice falls below a floor, prediction on fixed examples, latency budget | pytest with MLflow, custom scripts |
| Package | Image builds, loads and predicts in a clean container; dependency scan | Docker, pip-audit, Trivy |
Three model tests catch many bugs cheaply:
- Quality gate — fail the build if the validation metric drops below the current champion by more than a set margin.
- Slice tests — check the metric for each important group (region, device, new vs old customers), because an average can hide a failing group.
- Behaviour tests — a prediction must not change when an irrelevant detail changes (invariance), and must move the right way when a key input changes (directional test).
6. Monitoring, drift and retraining
A deployed model is never finished. The world changes, so quality decays. Monitoring answers: is the service healthy, is the input still normal, are the outputs sensible and, when the truth arrives, are the predictions still right?
| Layer | Example signals | Speed |
|---|---|---|
| System | Latency p95, errors, CPU, memory, queue length | Seconds |
| Data quality | Missing values, new categories, out-of-range values, schema changes | Minutes to hours |
| Data and prediction drift | Input and output distributions versus a reference window | Hours to days |
| Model quality | Accuracy, AUC, error against real outcomes | Delayed by label arrival |
| Business | Conversion, loss rate, complaints | Days to weeks |
Distinguish the kinds of change:
- Data drift (covariate shift) — the inputs change, for example a new customer segment.
- Concept drift — the relationship between inputs and the target changes, for example fraud tactics evolve.
- Training-serving skew — production features are computed differently from training.
To detect drift, compare a recent window of production data with a reference, often the training data. Common tests are the Population Stability Index (PSI), whose widely used rule of thumb treats under 0.1 as stable and over 0.25 as a major shift, and the Kolmogorov-Smirnov test for numeric features. Open-source libraries such as Evidently produce drift and data-quality reports and tests. A drift alert means look, not automatically retrain: check whether quality really fell.
7. Governance, documentation and readiness
Once a model affects customers, money or safety, people will ask: what is it for, how was it built, how well does it work for whom, and who is responsible? Governance means having written, findable answers.
A model card (Mitchell et al., 2019) is a short document that travels with the model. Datasheets for datasets (Gebru et al.) do the same for data.
| Model card section | Content |
|---|---|
| Intended use | What it is for, and what it must not be used for |
| Data | Sources, time span, known gaps, personal data handling |
| Performance | Metrics overall and per slice, with the test set named |
| Limits and risks | Known failure modes, bias findings, fallback behaviour |
| Operations | Version, owner, monitoring, retraining plan, rollback |
Keep an audit trail: who trained, approved and promoted each version, on which data, and which version produced which prediction. Log predictions with the model version and a request ID, within your privacy rules. Use role-based access so that not everyone can change production aliases or read raw personal data.
Frameworks to know:
The full lesson continues in the interactive course on this page.
Sources and references
- MLOps: continuous delivery and automation pipelines in machine learning
- MLflow documentation (tracking, models, model registry)
- DVC documentation: data versioning and pipelines
- KServe documentation: serving models on Kubernetes
- Evidently documentation: data and model monitoring
- Feast documentation: the open source feature store
- AI Risk Management Framework (AI RMF 1.0)
- Regulation (EU) 2024/1689 (EU AI Act)
Educational course on engineering practice. Tools and versions change quickly: check the official documentation of each tool before you adopt it. The regulatory points (EU AI Act, GDPR, Moroccan Law 09-08) are an overview, not legal advice. Ask your legal or compliance team about your own obligations.
All courses · AI & Data - Free courses, certificate from $7 | AMAADOR ACADEMY