Testing & Monitoring Machine Learning Models

The Complete Course

Cover Image


Course Overview

Learn how to test & monitor production machine learning models.

What is model testing?

You’ve taken your model from a Jupyter notebook and rewritten it in your production system. Are you sure there weren’t any mistakes when you moved from the research environment to the production system? How can you control the risk before your deployment? ML-specific unit, integration and differential tests can help you to minimize the risk.

What is model monitoring?

You’ve deployed your model to production. OK now what? Is it working as you expect? How do you know? By monitoring models, we can check for unexpected changes in:

  • Incoming data
  • Model quality
  • System operations

When we think about data science, we think about how to build machine learning models, which algorithm will be more predictive, how to engineer our features and which variables to use to make the models more accurate. However, how we are going to actually test & monitor these models in a production system is often neglected. Only when we can effectively monitor our production models can we determine if they are performing as we expect.

Why take this course?

This is the first and only online course where you can learn how to test & monitor machine learning models. The course is comprehensive, and yet easy to follow. Throughout this course you will learn all the steps and techniques required to effectively test & monitor machine learning models professionally.

In this course, you will have at your fingertips the sequence of steps that you need to follow to test & monitor a machine learning model, plus a project template with full code, that you can adapt to your own models




Course Introduction
Course Curriculum
Course Requirements
Approaching This Course
Complete Course Notes
All Course Slides
Course Scenario and Model Lifecycle

Deploying a Model to Production
Course Scenario - Predicting House Prices
Setup A - Python Install (Do not skip)
Setup B: Git Install (advanced users can skip)
Course Github Repo & Data
Download Data and Github Link
Setup C - Install Jupyter Notebook (Advanced users can skip)
Setup D - Install Initial Dependencies (advanced users can skip)
Introduction to the Dataset and Model Pipeline
The ML System Lifecycle
Testing Concepts for ML Systems

Testing Focus In This Course
Why Test
Testing ML Systems (Important)
Testing Theory
Testing Concepts - Exercise 1 Instructions
Exercise 1 Solution
Exercise 2 Instructions
Exercise 2 Solution
Exercise 3 Instructions
Exercise 3 Solution
Exercise 4 Instructions
Exercise 4 Solution
Unit Testing ML Systems

Python Code Conventions
Intro to pytest
Download Dataset from Kaggle
Using Tox
Codebase Overview
Preprocessing & Feature Engineering Theory
Unit Testing Preprocessing & Feature Engineering Code
git hygiene
Config Tests Theory
Unit Testing Config Code
Testing Input Data Theory
Unit Testing Input Data Code
Testing Model Quality Theory
Unit Testing Model Quality Code
Repo Tooling
Wrap Up
Docker Refresher

Section Overview
Docker Recap
Why Use Docker
Introduction to Docker Compose
Docker & Docker Compose Installation
[Windows Only] Docker Setup
Docker Exercise Instructions
Docker Exercise Solution
Integration Testing ML Systems

API Conceptual Overview
Integration Testing Code Base
Using the API Part 1
Windows Specific Docker Setup
Using the API Part 2
Integration Tests Theory
Integration Test Code
Benchmark Tests Theory
Differential Testing

Differential Testing Theory
Differential Testing Implementation
Shadow Mode

Shadow Mode Overview
Shadow Mode Theory
Testing Models In Production
Tests in Shadow Deployments
Shadow Mode DB Code Overview
Shadow Mode Setup Tests
Asynchronous Implementation
Populate DB with Shadow Predictions
Jupyter Demo - Setup
Jupyter Demo - Tests in Shadow Mode
Monitoring Metrics with Prometheus

Why Monitor ML Models
Monitoring Theory
Metrics for ML Systems
Prometheus & Grafana Overview
Windows Setup
Basic Prometheus Setup
Adding Prometheus Metrics
Setup Grafana
Infrastructure Level Metrics
Adding Metrics Monitoring to Our Example Project
Creating an ML System Grafana Dashboard
Monitoring Logs with Kibana

Monitoring Logs for ML
Elasticstack overview
Kibana Exercise
Integrating Kibana into our Example Project
Setting Up a Kibana Dashboard for Model Inputs

Appendix A: Python Basics

String Manipulation Example
Dot Product - Numpy Comparison
Numpy arrays vs. Python Lists
pytest example (WIP)


This is the second course I am completing from Christopher and Soledad. They have the most comprehensive courses on deploying machine learning in an industrial settings that I have taken. As a data scientist, I found the material I learned from these courses directly applicable to my daily work, and many of the tools they introduce either something I use on a daily basis or something I am going to look into using on a daily basis. Perhaps the most valuable course I have done ... (I have completed 30++ courses)

- Anders Albert


Still Not Sure If The Course Is For You?

Still not sure if this is the right course for you?

Here are some rough guidelines

Never written a line of code before: This course is unsuitable

Never written a line of Python before: This course is unsuitable

Never trained a machine learning model before: This course is unsuitable. Ideally, you have already built a few machine learning models, either at work, or for competitions or as a hobby.

Never used docker before: The second part of the course will be very challenging. You need to be ready to read up on lecture notes & references.

Have only ever operated in the research environment: This course will be challenging, but if you are ready to read up on some of the concepts we will show you, the course will offer you a great deal of value.

Have a little experience writing production code: There may be some unfamiliar tools which we will show you, but generally you should get a lot from the course.

Non-technical: You may get a lot from just the theory lectures, so that you get a feel for the challenges of ML testing & monitoring, as well as the lifecycle of ML models. The rest of the course will be a stretch.

Chris Samiullah

ML Engineer

Experienced software engineer and techlead.



You can always reach me at this email address: chris@christophergs.com