MLflow 3.0 for Gen AI Apps Deep Dive
(1 day)
Course Description
This course introduces MLflow from the ground up and then explores the new features in MLflow 3.0 that support generative AI. Learners will gain a foundational understanding of MLflow’s components—tracking, models, registry, and projects—and then dive into how MLflow 3.0 adds observability, evaluation, and governance for LLMs and agents.
Learning Objectives
By the end of this course, learners will be able to:
- Explain what MLflow is and describe its core components
- Install and set up MLflow in a Databricks environment
- Use MLflow Tracking to log experiments, runs, and parameters
- Manage models and versions with MLflow Registry
- Understand what’s new in MLflow 3.0 for gen AI
- Trace gen AI executions and capture prompts, outputs, and costs
- Evaluate gen AI quality with automated and human feedback loops
- Apply governance and monitoring for production AI systems
Audience
- Beginners who are new to MLflow and want to understand its basics
- ML engineers and data scientists interested in gen AI applications
- MLOps professionals evaluating unified tooling for both ML and gen AI
Prerequisites
- Basic familiarity with machine learning or generative AI concepts
- Some experience with Python and Databricks is helpful but not required
Course Outline
Module 1: Introduction to MLflow
- What MLflow is and why it exists
- Common challenges in ML/AI development it solves
- The four core components: Tracking, Models, Registry, Projects
Module 2: MLflow Fundamentals
- Tracking: Logging experiments, parameters, metrics, artifacts
- Models: Saving and serving ML models
- Registry: Versioning and promoting models
- Projects: Packaging and reproducibility
Module 3: Getting Started with MLflow
- Installation and setup in Databricks
- Configuring tracking servers and Unity Catalog integration
- First demo: Log and register a simple model
Module 4: What’s New in MLflow 3.0
- Unified platform for ML, deep learning, and gen AI
- New concepts: Tracing, evaluation, prompt registry
- Why 3.0 matters for LLMs and agents
Module 5: Tracing and Observability for Gen AI
- Automatic tracing of prompts, responses, tool calls, latency, and cost
- Supported libraries: OpenAI, LangChain, etc.
- Debugging complex agent workflows
Module 6: Versioning and Evaluation
- Versioning gen AI apps and prompts with LoggedModel
- Evaluation with built-in judges (safety, correctness, hallucination)
- Adding human-in-the-loop feedback
- Prompt registry for testing and optimization
Module 7: Monitoring and Governance
- Dashboards and unified monitoring for ML + gen AI
- Integration with Unity Catalog for governance
- Ensuring compliance and auditability
