MLflow 3.0 for Gen AI Apps Deep Dive
Contact us to book this course
Learning Track
Generative AI
Delivery methods
On-Site, Virtual
Duration
1 day
This course introduces MLflow from the ground up and then explores the new features in MLflow 3.0 that support generative AI. Learners will gain a foundational understanding of MLflow’s components—tracking, models, registry, and projects—and then dive into how MLflow 3.0 adds observability, evaluation, and governance for LLMs and agents.
Learning Objectives
By the end of this course, learners will be able to:
- Explain what MLflow is and describe its core components
- Install and set up MLflow in a Databricks environment
- Use MLflow Tracking to log experiments, runs, and parameters
- Manage models and versions with MLflow Registry
- Understand what’s new in MLflow 3.0 for gen AI
- Trace gen AI executions and capture prompts, outputs, and costs
- Evaluate gen AI quality with automated and human feedback loops
- Apply governance and monitoring for production AI systems
Audience
- Beginners who are new to MLflow and want to understand its basics
- ML engineers and data scientists interested in gen AI applications
- MLOps professionals evaluating unified tooling for both ML and gen AI
Prerequisites
- Basic familiarity with machine learning or generative AI concepts
- Some experience with Python and Databricks is helpful but not required
Course outline
- What MLflow is and why it exists
- Common challenges in ML/AI development it solves
- The four core components: Tracking, Models, Registry, Projects
- Tracking: Logging experiments, parameters, metrics, artifacts
- Models: Saving and serving ML models
- Registry: Versioning and promoting models
- Projects: Packaging and reproducibility
- Installation and setup in Databricks
- Configuring tracking servers and Unity Catalog integration
- First demo: Log and register a simple model
- Unified platform for ML, deep learning, and gen AI
- New concepts: Tracing, evaluation, prompt registry
- Why 3.0 matters for LLMs and agents
- Automatic tracing of prompts, responses, tool calls, latency, and cost
- Supported libraries: OpenAI, LangChain, etc.
- Debugging complex agent workflows
- Versioning gen AI apps and prompts with LoggedModel
- Evaluation with built-in judges (safety, correctness, hallucination)
- Adding human-in-the-loop feedback
- Prompt registry for testing and optimization
- Dashboards and unified monitoring for ML + gen AI
- Integration with Unity Catalog for governance
- Ensuring compliance and auditability
