Using R on Databricks
(1/2 day)
Course Description
This course shows R developers how to run, scale, and automate R workloads in Databricks. Learners will use sparklyr for distributed data processing, connect from notebooks or IDEs, and explore tools like brickster to streamline development.
Learning Objectives
- Understand how R integrates with Databricks
- Run R code in notebooks and IDEs
- Scale analysis with sparklyr on Databricks clusters
- Automate R jobs and workflows
- Apply best practices for performance and migration
Audience
- R developers wanting to scale analysis
- Data scientists bringing R into Databricks workflows
Prerequisites
- Basic R knowledge
- No Spark or Databricks experience needed
Course Outline
Module 1: R on Databricks Overview
- R support in Databricks
- sparklyr (preferred) vs. SparkR (deprecated)
- New tools for R users: brickster, connectors, docs
Module 2: Running R in Notebooks and IDEs
- Connecting to Databricks clusters
- Distributed transforms with familiar dplyr syntax
- Handling large datasets efficiently
Module 3: Developer Tools and Integration
- Managing clusters and jobs with brickster
- Importing R code, using packages, and version control
- MLflow tracking with R
Module 4: Automating Jobs and Workflows
- Scheduling R notebooks as jobs
- Orchestrating workflows with R + Python/SQL
- Monitoring and logging R jobs
Module 5: Best Practices
- Migrating from SparkR to sparklyr
- Performance tuning tips
- Limitations and workarounds
