Strategic deployment of dbt Cloud with Databricks and Delta from Coalesce 2023

"We think we can actually do quite a bit with a relatively simple, powerful strategy, and it really is defined by how powerful dbt can be."

Chris Davis, Senior Staff Engineer at Udemy, shares Udemy’s approach to improving developer experience using Databricks’ Delta Lake and dbt Cloud. Davis discusses the challenges Udemy’s team faced, the strategies they implemented, and the improvements they saw.

The importance of a remote-first, unified software development lifecycle

A unified software development lifecycle is crucial for improving developer experience.

Previously, Udemy faced challenges with table growth, unclear ownership of data assets, and varied development environments. Chris highlights using dbt Cloud and a remote-first strategy as a key factor in creating a unified approach. The main advantage is that it provides better governance and easy onboarding for developers.

Chris explains, "Being remote is a big contrast from what we're currently doing in the rest of our stack, and we hope to use that as sort of a magnet to pull in more developers for us." He also explains how they’ve integrated coding, querying, analysis, exploration, and a PR opening into one seamless remote experience, which made a significant difference to their workflow.

The role of dbt CLI and Delta in improving developer experience

dbt CLI and Delta Lake are crucial tools in improving the developer experience. According to Chris, the dbt CLI was highly valuable for a modern data stack and can help increase the number of nodes in a project, resulting in a more efficient and streamlined workflow.

He emphasizes, "High leverage…it's really valuable for a modern data stack.” On Delta Lake, Davis cites the simplicity, unification, and quick access that it provides to data as significant advantages.

The use of the Kimble approach in data modeling for improved developer experience

Chris highlights the use of the Kimble approach in data modeling as a key factor in improving developer experience. The Kimble approach is a solid base for data modeling and also helps with cross-functional data partnerships within the team.

Chris explains, "Really simple, modified Kimble. We think that that's a super useful approach and data modeling here to stay." He explains that this approach enables the team to work with cross-functional data partners more effectively, providing a more streamlined and efficient developer experience.

Chris’ key takeaways on Databricks, Delta, and dbt Cloud

• Udemy faced challenges with table growth, unclear ownership of data assets, and varied development environments
• dbt Cloud and Databricks’ Delta Lake were key tools in improving the developer experience, offering a unified and remote-first development environment
• The company uses a modified Kimble approach for data modeling and implements a simple production configuration for its jobs
• The future roadmap includes scaling up the number of users and assets, improving CI/CD, and leveraging more features of dbt Cloud