Scaling efficient data operations in large organizations with dbt from Coalesce 2023

"If there's only one takeaway from today, just think about simplifying your branching model, and it will take you a very, very long way."

Joris Van den Borre, founder and CEO of Tropos.io, discusses scaling dbt within large organizations. He shares insights on how to manage complex dbt projects, standardize practices, and automate processes to improve efficiency and maintain control as teams scale.

Simplifying dbt practices for scalability is essential for large organizations

Joris acknowledges the complexity that can come with scaling dbt practices within large organizations, suggesting that simplification is key. He proposes that a crucial first step is to standardize the branching model to avoid issues with increased scale.

"The better we do our work, the more delivery teams will start popping up…we want to make sure that every delivery team can be treated in the same way," he explains. "If there's only one takeaway from today, just think about simplifying your branching model, and it will take you a very, very long way."

Prioritizing governance over technology in dbt rollout

Joris highlights the importance of governance in the process of dbt rollout. He emphasizes that although the technology works, what matters more is how well the team is organized.

He strongly advocates for creating processes, setting blueprints, and constantly capturing feedback as part of the governance process, emphasizing, "Capturing the feedback is actually the most important part of it all."

Joris explains that the central capability team not only sets up processes, but also actively seeks input from various teams, making adjustments based on their unique needs. This iterative feedback loop ensures that dbt practices evolve with the organization's changing requirements.

The importance of automating and standardizing dbt

Joris emphasizes the value of automating and standardizing dbt to ensure a smooth operation. By establishing a simplified branching model, automating workflows, and implementing standardized processes, organizations can ensure consistency across projects and teams. This approach is particularly beneficial when dealing with multiple delivery teams.

Joris suggests using tools like Terraform to create dbt projects and stresses the importance of having standardized practices. "After a while, you can start coming up with these guardrails, and if you have a guardrail, you can actually program that in some sort of standardized workflow format," he explains. He also suggests using dbt's new Cloud CLI to integrate both worlds of dbt, which could help standardize practices and increase efficiency.

Joris’ key insights

• Standardizing practices and automating processes can help manage complex dbt projects and improve efficiency
• A simplified branching model in dbt can help in scaling up to multiple teams
• Automating the creation of dbt projects can help standardize the way a project should look like
• Integrating dbt Cloud CLI can help manage dbt projects better
• Being part of the dbt community can provide valuable support and insights for dbt projects