Transforming loan warehousing with scalable data pipelines: How dv01 uses dbt Cloud and BigQuery to empower financial analysts from Coalesce 2023

"You can build a team that creates business value by marrying engineering and subject matter expertise."

David Maguire, Data Engineer at dv01, explains how to build cross-functional data teams that create business value by combining engineering and subject matter expertise. He uses the example of his own company, dv01, where they used SQL as a unifying language and dbt for flexible orchestration and transparent data pipelines. He also discusses the process of training and mentoring analysts to become analytics engineers and the importance of a test-driven culture.

Creating cross-functional data teams, enhanced by subject matter expertise, can significantly increase business value

David emphasizes the importance of building cross-functional data teams to create business value. By combining engineering expertise with deep domain knowledge, these teams can efficiently tackle complex business problems.

David explains, "We want to build a team heavy on subject matter expertise, but with enough engineering to be self-sufficient." He advocates for the use of SQL as a unifying language, "because it's a single language that can be used by both analysts and engineers."

He elaborates on the team composition, explaining that it’s comprised of data engineers, analytics engineers, and financial analysts–each playing a distinct yet overlapping role. "The primary responsibility is to work end-to-end within the dbt pipeline," he says, outlining the role of analytics engineers.

SQL and dbt were chosen as core technologies for their capacity to unify workflows and maintain data pipeline transparency

David highlights SQL and dbt as core technologies in building cross-functional data teams. SQL was chosen for its ability to break down silos, thus enabling effective collaboration, while dbt was favored for its ability to make SQL organized, modular, and testable.

David elaborates, "SQL breaks down silos. This was a big key to success with this team." He adds that dbt “makes SQL organized, modular, and testable... it allows flexible orchestration and transparent data pipelines."

dbt's ability to make intermediate models transparent was particularly useful, as it allowed financial analysts to quickly identify issues, thus making the team more effective.

The implementation of a test-driven culture shifted the team from reactive to proactive

"Instituting a culture of test-driven development was crucial for the team's success."

David underlines the benefits of implementing a test-driven culture within the team. Test-driven development, a concept familiar to engineers, was pushed across the team to ensure the robustness of their data pipelines and reporting suite.

"We wanted to promote those ideas across the team. The testing starts with the analysts," David explains. He emphasizes that involving the analysts in the testing workflow was crucial to ensuring the pipelines and reporting suite's robustness.

This testing culture allowed the team to shift from a reactive to a proactive approach. The implementation of a data quality report card also provided a comprehensive overview of the state of their data, aiding in quick problem diagnosis and resolution.

Insights surfaced

• SQL can serve as a unifying language that can be used by both analysts and engineers, breaking down silos and facilitating collaboration
• dbt makes SQL organized, modular, and testable, and allows for transparent data pipelines
• The team structure involved three key roles: data engineers, analytics engineers, and financial analysts. Each role had a different focus, but there was some overlap, allowing for flexibility and adaptability
• Training and mentoring existing analysts to become analytics engineers was a key part of the process. This involved SQL training, personalized learning plans, and pair programming
• Instituting a culture of test-driven development was crucial for the team's success. This involved defining core competencies, involving analysts in testing, and implementing a data quality report card