Implementing dbt at scale: Practices and learnings from Prime Therapeutics' data team from Coalesce 2023

“We have had very very positive feedback from our clients…You know, they come to us and request something that used to be months before they could get a new report, or new data. We're sometimes getting those out in a week or days.”

- Jill Aakhus, Principal Analytics Engineer at Prime Therapeutics

Jill Aakhus, Principal Analytics Engineer at Prime Therapeutics, and Anubhav Pareek, Data Analytics Engineer at AICG discuss a case study involving a collaboration between AICG Consulting and Prime Therapeutics. They also focus on how dbt was instrumental in solving some of Prime Therapeutics’ data-related problems.

dbt as a tool to enhance data consistency and accelerate data reporting

dbt was instrumental in addressing the challenges faced by Prime Therapeutics in managing their vast data resources. The tool helped enhance the speed, accuracy, and consistency of data reporting, making it more useful for business decision-making.

Jill states, "dbt really helped us with our sources. It really helped with our transformations…and then really [gave] that ownership to our business." She also emphasizes that dbt's testing capabilities were key to enhancing data accuracy and consistency.

Anubhav adds, "[dbt] also lets you create documentation as you generate the model. The fact that you can generate documentation easily and quickly, once you update your model, also really helps the team."

The importance of a strong data culture and process adjustments

Anubhav and Jill highlight the importance of process adjustments and governance boards in fostering a strong data culture. Prime Therapeutics had to redefine its processes and establish governance boards to ensure data consistency and accuracy.

Jill states, "We really had to go back and say we got to start over. What are our processes? How are we going to fix this? What are we going to do?" She elaborates that the governance boards were comprised of higher-level management and subject matter experts who decided what needed to be converted and how.

Anubhav also points out that dbt works best when there is a strong data culture foundation. He states, “A strong data foundation, a data culture foundation, would enhance the experience that you can get from your dbt implementation."

The role of dbt in addressing industry-specific data challenges

Prime Therapeutics used dbt to integrate pharmacy data with medical data, resulting in more consistent and faster reporting.

Jill explains, "We are trying to integrate the pharmacy data with the medical data, which not all pharmacy companies can do. But because of that, we had a lot of data coming from different places." She highlights that dbt was instrumental in automating the application of security policies, thereby saving time and ensuring data protection.

Anubhav adds, "Given the industry nature of the data…it was really nice and helpful that when the data loads, those policies were automatically implemented on the tables without having to go in manually..."

Insights surfaced

• Prime Therapeutics faced challenges in data management due to inconsistencies, lack of reporting, and siloed operations.
• The company had to rebuild its data processes, create governance boards, and standardize naming conventions to overcome these challenges
• dbt was instrumental in solving these challenges by helping with data sources, automating transformations, and giving ownership to the business
• dbt also helped in parallel execution of data loads and automated application of security policies on data loads
• The implementation of dbt led to more consistent reporting, faster work completion, and positive feedback from clients