Rapid deployment of data products using dbt at nib Health from Coalesce 2023

"We want to take these learnings and apply it to all the things that we do."

Pip Sidaway, Data Product Manager at nib, explains how to use dbt for rapid deployment of a data product. She shares her experiences using dbt, the importance of automation, the use of macros, and the benefits of detailed upfront work in the development process.

Penguins can inspire efficient data management

Pip draws a unique parallel between the journey of migrating from old data systems to the breeding migration of emperor penguins. She likens the challenges of the data migration journey to the difficulties the penguins face while traversing challenging terrains and braving harsh weather conditions.

"We have a long journey ahead of us," Pip says. "Penguins are not really built for walking. They waddle. Some fall down. Not all of them might make it, and I kind of feel like this is our data journey as well." She explains that like penguins, not all data migration efforts are successful on the first try. It often requires continuous attempts and improvements.

A modern data stack can improve efficiency and accelerate data processing

nib Health Funds moved to a modern data stack and saw significant improvements in their data processing speeds. They also experienced noteworthy gains using a combination of Snowflake and dbt.

"There was a process where a job would take two days to run, and oftentimes, it would fail and we'd have to start it again...but in dbt, [it took] 20 minutes," Pip shares. This significant reduction in data processing time allowed for more efficient operations and quicker data availability.

Dedication to upfront work in data migration can prove beneficial

"The biggest thing we learned is how important it is to do all the upfront work…"

Pip stresses the importance of thorough, upfront work in data migration projects. She explains that having all the definitions and calculations at the start of a project can streamline the process and make it easier.

"…having those definitions, calculations, and things upfront just made it so much easier," she says. She also highlights the importance of running tests throughout the process to ensure accuracy and data quality.

Pip's future vision for data management

Pip shares her future vision for data management at nib, which includes setting up more distinct data products and moving toward event-driven processing. She also hopes to implement more testing and create a product dashboard that shows how many users are using a data set, when it was last run, and data quality metrics.

"My ultimate goal is a product dashboard–so having a dashboard that shows you how many users are using my data set, how much did it cost to produce, when was it last run, and data quality metrics," Pip says. Her goal for the future of data management at nib is to make the process as efficient and user-friendly as possible.

Pip’s key insights

• dbt and Snowflake can be a powerful combination for data wrangling, reducing process time significantly
• Having detailed definitions and calculations upfront can simplify the development process and increase efficiency
• Automating the process of applying data security policies and tagging at a column level can save time and ensure consistency
• Regular testing throughout the development process can help identify and address issues early on