dbt migration planning and auditing at phData from Coalesce 2023

"The whole goal here isn't that you're going to come up with a plan that is going to be perfect and totally get you through your migration... Let’s give these things some thought."

- Chris Johnson, Solution Architects at phData

Solution Architects from phData, Dakota Kelley and Chris Johnson, share strategies for migrating onto dbt. The speakers discuss why they’ve migrated to dbt, common issues they’ve encountered, and strategies to ensure success. They also discuss the importance of adopting a sound migration strategy and the need for a robust validation and auditing process.

Migrating onto dbt can optimize data pipelines and improve governance support, but it requires a thorough strategy

Chris and Dakota both highlight the benefits of migrating onto dbt. They emphasize the value of dbt's advanced security options, automation capabilities, and transparency, which can help increase team agility and improve data pipelines.

Chris explains, "Modern tools like dbt have been designed with a lot of governance features built in, a lot of automation capabilities…a lot of integrations with third-party platforms..." He elaborates that this helps increase team agility and allows them to implement data pipelines that are more performant.

Dakota adds, "We have done well over 40 migrations between Core [and] Cloud–small shops to extremely large enterprises, and we've seen that there are common patterns regardless of the size of your implementation, [or] the size of your organization…" However, they both stressed the importance of avoiding common pitfalls like treating modern tools like legacy tools, not changing your paradigm, and lifting and shifting with no refactoring.

Implementing and following Git best practices is crucial in ensuring successful migration

Both experts stress the importance of adopting and following Git best practices during the migration process. Effective use of Git can ensure the smooth transitioning of data pipelines and improve the reliability of the new system.

Dakota also emphasizes the importance of testing, stating, "The best way to test is to test very heavily at the front. Test on your source system. Make sure the source data is fresh. Make sure the shape of your source data is the way you expect it to be–the way you expect the data to be connected. Focus on that in those intermediate layers, maybe with some lighter testing...and then that final data model needs to be heavily tested as well so that it meets your business requirements. This allows you to get a good balance…[and] avoid alerting fatigue."

Incorporating transparency, efficient architecture, and FinOps into the dbt migration strategy can enhance performance and cost-effectiveness

Both presenters highlight the importance of transparency, efficient architecture, and financial operations in a dbt migration strategy. By focusing on these areas, organizations can enhance performance and cost-effectiveness.

Dakota urges, "Spend time centralizing your artifacts. dbt generates a massive amount of artifacts... you can centralize all of those artifacts with the dbt artifacts package, and now you have a way to visualize how your dbt project is running."

Chris adds, "Leverage proven architecture and structural approaches... figure out what you're going to use as a code repository. How are you going to do branching? What will your branching strategy be?"

Chris and Dakota’s key Insights on migrating onto dbt

• The main challenges of older legacy technologies include the lack of governance support, a lack of automation, and difficulty in adapting or customizing these older tools
• Modern tools like dbt have been designed with governance features built in, automation capabilities, and more transparency
• It's crucial to avoid treating modern tools like legacy tools and doing lift and shift with no refactor
• It's important to adopt and follow Git best practices, leverage proven architecture and structural approaches, and include validation and auditing as part of the migration strategy