Using and customizing Fivetran dbt packages from Coalesce 2023

“The way we make packages—we kind of bank on this level of standardization, but with that…there may be obstacles at the individual scale. And that’s who’s actually using the packages: individuals. But they are surmountable, I promise.”

Jamie Rodriguez, Senior Analytics Engineer at Fivetran, explains how to customize standard Fivetran dbt packages for your individual needs. He addresses common obstacles users may face using these packages “straight out of the gate,” and he presents several examples of how these packages have been successfully customized.

Overcoming the obstacles of opinionated models in dbt packages

dbt packages have the inherent challenge of opinionated models, which, while designed to cater to the majority, may not suit certain unique or uncommon cases. Jamie explains how you can modify these models to better fit individual needs.

He explains, "...if you fall into an uncommon or unique use case, you might disagree with some of these assumptions that are baked into the code." Jamie demonstrates how to override these models by disabling the package's version of the model and creating a new one with the same name within one's project. This, he maintains, will allow the package to seamlessly point to the user's version.

Using the example of a Shopify package, Jamie demonstrates that even when opinionated logic was overridden, it was still possible for the package to function correctly. "Your stakeholder will have the metrics that they want.”

Handling nuanced schemas in dbt packages

Jamie covers the handling of nuanced schemas in dbt packages. Even with standardized schemas, there can be room for customization which can create mismatches between fields and tables that the package expects and what is actually present.

He acknowledges, "These largely standardized schemas can still have these subtleties that make designing and adopting the package a little trickier than one might think." Jamie introduces a solution in the form of “fivetran_utils.union_data,” a macro that unions multiple connectors' source tables together, allowing the package to run on them all at once.

Customizing fields in dbt packages

The last challenge Jamie addresses is the inclusion of custom, but necessary, fields in dbt packages that aren't typically included in package models by default. Jamie introduces the concept of “pass-through” columns to tackle this issue.

He elaborates, "Pass-through columns...are fields that are not included in the package models by default but that you can configure…” He explains that these columns could be simple or complex, depending on the user's requirements.

Using the example of a NetSuite package, Jamie demonstrates how the Fivetran analytics team used 55 different pass-through columns to recreate and validate their models. He stresses, "...these packages are truly living, breathing projects and absolutely not set in stone."

Insights surfaced

• Fivetran dbt packages are standalone projects that can be installed and referenced in your own project. They often tackle a problem area that's shared by many people
• While these packages aim to satisfy the standard use case, there can be obstacles at the individual scale. These include opinionated models, nuance schemas, and having very custom but necessary fields
• You can override the package in a targeted manner to change certain aspects or use “union_data” to union multiple connectors source tables together
• The packages are continually evolving and improving based on community feedback
• Fivetran’s team used 55 different pass-through columns to completely recreate and validate their models, demonstrating the flexibility and adaptability of their packages