Apps, dashboards, catalogs, metrics, syncs, streams, dumps, dictionaries, reports, alerts, pipelines... the "modern data stack" may take 30 minutes to spin up, but the work certainly doesn't end there. The greater availability of data and innovative tooling has created dozens of new ways for data teams to provide value, but it has also increased the complexity of managing it. The result may understandably be perceived by many stakeholders as a modern "black box" that does "data things". And even when we implement more precise terminology -- such as data products -- it can be a challenge to align across the company, much less market, monitor and govern the actual resources.
Yet, making modular data products is one of the key challenges to scaling a data platform. Tightly coupled systems can't scale, and teams can't separate responsibilities without product boundaries. In this talk, I'll present some of the challenges that our team at Immuta faced when scaling a "monolithic" platform: New team members struggling to navigate a complex dbt project; data consumers not being aware of the many tools and assets available to them; executive leaders not having a clear insight into the accomplishments and concrete roadmap of the team.
I will then detail the steps we took to re-organize our operations to create better walls around data products, from ingestion pipelines to dbt models to the consumption layer. I'll outline how we fleshed out our team's product catalog and set metadata expectations for each product, such as ownership, regulations, and quality. Attendees will walk away with some recommendations for creating their own data product strategy that will be useful no matter what scale they are operating on.
Join the chat in the #coalesce-Immuta channel (https://bit.ly/3umxQTM). If you’re not yet a member of dbt Community Slack, sign up at https://www.getdbt.com/community/join-the-community
Want to sponsor this event? contact us.