Unlocking the full potential of dbt projects: Insights from Avenue One and Atlan on active metadata and data mesh strategies from Coalesce 2023

"I really do see something like a data catalog becoming the center of the stack in an AI-first world."

- Sean Rober

Austin Kronz, Director of Data Strategy for Atlan, and Sean Rober, Head of Data for Avenue One, team up to discuss the challenges faced by data teams and the role of a data catalog in addressing them. They share their experiences using Atlan, a modern data catalog, and how it’s helped their respective organizations. They also touch upon future possibilities and advancements in data management and AI.

Metadata management tools need to evolve from being passive to active

Austin highlights the limitations of existing metadata management tools, pointing out that they were often closed off, siloed, or in need of manual intervention. He claims that these tools needed to evolve to provide more automated and connected experiences to meet the growing demands of data users.

"Whenever we went and looked at the market, tools tend to be closed off. If we wanted to extend the platform, it was kind of like building around the tool instead of building with the tool," says Austin. He also comments on the disjointed approach of many tools, saying, "You might have a data dictionary that the data engineering team is using, and somewhere in the business is an Excel file or a Google sheet that has business metrics. And never the two shall meet."

To address these issues, Austin proposes a vision for the future of metadata management tools with a more connected experience, an extensible platform, and automation. He questions, "If I have a use case that is beyond what the technology has out of the box, how can I just programmatically include that into the tool?"

Self-service analytics is a vital part of modern data strategy

Sean emphasizes the importance of self-service analytics in their data operations. By using a modern data stack, Avenue One aimed to democratize data and make it more accessible to users, reducing the need for a separate business intelligence team.

"We've tried to self-serve this BI to the greatest extent possible and really leverage this modern data stack concept with the concept of self-service analytics," says Sean.

He also mentions the utilization of Atlan, a data catalog that helped them manage their vast amount of data. Sean adds, "Assigning data stewards was a huge piece of beginning your governance journey." The data catalog helped them monitor data sources and trace back errors, improving their overall data management strategy.

The role of data catalogs in an AI-first world

In the future, data catalogs could play a central role in the data stack, particularly in an AI-first world. Both Austin and Sean see the potential of data catalogs in bridging the gap between data producers and consumers, making data interpretation easier and more efficient.

Sean speculates that data catalogs could become vital in an AI-first world: "I really do see something like a data catalog becoming the center of the stack in an AI-first world." He adds that data catalogs mesh well with semantic layers and how AI interprets data.

Austin agrees, noting, "Without something like a catalog, it's challenging to have one single place where multiple personas can live." He stresses the importance of having a single place where consumers and producers could meet and see data catalogs as a potential solution.

Key insights from Austin and Sean

• Data teams often face challenges such as managing vast amounts of data, dealing with broken dashboards, and understanding complex metrics
• Atlan is a modern data catalog that aims to provide a solution for these challenges. It offers features like a connected experience, extensibility, and automation
• Avenue One found Atlan helpful in assigning data stewards, tracing back column-level lineage, tying glossary terms to data assets, and creating support tickets directly
• The future of data management could see more automation and AI involvement, and tools like Atlan could become the center of the stack in an AI-first world