5 Steps to Data Mesh Nirvana from Coalesce 2023

"You need to understand who is using what, what is the most popular...and also holistically monitor the health of all these things in a single place."

In this session, Pardu Gunnam, CEO and co-founder of Metaphor, discusses the five steps to achieving data mesh nirvana and the role of data catalogs in implementing data mesh. He touches upon the key principles of data mesh and shares insights from companies that successfully use it.

The importance of adopting a data mesh framework for data governance

Data mesh, first introduced by Zhamak Dehghani from Thoughtworks, is a method for managing data governance within organizations. Data mesh involves the division of data into various domains by treating data as a product, providing self-service infrastructure as a platform, and ensuring federated computational governance.

According to Pardu, "Data mesh is built on the principle of domain-driven data ownership," meaning data should be divided according to business groups, each with its own ownership. He also emphasizes treating data as a product and providing a self-service platform that empowers teams to operate the infrastructure.

Lastly, Pardu discusses the importance of federated computational governance, explaining, "It's also extremely important to make sure these domains are well governed."

The role of a data catalog in successfully implementing a data mesh

"Catalogs are not just a listing of information. They're also aggregating information from various sources, so they’re kind of like an aggregator and act as a source of truth."

Pardu highlights the crucial role of a data catalog in successfully implementing a data mesh. A data catalog serves as a source of truth for all metadata, aids in the search and discovery of data assets, enables collaboration, and provides visibility and monitoring of data assets.

Pardu explains, "The catalog needs to give enough collaborating capabilities for these people to operate the data mesh across the organization." This includes providing insights on data usage, data quality issues, changes in data, and more.

“You have all these personas engaging with your data products and domains through the catalog as a data governance team or as various owners or consumers. You need to understand who is using what, what is the most popular, what is the quality of it, and also holistically monitor the health of all these things in a single place."

5 steps to data mesh nirvana

1. Define ‌domains and subdomains
2. Federate responsibilities across the organization
3. Manage data products effectively
4. Understand and leverage your data products
5. Drive adoption of data mesh principles across the organization

The significance of driving the adoption of data mesh across an organization

"A good success measure is adoption across your organization. How do you drive adoption of data mesh, and how do you drive adoption of these principles across your organization successfully."

Driving the adoption of data mesh across an organization is also crucial. Pardu explains that the success of implementing the data mesh framework largely depends on how well it is adopted across an organization.

To drive the adoption of data mesh, personalize the experience for different personas, embed the catalog into natural workflows, and incentivize participation. Pardu elaborates, stating, “Engineers, Analysts, or users always alienate the governance process. They always think that governance is some top-down initiative which has nothing to do with how they work with the data. So how do you make it part of the process, and how do you make them contributors? By making governance a team sport.”

Pardu’s key points on data mesh

• Data mesh involves dividing an organization's data into various business groups or domains
• A data catalog can play a crucial role in enabling the principles of data mesh within an organization
• The five steps to data mesh nirvana include defining the domains and subdomains, federating responsibilities across the organization, managing data products effectively, understanding and leveraging data products, and driving the adoption of data mesh principles across the organization
• A data catalog can serve as a source of truth, providing important metadata and enabling collaboration
• The adoption of data mesh can be driven by personalizing the experience for different personas and embedding the data catalog into a natural workflow