Listen to article:
By Patrick Elder
Director, ECS Data & AI CoE
Jason Turner
Senior Data Solutions Architect
Thomas Eldering
Data Solution Architect
Charise Arellano
Portfolio Director
How the Data Inventory Provides a Firm Foundation for a Distributed, Sustainable, Federally Compliant Data Ecosystem
As government agencies begin to adopt AI/ML tools and prioritize data management, data sharing, and data reuse as part of their daily operations, it’s essential to build a framework that is flexible, scalable, and ready, today, to meet the needs of tomorrow.
In the first article of our series on meeting the Federal Data Strategy (FDS), ECS explored the idea of a data mesh, or distributed architecture for enterprise data management. Data mesh architecture provides the blueprint for building an AI-ready enterprise. The data inventory, an always-growing index of all the data within an organization, is the crucial foundation for that architecture and the reference point for data throughout the enterprise.
FIG. 1: The FDS 10-Year Vision
In this article, we will review some best practices and requirements for a data inventory, recommend a technical framework, and discuss how the inventory works to support the data mesh and, ultimately, helps federal organizations meet the standards of the FDS and the 2018 Evidence Act.
Data Mesh and Data Inventory
First, a review of our core concepts of data mesh and data inventory:
Data mesh represents the next step in the evolution of data architecture. It’s a maturation from data lakes — which tend to deteriorate into unusable “data swamps” over time — to a more flexible, comprehensive solution that focuses on data products and strategic data assets. It also helps organizations develop practices to secure, manage, share, and create a common language for data within their enterprise while preparing to implement new AI-enabled tools.
The Data Mesh Pyramid
Data governance and intentional maturity provide a backbone for improved visibility into data use and ultimately data products.
The data inventory is a high-level index of the data in the enterprise that is both machine and human readable. Like an index at the end of a textbook, it documents an essential set of facts about the data and acts as a foundational resource for referencing and finding data in the enterprise.
In a mature data mesh, the data inventory acts as an index of raw material that can be developed into data products, which are published in a robust data catalog. Users can access those data products to support mature data services and analytics, positioning agencies to make use of emerging and anticipated technologies like AI/ML, ontology integration, and semantic reasoning.
It’s worth taking a moment to make a clear distinction between the data inventory and data catalog, as it may not be immediately clear when a data consumer would want to seek out data from one versus the other. The key difference lies in the nature of the data being sought and who is seeking it. Data catalogs contain ready-to-use data products. For the vast majority of your enterprise’s consumers, the catalog is where they will go to retrieve the relevant data products they need to drive their services and/or analytics. The inventory, on the other hand, simply confirms that the data exists somewhere in the organization, even if there’s no current way to make use of it. Hence, it’s your enterprise’s domain experts who will use the data from the inventory to curate new data products for the benefit of the enterprise.
Meeting the Federal Data Strategy With ECS
For federal agencies, the transformation into AI-ready, machine-readable, and machine-reasoning organizations is at hand. ECS’ experts are committed to helping your organization evolve toward the enterprise model of the future.
REFERENCE: Dehghani, Zhamak. Data Mesh: Delivering Data-Driven Value at Scale. O’Reilly Media, 2022.
GLOSSARY
Data Mesh — A decentralized data architecture that organizes data by a specific business domain, provides more ownership to the producers of a given dataset, and allows them to set data governance policies, enabling self-service use across an organization.
Data Inventory — An index of all the data within an organization and the foundation for data mesh architecture.
Data Product — A self-contained data “container” that directly solves a business problem or is otherwise monetized; e.g., an application or tool that uses data to help businesses improve their decisions and processes.
Data Catalog — Uses metadata — data that describes or summarizes data — to create an informative and searchable inventory of all data products within an organization.
