By Patrick Elder
Director, ECS Data ＆ AI CoE
Jason Turner
Senior Data Solutions Architect
Thomas Eldering
Data Solution Architect
Charise Arellano
Portfolio
Director
For the last several years, chief data officers (CDOs) across the federal government have been developing practices to leverage the use of data as a strategic asset. Their goal: to meet the standards laid out in the Federal Data Strategy (FDS), a 10-year vision for how the federal government will accelerate the use of data to deliver on critical missions, serve the public, and steward resources while protecting security, privacy, and confidentiality. The FDS is just one aspect of The Foundations for Evidence-Based Policymaking Act of 2018, also known as the Evidence Act, which requires federal agencies to develop evidence in support of policymaking.
FIGURE 1: The FDS 10-Year Vision
Government CDOs face several key challenges in executing this vision:
- Creating a common language for data within their organizations.
- Preparing to implement new artificial intelligence (AI)-enabled data tools.
-
Avoiding the pitfalls of centralized, “data lakes” architecture, such as the birth of unwieldy “data swamps” and slow updates.
The best way to meet the operational principles and best practices of the FDS and begin leveraging data to its full potential is to move away from traditional data architecture and toward a distributed model, also known as data mesh. Data mesh allows organizations to provide data products through a data catalog, share data through data services, and perform analysis for effective and timely data-driven decisions.
The Data Mesh Pyramid
Data governance and intentional maturity provide a backbone for improved visibility into data use and ultimately data products.
FIGURE 2: The Data Mesh Pyramid shows a simplified view of data mesh architecture that can help federal agencies achieve their missions and meet the standards of the FDS and the Evidence Act.
Facilitating Collaboration With a Data Inventory
Data mesh architecture is built on the foundation of an always-growing data inventory, an index of all the data within an organization. The data inventory:
- Automates data discovery (reducing manual inputs and helping ensure data quality in the collection of metadata)
- Enables data maturity improvements over time
- Promotes data governance best practices across an organization
The M-19-18 memo published in 2019 by the Office of Management and Budget (OMB) articulates a key practice of the FDS: “[Agencies shall] maintain an inventory of data assets with sufficient completeness, quality, and metadata to facilitate discovery and collaboration in support of answering key agency questions and meeting stakeholder needs.” A complete and robust data inventory provides a resource from which basic information about datasets can be curated into data products published to the data catalog, adding valuable details and context.
Data products are consumable products of a mission, including not only data but code and infrastructure, that serve a specific function and include information such as owner, access URL, security level, data type, geospatial extent, and more. Ultimately, this results in accessible, usable, secure, and quality data products becoming available in the data catalog. That, in turn, enables consumers to access, combine, and analyze data at mission speed, removing the obstacle of antiquated, manual processes.
Better, Timelier Data Products
Unlike traditional data architecture, data mesh turns local data producers into data product owners. The immediate benefits of this shift include:
- Timely and more accurate data products are managed by data product owners who understand the domain/functional area.
- Freed from having to maintain data architecture documentation and manage access to their source systems, data product owners can focus on delivering mission value.
- Faster AI evolution at the local level, creating a loop of better algorithms informing better data products.
Once an organization has cultivated a data catalog containing ready-to-use and easily accessible products, future duplication of data collection and transformation is eliminated. Also, as an organization’s technology matures, machines will look for data, examine the security configuration, and determine if the data is available in a format that the machine can use for its intended purpose. This type of automation, made possible by data mesh, saves time and increases data security and trustworthiness.
While an inventory of data assets is the beginning of the journey toward data driven activities, it alone is not enough. Data products represent a paradigm shift, with the result being data accessibility that enables discovery and collaboration at mission speed.
Meeting the Federal Data Strategy With ECS
There’s no doubt that moving away from traditional data architecture and toward data mesh improves an organization’s ability to achieve its mission. But, understanding the “why” behind the shift versus the “how” of making it happen are quite different. This requires organizations to change the way they’ve been thinking about data for decades, empowering domain experts to govern themselves to a greater degree and helping you capture the full value of your data.
Helping organizations make that shift is where ECS’ experts come in. We are committed to helping federal agencies meet the standards of the FDS and the Evidence Act. We will guide you through the data mesh pyramid, help you achieve data maturity and promote data governance best practices, and transform you into an AI-ready, machine-readable organization. With a data mesh architecture, federal agencies can achieve compliance with the Evidence Act, provide public access to data assets, and progress toward data-driven activities through addressable, discoverable, interoperable, understandable, secure, trustworthy, valuable, and natively accessible data products.
