Page Comparison

Table of Contents

...

Data Governance problems

Without effective data governance, data inconsistencies in different systems across an organization might not get resolved. For example, customer names may be listed differently in sales, logistics and customer service systems. That could complicate data integration efforts and create data integrity issues that affect the accuracy of business intelligence (BI), enterprise reporting and analytics applications. In addition, data errors might not be identified and fixed, further affecting BI and analytics accuracy.

Data Governance programs

compliance with data privacy and protection laws, such as the European Union's GDPR and the California Consumer Privacy Act
(CCPA). An enterprise data governance program typically includes the development of
common data definitions and standard data formats that are applied in all business systems,
boosting data consistency for both business and compliance uses.

<<key < FACTUR3DT.IO - measure Data Governance Value, Impacts in the organization against defined OKRs, KPIs in the Data Governance Program

<<key < key benefits possible include

setting, implementating and enforcing well defined data governance policies

data access controls and management

data quality management against internal standards and external regulations

compliance with data management policies and regulations

easy data availability for users and services

automation of data management tasks

data valuation based on usage vs objectives

compliance with data usage controls - PII, GDPR, CCPA etc industry data controls PCI HIPAA etc

...

Data Governance Components

mission statement for the program, its goals and how its success will be measured, as well as decision-making responsibilities and accountability for the various functions that will be part of the program that is published

Data Governance Tools

data governance software can be used to automate aspects of managing a governance program. While data governance tools aren't a mandatory framework component, they support program and workflow management, collaboration, development of governance policies, process documentation, the creation of data catalogs and other functions. They can also be used in conjunction with data quality, metadata management and master data management (MDM) tools.

Data Governance

...

Program Steps

steps to take, including the following to-do items:

identify data assets and existing informal governance processes;
increase the data literacy and skills of end users; and
decide how to measure the success of a governance program.
identify and update existing data management policies to meet program objectives, compliance and related regulations
identify and create the appropriate metrics, data controls and audit procedures for the policies
setup the data governance team and responsibilities including data stewards
define the tools needed for each stakeholder to effectively manage their governance responsibilities
setup data classifications to support data policies and access controls ( AATR etc )
define common business meta data models and glossaries that map to the industry, the business and related standards
build and maintain the enterprise data catalog with all internal and external sources, related controls for quality & currency,
define all clients for the catalog resources and their related responsibilities and access to data catalog sources

How to Build a Data Catalog - tech target article

Data Catalog: A Comprehensive Guide

Smith, Anne Marie, Ph.D. (2022). "How to build a data catalog: 10 key steps."

Table of Contents

Introduction
- Definition and Importance
Why Data Catalogs are Essential
10 Steps to Building a Data Catalog
- 3.1. Document Metadata Management Value
- 3.2. Identify Data Stewardship Uses
- 3.3. Design a Subject Area Model
- 3.4. Build a Data Glossary
- 3.5. Build a Data Dictionary
- 3.6. Discover Metadata from Sources
- 3.7. Profile the Data
- 3.8. Identify Relationships Among Data Sources
- 3.9. Capture Data Lineage
- 3.10. Organize the Catalog for Users
Best Practices for Building a Data Catalog
Conclusion
References

Introduction

A data catalog serves as a centralized reference tool enabling various users to explore, understand, and utilize data sets effectively. It collects metadata from diverse sources to create a searchable inventory, enhancing metadata management across an enterprise.

Why Data Catalogs are Essential

The primary goal of a data catalog is to overcome the challenges posed by data sprawl across different stores, making it hard for users to find relevant data. By offering a unified view and built-in search capabilities, data catalogs ensure operational and analytics initiatives are more effective, supporting data-driven decision-making.10

Steps to Building a Data Catalog

3.1. Document Metadata Management Value

Highlight the benefits of metadata management to data governance, emphasizing the improved data quality and operational effectiveness it brings.

3.2. Identify Data Stewardship Uses

Distinguish between data catalogs, business glossaries, and data dictionaries to utilize each for effective metadata management.

3.3. Design a Subject Area Model (SAM)

Develop a SAM based on business uses of data, indicating data's location beyond system constraints, crucial for the data catalog's structure.

3.4. Build a Data Glossary

Create an enterprise-wide business glossary in collaboration with business data stewards, providing a foundational knowledge base for data catalog content.

3.5. Build a Data Dictionary

Compile comprehensive descriptions and mappings of data entities to guide metadata integration into the data catalog.

3.6. Discover Metadata from Sources

Identify and record metadata sources across the organization's databases and repositories for inclusion in the data catalog.

3.7. Profile the Data

Generate informative data profiles to aid users in understanding catalog metadata, focusing on both technical and business metadata aspects.

3.8. Identify Relationships Among Data Sources

Uncover and document data relationships across different systems to facilitate comprehensive data understanding and usage.

3.9. Capture Data Lineage

Utilize ETL tools for data lineage documentation, tracking data origins and flows for error tracing and user understanding.

3.10. Organize the Catalog for Users

Design the data catalog with a user-centric approach, ensuring accessibility and ease of use for data consumers.

Best Practices for Building a Data Catalog

Ensure data security and privacy through user permissions and sensitive data tagging.
Foster collaboration with user interaction features like rating, commenting, and chatting.
Develop user training programs for effective data catalog utilization.
Establish a maintenance process to keep the catalog current with evolving data assets and business needs.

Conclusion

A well-planned and implemented data catalog is integral to modern data management, offering invaluable support to data governance, metadata management, and user empowerment. It paves the way for a more informed and efficient operational and analytical environment.

References

Smith, Anne Marie, Ph.D. (2022). "How to build a data catalog: 10 key steps."

...

My career is managing data anywhere: goalie, janitor, whisperer, therapist

m Data Mgt Concepts

Role keys- alignment on IT architecture, plans, priorities, business partnerships internal & external

SDP - virtual teams > discovery > assessment > plan > design > test > train > rollout > support > outcomes & impacts feedback

From IT EA, EDM, Digital Transformation programs, create related program for EDG that is aligned, integrated w BUs, clients, vendor services to meet goals on EDM quality & services OKRs, DT, Compliance ( internal and external audits )

EDM - Enterprise Data Mgt - 4 R keys - data & services are: RIGHT, RELIABLE, RESPONSIVE, REACTIVE across the enterprise, clients and vendor services

DLT Architecture part of EA services tower w other IT lines > governance support on DLT, data > partners with innovation teams

EDM app services, infrastructure, support tools

EDM - Commercial Tools, Solutions

Compare IBM, Oracle, Collibra, Microsoft, < see Gartner Magic Quadrant for Data Governance

Informatica, Collibra for MDM or ?

Couchbase for distributed NoSQL w JDBC tools, reporting

...

Tools - EA, SA, Archimate, Plantuml, MDM repo ( NOT commercial ones but maybe integrated w Oracle )

App Mgr >

Data Security - SailPoint, Guardium,

EDM - Open-Source Tools, Solutions

Pulsar & Solace on global events services

DLT w Firefly, Fabric, Besu

MySQL, Postgres, Oracle, Snowflake, MongoDB, Aws Aurora

Messaging > Kafka, MQ, ActiveMQ, Artemis ?