m Data Mgt Concepts
- jim mason
- jimm1
Key Points
- Goals - get the right data, with the right quality in the right place for analytics and other applications
References
| Reference_description_with_linked_URLs____________________________ | Notes______________________________________________________________ |
|---|---|
https://drive.google.com/open?id=1YemwxzfSGxJZ1_-IYmSzK7jC8srjGbpJ | data-mgt-models-4-use-cases-ebook-Paxata Ebook - The 4 Data Prep Styles - Picking the Right One.pdf |
Manager Data Analytics JOBD shows the domains covered pdf | well defined data mgt services role ** |
Key Concepts
Potential Value Opportunities
How is my data protected? Data Goalie
Where the hell is my data? Data Finder
an search, inventory and identity problem
automated data catalogs of sources
automated subscribeable data events
Who accessed my data? Data Historian
Who should see my data?
Who can see my data?
Who did see my data?
SAM - Smart Access Mgr for data
What the hell happened to my data? Data Doctor
a GRC issue
Data Message Producers, Listeners, Consumers
a Data Message is an instance of Data Message Class
a Data Message instance has a guid id
a Data Message can composed from other Data Message instances
a Data Message has a life cycle: created > < repeat > > available > consumed > processed
a Data Message has a Data Message History
a Data Message has a Data Message Type or a default handler
a Data Message Type has a digital twin recorded to a permanent journal
DMTT shows what happened to that data message
An example of a service for DMTT is https://debezium.io/
a streaming service for database change events with data
Debezium is an open source distributed platform for change data capture. Start it up, point it at your databases, and your apps can start responding to all of the inserts, updates, and deletes that other apps commit to your databases. Debezium is durable and fast, so your apps can respond quickly and never miss an event, even when things go wrong.
Most commonly, you deploy Debezium by means of Apache Kafka Connect. Kafka Connect is a framework and runtime for implementing and operating:
Source connectors such as Debezium that send records into Kafka
Sink connectors that propagate records from Kafka topics to other systems
The following image shows the architecture of a change data capture pipeline based on Debezium:
Who will fix my data ? Data Janitor
SDM - Smart Data Manager
Knows metadata mgt policies and master data mgt policies are
detects anomalies in data vs policies
useful where standard data definitions for validation need to be dynamically responsive
Normally simple data validation by data type can cover many data validation needs prior to posting data
SDM really applies to complex or dynamic data policy enforcement
SDM can connect CDC services like Debezium through the standard CDC service interface and configuration file
A default CDC configuration for Debezium is provided but other CDCs can be added
What is my data telling me? Data Whisperer
a semantic value problem - is it useful? how? why?
Interpreting data against scenarios for:
goals - sli - okrs
context
history
known or needed data system investments
forecasts
with impact forecasts before and after > FACTUR3DT.io
How can I feel good about my data ? Data Therapist
There mitigations, recoveries, opportunities and more
How can my data add value? Data User for VCE - Value Chain Economy
How can I monetize my data? Data Provider for tokenized Web3 data marketplace
Summary of Data Management Responsibilities -
Manager Data Analytics JOBD shows the domains covered pdf
well defined data mgt services role **
The Manager, Data & Analytics will be responsible for helping to define and then execute upon an enterprise data strategy to ensure the business is able to achieve the greatest ROI on its data assets. A large part of this data strategy will involve enhancing the data & analytics capabilities of the business to extend beyond the traditional data warehouse which exists today. This will include the creation of a data platform which will enable the business to discover, access and use information to drive analytical use cases. This position will be expected to have very strong competencies in both the technical domain, as well as the retail functional domain. This position will manage a team consisting of Data Engineers, BI Developers, Data Analysts as well as Data Scientists to not only deliver data & analytical capabilities, but to also use these capabilities and provide business analytics services to the business.
JOB RESPONSIBILITIES - Data Management Domains
Data Strategy – Work with IT and business leadership to define and execute an enterprise data strategy that promotes intentional data usage, based on a foundation of sound data ownership and a profound awareness of data assets.
Data Engineering – Ensure that the data & analytical capabilities can scale to meet any volume, velocity, or variety of data.
Business Intelligence – Ensure standard corporate reporting and dashboards are created and maintained to provide information needed to drive the business. Reports and dashboards should be created for clearly defined purposes to answer commonly asked business questions.
Business Analytics – Establish and operationalize analytical capabilities, guiding data scientists to create machine learning models to help derive value from the enterprise’s data assets.
Data Governance – Ensure data remains accurate and relevant by establishing and operationalizing data governance standards.
Business Alignment - Be an active participant in the business and a student of the business. Pursue a deep understanding of business functions. While providing the best possible technical solutions and services, act as an advocate for your business partners and the business at large.
New and Leading Edge Technologies - Maintain a curiosity, an awareness and knowledge of new technologies, their possible advantages and applicable relevance to our business.
Production Support - Maintain 24/7 availability to support production services.
Policies and Procedures - Support and adhere to departmental policies and procedures, including but not limited to Project Management, Change Management, and Issue Resolution.
Security & Compliance - Support the efforts of the Security and Compliance associate(s) in the IT department to ensure that all necessary steps are taken to achieve the security and compliance goals of the department and the company.
Vendor Relationship Management - Maintain good working relationships with our select group of third-party vendors, while also holding them accountable for quality service and results.
Customer Relationship Management - Maintain good working relationships with business associates at all levels of the organization. Be honest, open, respectful, and helpful. Be empathetic toward business associates, be a good listener and provide pragmatic solutions to their challenges.
OTHER CHARACTERISTICS
Results oriented with a high degree of resilience and perseverance. Ability to maintain focus on goals and objectives that deliver results.
The ability to coach, develop and inspire others. A trusted advisor with confidence and low ego.
Acts with urgency to resolve issues impacting service or sales.
Weighs the impact on the customer in planning and decision making.
Must have the courage to push back but possess the wisdom and maturity to do so tactfully.
A player/coach who has the ability and is willing to roll up his or her sleeves when required.
Must be both hands on and strategic with the ability to navigate between the two.
Skilled in change management with the capacity to work across all levels of the organization.
Exhibits optimism and enthusiasm. People feel positively challenged when working with this person. Is responsive and enjoys helping other achieve their goals
Strong Ethics and Discretion: Must be a steward of ethics and discretion when it comes to handling sensitive information.
QUALIFICATIONS AND REQUIREMENTS
Bachelor’s degree required in Computer Science, Information Technology, Business, Mathematics or related field.
Minimum of 7 years’ experience in a business analyst role, and a minimum of 3 years in lead and/or management roles, preferably in the retail sector. Must have a strong background in business intelligence, understanding concepts of data visualization to create effective dashboards, as well as the ability to help identify and define new KPIs and metrics.
Must have an exposure to data science and machine learning, understanding the business benefits and common business use cases of each as well the ability to determine which technique/algorithm should be used for common business use cases.
Ability to understand and work with data, both abstractly and in practice. Must be able to translate business problems, processes, or situations into a conceptual data model which can then be extended to logical and physical data models.
Must also be comfortable using pivot tables for ad-hoc analysis.
Other Data Management Keys
Business Data Quality and Value Metrics
EEP concept for all data - no trusted data sources
Data risk and impact analysis - what quality issues should be addressed in what order?
Application data recovery capabilities
Data replays needed for AI, ML apps training etc
Data interchange methods - aliases, mappings, synonyms, directory services etc
Data source RAS ratings
From data to objects for event-driven business operations
example - why do I need a report? what am I doing with the data? why? what's the value? the risks? the costs? the impacts?
Potential Challenges
Candidate Solutions
Step-by-step guide for Example
sample code block