m Data Quality
Key Points
- Account Aggregation is a foundation financial data service
- future products can provide more data, function and analytic components of wealth and money management
- Data pipelines connect, transform data sources to data targets in batches or event streams
- Multiple roles to add value in data services: provider, aggregator, agent, cataloger, manager, notifier, security, compliance, transformer, processor, usage, logger, analytics, indexer, archiver, finder, viewer, presenter, access controller, resource manager, identity manager, smart cache
- what are the open-source components we can leverage?
- common architecture for definitions, development, services, deployment, management, support across all platforms is key
- conceptual model: client app > ds broker > ds agent > ds service > resources > ds adapters > client app
- common architecture standards is key, ideally using established orgs ( SOC 2, MOBI, ISO, identity foundation, IEEE, IETF, ISO-20022 fin dsl )
- focus on domains to add value, market segmentation: banking, wealth management, insurance, credit, loans,
- provide leading suite adapters ( eg Salesforce, SAP, HIE etc )
- identity current solution providers for financial data services by domain: equitites, bonds, accounts, purchases, orders
- a data service is a service to other solutions - not a top level solution to users or organizations ( see SWT MySQL rbac data services layer w session and global data frames )
- account aggregator ds value-adds for VSM, VCN
- aggregation, audit, analytics, events, notifications, data trackers, identity, registrations, rbac, dqm, usage x user - data, sql extensions, consent mgt
- find best aggregators and work on SWOT reputations, credentials, consents, privacy policies in standards orgs
- study full Homenet syndication life cycle from market dev through support, retention - api and sftp interfaces
- what are the data products by segment? how are they priced? reward programs for usage over forecast? revenue gen or cost of business?
- aggregation audit >> what do we have now?
- tech study>> data bricks, spark, container spring boot msvc, rbac fwks
- marketing study>> service business case - who needs? value? other providers SWOT ranked? opporunity sized? success path? success kpi?
- tech costs>> give mid-range providers
- more
References
Reference_description_with_linked_URLs_______________________ | Notes______________________________________________________________ |
---|---|
https://www.w3resource.com/mongodb/nosql.php | SQL compared to NoSQL article |
https://github.com/aidtechnology/lf-k8s-hlf-webinar | Kafka on Kubernetes Tutorial |
https://www.informatica.com/in/products/data-quality/informatica-data-quality.html | Data Quality concepts from Informatica |
https://pages.databricks.com/WB-azuretraining-01.html | Databricks video tutorials on Azure - engineering, analytics, data science |
Articles | |
https://www.imperva.com/learn/data-security/soc-2-compliance/ | SOC 2 Data Quality Standards for Data Services |
https://drive.google.com/open?id=1W1vHQ4oxOJUB_OSX1B5c9L06HRNjXXK- | Dummies Guide to Cloud Enterprise Data Platforms |
https://drive.google.com/open?id=1zipd_DJ_GxoCRgUlEtMrzErT2jntkSDv | IBM Cloud Pak |
Key Concepts
Data Quality - SOC 2 Data Quality Standards for Data Services
https://www.imperva.com/learn/data-security/soc-2-compliance/
SOC 2 is an auditing procedure that ensures your service providers securely manage your data to protect the interests of your organization and the privacy of its clients. For security-conscious businesses, SOC 2 compliance is a minimal requirement when considering a SaaS provider.
Developed by the American Institute of CPAs (AICPA), SOC 2 defines criteria for managing customer data based on five “trust service principles”—security, availability, processing integrity, confidentiality and privacy.
DQM dimensions and kpi
- Completeness.
- Uniqueness.
- Timeliness.
- Validity.
- Accuracy.
- Consistency.
- Auditabilty
- Utility ( consumption )
- Recoverability
- Authenticity
- Authorization
- Cost
- Risk
- Resources consumed ( efficiency )
- Revenue impacts ( likely indirect )
- Data. Quality. Dimensions.
Data Quality Concepts from Informatica
https://www.informatica.com/in/products/data-quality/informatica-data-quality.html
DQ Concepts
Big Data Quality Framework Concepts
rajg
Potential Value Opportunities
VCN opportunities by market segment
account aggregator ds value-adds for VSM, VCN
aggregation, audit, analytics, events, notifications, data trackers, identity, registrations, rbac, dqm, usage x user - data, sql extensions, consent mgt
SWT Open Data Services Platform
Integrated architecture and services for:
job managment
data ingestion in batch
data ingestion via api
data_service_type_______________ | data_solution_options______________________ | Notes_________________________________ |
---|---|---|
distributed services orchestration | Kubernetes | provides overlay network on other platforms, networks need smart caches, federation, replicas, locality |
distributed messaging | Kafka | |
distributed services | gRPC | |
REST API | check StrongLoop, Grails, Feathers, Next.js or ? | |
Potential Challenges
Existing Account Aggregators
Mint
Betterment
Candidate Solutions
Step-by-step guide for Example
sample code block