Key Points

1. Account Aggregation is a foundation financial data service
   1. future products can provide more data, function and analytic components of wealth and money management
2. Data pipelines connect, transform data sources to data targets in batches or event streams
3. Multiple roles to add value in data services: provider, aggregator, agent, cataloger, manager, notifier, security, compliance, transformer, processor, usage, logger, analytics, indexer, archiver, finder, viewer, presenter, access controller, resource manager, identity manager, smart cache
   
   1. what are the open-source components we can leverage?
4. common architecture for definitions, development, services, deployment, management, support across all platforms is key
5. conceptual model:   client app >  ds broker > ds agent > ds service > resources > ds adapters > client app
6. common architecture standards is key, ideally using established orgs ( SOC 2, MOBI, ISO, identity foundation, IEEE, IETF )
7. focus on domains to add value, market segmentation:  banking, wealth management, insurance, credit, loans,
   1. provide leading suite adapters ( eg Salesforce, SAP, HIE etc )
8. identity current solution providers for financial data services by domain: equitites, bonds, accounts, purchases, orders
9. a data service is a service to other solutions - not a top level solution to users or organizations
10. account aggregator ds value-adds for VSM, VCN
    1. aggregation, audit, analytics, events, notifications, data trackers, identity, registrations, rbac, dqm, usage x user - data, sql extensions, consent mgt, session workspaces, frames
11. find best aggregators and work on SWOT reputations, credentials in standards orgs
12. study full Homenet life cycle from market dev through support, retention - api and sftp interfaces
13. what are the data products by segment? how are they priced? reward programs for usage over forecast,
14. aggregation audit >> what do we have now?
15. tech study>> data bricks, spark, container spring boot msvc, rbac fwks
16. marketing study>> service business case - who needs? value? other providers SWOT ranked? opporunity sized? success path? success kpi?
17. tech costs>> give mid-range providers
18. more

References

Key Concepts

Data Quality - SOC 2 Data Quality Standards for Data Services

https://www.imperva.com/learn/data-security/soc-2-compliance/

SOC 2 is an auditing procedure that ensures your service providers securely manage your data to protect the interests of your organization and the privacy of its clients. For security-conscious businesses, SOC 2 compliance is a minimal requirement when considering a SaaS provider.

Developed by the American Institute of CPAs (AICPA), SOC 2 defines criteria for managing customer data based on five “trust service principles”—security, availability, processing integrity, confidentiality and privacy.

DQM dimensions and kpi

SQL compared to NoSQL

https://www.w3resource.com/mongodb/nosql.php

https://drive.google.com/open?id=1UAwrKXa9PN3SG4V6IrVfLzdb3L_lgwnw

Transcript

Relations						Basic Relations
Data Manipulation Language(DML) No predefined schema
Indexes							Indexes
Structured query language(SQL)	Limited query capabilities
Transactions						Generally No
Structured and organized data	Unstructured and unpredictable data
Only Values						Values & references
Strongely typed					Loosely typed
Joins							No Joins

Redis DB Cache Management

https://www.w3resource.com/slides/redis-nosql-database-an-introduction.php

https://drive.google.com/open?id=1E3-abaRe0dSBiGCrJJqJ9hx-1P5HDDgL

Redis (Remote DIctionary Server) is an open source, networked, single threaded, in- memory, advanced key-value store with optional durability. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

★ Developer(s) : Salvatore Sanfilippo, Pieter Noordhuis
★ Initial release : April 10, 2009
★ Stable release : 2.8.12 / June 23, 2014
★ Development status : Active
★ Written in : ANSI C
★ Operating system : Cross-platform
★ Type : Key–value stores
★ License : BSD

Redis : What for?

★ Cache out-of-process
★ Duplicate detector
★ LIFO/FIFO Queues
★ Priority Queue
★ Distributed HashMap
★ UID Generator
★ Pub/Sub
★ Real-time analytics & chat apps
★ Counting Stuff
★ Metrics DB
★ Implement expires on items
★ Leaderboards (game high scores)
★ Geolocation lookup
★ API throttling (rate-limits)
★ Autocomplete
★ Social activity feed

Who's using Redis?

★ Twitter
★ Github
★ Weibo
★ Pinterest
★ Snapchat
★ Craigslist
★ Digg
★ StackOverflow
★ Flickr
★ 220.000.000.000 commands per day
★ 500.000.000.000 reads per day
★ 50.000.000.000 writes per day
★ 500+ servers 2000+ Redis instances!

key Features

★ Atomic operations
★ Lua Scripting
★ Pub/Sub
★ Transactions
★ Master/Slave replication
★ Cluster (with automatic sharding)*
★ Automatic failover (Redis Sentinel)
★ Append Only File (AOF) persistence
★ Snapshot (RDB file) persistence
★ Redis can handle up to 232 keys, and was tested in practice to handle at
least 250 million of keys per instance.
★ Every list, set, and sorted set, can hold 232 elements.

key Advantages & Disadvantages

► Advantages

★ Blazing Fast
★ Robust
★ Easy to setup, Use and maintain
★ Extensible with LUA scripting

► Disadvantages

★ Persistence consumes lot of I/O when using RDB
★ All your data must fit in memory

Redis in-memory

► Redis is an in-memory but persistent on disk database, so it represents a different trade off where very high write and read speed is achieved with the limitation of data sets that can't be larger than memory.

Why GraphQL

Glide - open-source GraphQL front-end for Salesforce API set

https://postlight.com/trackchanges/introducing-glide-graphql-for-salesforce

Glide is a free toolkit for engineers who are building Salesforce integrations into their apps. Glide allows you to seamlessly execute GraphQL requests against any Salesforce instance with minimal setup. The code is free and open source. Get Glide on GitHub.

Developers love working with GraphQL, because it brings a strongly-typed schema to your application’s data, and gives clients control over requesting exactly what they need, no more and no less. This reduces network traffic (and latency) compared to REST, where your app has to make multiple calls to various endpoints to retrieve the data it needs (plus some it doesn’t). When GraphQL is paired with React, it’s an incredible development toolchain.

Glide is the quickest way to prototype and build GraphQL applications backed by Salesforce data. Glide automatically introspects your Salesforce data models, and creates an intuitive and idiomatic GraphQL schema out of the box. Glide also handles Salesforce authentication for you, so you can hit the ground running and start prototyping right away.

Glide can spin up a GraphQL server pointing at your Salesforce instance, or you can import Glide into your existing server code and deploy it. With Glide, you can start making GraphQL queries and mutations on your Salesforce data in two commands. Step-by-step instructions to get started are available in the docs.

Potential Value Opportunities

VCN opportunities by market segment

account aggregator ds value-adds for VSM, VCN

aggregation, audit, analytics, events, notifications, data trackers, identity, registrations, rbac, dqm, usage x user - data, sql extensions, consent mgt

SWT Data Platform Comparisons workbook 

data-platforms-v1 gsheet link

SWT Open Data Services Platform

Integrated architecture and services for:

job managment

data ingestion in batch

data ingestion via api

Potential Challenges

Existing Account Aggregators

Mint

Betterment

Candidate Solutions

Open-source NoSQL Databases

https://www.geeksforgeeks.org/top-10-open-source-nosql-databases-in-2020/

m Apache Data Services#Cassandra - open-source high-performance database that is provably fault-tolerant both on commodity hardware or cloud infrastructure

Apache Cassandra is a free and open-source high-performance database that is provably fault-tolerant both on commodity hardware or cloud infrastructure. It can even handle failed node replacements without shutting down the systems and it can also replicate data automatically across multiple nodes. Moreover, Cassandra is a NoSQL database in which all the nods are peers without any master-slave architecture. This makes it extremely scalable and fault-tolerant and you can add new machines without any interruptions to already running applications. You can also choose between synchronous and asynchronous replication for each update.

m Data CouchDb - Couch DB can also scale up for more demanding projects into a cluster of nodes with multiple servers.

Apache CouchDB is an open-source project and a single node database that allows you to easily store your data and access it when you need it. Couch DB can also scale up for more demanding projects into a cluster of nodes with multiple servers. It supports the HTTP protocol along with the JSON data format and also integrates with HTTP proxy servers. Apache CouchDB is designed for reliability with a crash-resistant structure that supports “Offline First” applications and a system that saves data redundantly so that it is never lost and available in a state of emergency.

Apache HBase - distributed Hadoop database

m Apache Data Services#HBase

Apache HBase is an open-source distributed Hadoop database that can be used to read and write to big data. HBase has been constructed so that it can manage billions of rows and millions of columns using commodity hardware clusters. This database is based on the Big Table which was a distributed storage system created for structured data. Apache HBase has many different capabilities including scalability, automatic sharding of tables,  consistent reading and writing capabilities, support against failure for all the servers, etc.

MongoDB - a general-purpose distributed database with ACID support

MongoDB is a general-purpose distributed database created for the application developers in this generation to use in the cloud. This is a document database that stores the data in JSON-like documents which is much more powerful and efficient than the traditional row and column databases. MongoDB also supports various methods of searching such as geographical searching, text searching, graph searching, etc. Another advantage of MongoDB is that it provides first-class security for its clients including SSL, firewalls, encryption, etc. And the best thing is you can also create visualizations using MongoDB data and connect with any Business Intelligence tools that are compatible with the MySQL protocol.  

m MongoDB

m MongoDb usage basics

Neo4j - a graph-based database

Neo4j is a graph-based database that is excellent in handling not only data but also data relationships. Since Neo4j connects the data when it is stored in the database, it can access the data again much faster than conventional databases. Each data record has direct pointers to all the other data records it is connected with and this underlines the power of the database. Neo4j also uses Cypher queries that are much faster and simpler to write than SQL queries and since it doesn’t have any tables there is no need to bother about joins. Neo4j also provides drivers for  Java, .Net, JavaScript, Python and Go in an official capacity whereas the open-source community contributors provide many other drivers like Ruby, PHP,  R, C, C++, etc.

Risk - distributed NoSQL database that is highly resilient

Riak is a distributed NoSQL database that is highly resilient and ensures data accuracy. It is created using multiple clusters that make sure data is not lost even in the event of a hardware failure and read/write operations can continue smoothly. Riak is designed using a key/value specification that solves many challenges in the management of big data such as tracking user data, copying the data in various locations all over the world, storing connected data, etc. Some of the features of Riak include scalability, operational simplicity, resiliency, complex query support, etc. It can also integrate with Apache Spark to provide real-time analysis of Spark.

Redis - in memory database that supports many different data structures such as strings, lists, hashes, sets, sorted sets

https://redis.io/

Redis is an open-source database that supports many different data structures such as strings, lists, hashes, sets, sorted sets, etc. It is written in ANSI C and it can be used with almost all the programming languages and Linux and OS X operating systems. Redis works with an in-memory dataset to preserve its extremely fast performance and the implementation uses the fork system call to create a duplicate of the current process with the data so that the parent process can continue its operations with the existing clients and the child process can create a data copy on the disk.

In memory databases

Well-known as a "data structure server", with support for strings, hashes, lists, sets, sorted sets, streams, and more.

Server-side scripting with Lua and server-side stored procedures with Redis Functions.

Horizontal scalability with hash-based sharding, scaling to millions of nodes with automatic re-partitioning when growing the cluster.

Replication with automatic failover for both standalone and clustered deployments.

Keeps the dataset in memory for fast access, but can also persist all writes to permanent storage to survive reboots and system failures.

Redis use cases

Redis' versatile in-memory data structures enable building data infrastructure for real-time applications that require low latency and high-throughput.

Redis' speed makes it ideal for caching database queries, complex computations, API calls, and session state.

The stream data type enables high-rate data ingestion, messaging, event sourcing, and notifications.

Redis Stack

Redis Stack Server lets you build applications with searchable JSON, time series and graph data models, and extended probabilistic data structures.

Get productive quickly with the Redis Stack object mapping and client libraries.

Hypertable - keeps the data sorting using a primary key, unlike most other NoSQL databases that use the hash table design

Hypertable is NoSQL open-source database that was designed to combat the scalability problem that appears in all relational databases. It was based on the Google Big Table design and written in C++. Hypertable runs in both Linux and Mac OS X. It is also suitable for a wide range of applications as it keeps the data sorting using a primary key, unlike most other NoSQL databases that use the hash table design. Hypertable is also suited to provide maximum efficiency with minimum performance and stability costs which makes it extremely cost-efficient.

RavenDB - benefits of a NoSQL database with all the conveniences of a relational database and high performance at low cost

RavenDB is a NoSQL document database that provides the benefits of a NoSQL database with all the conveniences of a relational database. It also offers fully transactional (ACID) data integrity so you can use it along with your existing SQL databases to get the most out of both types. This database is also highly scalable and it can create new nodes to keep up with increasing data traffic. RavenDB is available for installation on-premises as well as in the form of a cloud service provided by Amazon Web Services, Azure, etc.

https://ravendb.net/why-ravendb/acid-transactions

ACID transactions

RavenDB is one of the first nonrelational databases to offer ACIDity not just for a single value, but for multiple values throughout your database as well. And as a distributed database, it also offers ACID guarantees throughout your database cluster. Your developers are exempt from handling the numerous scenarios of partial data transfers and the intricacies of data storage, and are free to focus on building robust applications and deliver value to the business.

Low cost performance

In some cases, data is streamed from small edge points like sensors embedded in machinery, clothing and even the human body, and then relayed to servers for immediate processing. On a Raspberry PI, a $25 machine running on low powered ARM chips and a mere 1 GB of RAM, RavenDB can handle over 13,000 reads per second and over a 1,000 writes per second. This is more than enough for most small to medium applications. For demands higher than that, a single RavenDB server running on a machine of less than $1,000 can handle over 150,000 writes per second and over a million reads per second.

Google Data Products - Data to AI

https://www.linkedin.com/posts/rajkgrover_dataarchitecture-api-governance-activity-7168518478558085121-hM1V?utm_source=share&utm_medium=member_desktop

Image Source: Data to AI, Google Cloud

diagram below misses several key dimensions >>. trusts, governance, value ( VCRS )

Step-by-step guide for Example

sample code block

Recommended Next Steps

Related articles