| Table of Contents |
|---|
...
Key Concepts
Data Use Cases & Decision Match Data Processing Flows
Functional Data Layers Architecture
...
https://drive.google.com/open?id=1ryNqjIE3LCY6Jxik_4V-urf2aCp9QLr0
2024-07-eb-big-book-of-data-engineering-3rd-edition.pdf
Guidance and Best Practices
Databricks Assistant Tips and Tricks for Data Engineers
Applying Software Development and DevOps Best Practices to Delta Live Table Pipelines
Unity Catalog Governance in Action: Monitoring, Reporting and Lineage
Scalable Spark Structured Streaming for REST API Destinations
A Data Engineer’s Guide to Optimized Streaming With Protobuf and Delta Live Tables
Design Patterns for Batch Processing in Financial Services
How to Set Up Your First Federated Lakehouse
Orchestrating Data Analytics With Databricks Workflows
Schema Management and Drift Scenarios via Databricks Auto Loader
From Idea to Code: Building With the Databricks SDK for Python
Data Architecture vs Enterprise Architecture
...
Data Lake Platform Concepts
khub.Data Services - Candidate Solutions
DWH > Data Lake > Lake House > Data Mesh concepts
...
data-delta-lake-up-&-running_er2.pdf. link. OReilly ebook
...
https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_a_WebSocket_server_in_Java
Writing WebSocket servers](/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers
...
disributed-db-sharding-strategies1.pdf
This article looks at four data sharding strategies for distributed SQL including algorithmic, range, linear, and consistent hash.
Data sharding helps in scalability and geo-distribution by horizontally partitioning data. A SQL table is decomposed into multiple sets of rows according to a specific sharding strategy. Each of these sets of rows is called a shard. These shards are distributed across multiple server nodes (containers, VMs, bare-metal) in a shared-nothing architecture. This ensures that the shards do not get bottlenecked by the compute, storage, and networking resources available at a single node. High availability is achieved by replicating each shard across multiple nodes. However, the application interacts with a SQL table as one logical unit and remains agnostic to the physical placement of the shards. In this section, we will outline the pros, cons, and our practical learnings from the sharding strategies adopted by these databases.
Data Driven Organization Maturity Levels
...