Key Points

Data pipelines connect, transform data sources to data targets in batches or event streams
Kafka directory services definitions stored as metadata with optional data schemas
brokers connect clients, producers to topics ( queues ) using Kafka directory services
handles many types of producers, consumers, topics (queues), replication, partitions
messages can be consumed by multiple clients, sent by multiple producers, serialized, synch or asynch
clients must poll a queue - there is DIRECT NO PUSH options ( eg web sockets model etc )
producers push message to topics queues
queues can be persistent
REST proxy available to interface to queues
Future option to replace Zookeeper directory with distributed metadata services ( DMS )

References

Reference_description_with_linked_URLs_______________________	Notes______________________________________________________________
m Apache Kafka

https://projects.apache.org/projects.html?category	Apache Big Data projects
https://www.educba.com/my-courses/dashboard/	Data engineering education site: educfba
m Apache Data Services


Kafka
https://kafka.apache.org/	Apache Kafka home
https://kafka.apache.org/quickstart	Kafka quickstart
https://kafka.apache.org/uses	Kafka use cases
https://kafka.apache.org/documentation/	Kafka documentation
https://drive.google.com/open?id=1s05zoAUC3kkW6ZDX_LGjMRptySCToZ_a https://gist.github.com/fagossa/ff43fb522369efb2c88a82c8c72dec80 https://www.datadoghq.com/blog/collecting-kafka-performance-metrics/ https://www.datadoghq.com/blog/monitor-kafka-with-datadog/	Kafka concepts overview part 1 part 2 part 3
https://github.com/aidtechnology/lf-k8s-hlf-webinar https://drive.google.com/open?id=17S6ONAr7n3FyIY7-dknBsIs2vKFrY3xr	Kafka on Kubernetes Tutorial
https://drive.google.com/open?id=1mOk7UJ_cF0EIXt2QWVeWCFJg7bIm8Nc1	A Nodejs client app using Kafka overview
https://drive.google.com/open?id=1gwyX26VkGlI1wOiU2bgLk_ZaJIQNJ6aq	ebook - Kafa Streams in Action
https://www.confluent.io/online-talks/	Kafka talks online
https://drive.google.com/open?id=1mYlwU0z8hirNQrlPJYtWFqaUh9P6xnhy	Kafka event streams with MongoDB connector
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals	Kafka Improvement Proposals
https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ ZooKeeper+with+a+Self-Managed+Metadata+Quorum	KIP-500 RFC - replace Zookeeper for Kafka MDM
https://drive.google.com/open?id=10T8h05mecdU12FTmSflww4HfZNKghx1_ kafka-guide-2019-confluent-kafka-definitive-guide-complete.pdf	Kafka Guide - 2019 - Confluent
https://assets.confluent.io/m/7a91acf41502a75e/original/20180328-EB-Confluent_Designing_Event_Driven_Systems.pdf?_ga=2.165530684.2462 0135.1589381838-1712595248.1587214799 kafka-ebook-20180328-EB-Confluent_Designing_Event_Driven_Systems.pdf	Kafka - Design Event Driven Systems ebook
https://www.confluent.io/resources/kafka-summit-2020/	Kafka 2020 Summit with presentations **
https://kafka-tutorials.confluent.io/?_ga=2.255305614.740418466.1611961981-2041899468.1610139243	Kafka 101 Tutorial - Event Streaming start ***
https://www.confluent.io/resources/kafka-summit-2020/apache-kafka-and-ksqldb-in-action-lets-build-a-streaming-data-pipeline/	Kafka and KSQL Tutorial - create a data pipeline ***

https://talks.rmoff.net/JuUJ8l#sSlsXRY	Apache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline
https://talks.rmoff.net/yE9NeV	Apache Kafka - From Zero to Hero with Kafka Connect
	Apache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline! by Robin Moffatt

https://banzaicloud.com/blog/supertubes-focal/	Using Supertubes, Kafka and tools to automate POD recoveries


Compare to other open-source distributed file management systems
https://blockonomi.com/interplanetary-file-system/ ipfs-blockonomi.com-What is IPFS Interplanetary File System Complete Beginners Guide.pdf	IPFS tutorial
	UDT