| Table of Contents |
|---|
Key Points
...
Message Brokers include: IBM MQ, ActiveMQ, RabbitMQ, Kafka etc
While ESB tools are a reliable way to connect applications, their technology is becoming more obsolete as the cloud dominates digital ecosystems and as companies experience a rate of growth that requires even faster adaptability across their systems.
ESB comeback?
unless you are running everything on a single cloud platform and have low needs for integration with customers, vendors and partners, an ESB ( or Event Bus ) may speed integration. If everything you need is on your cloud platform you are probably fine using just what AWS, Azure etc offer as commercial services.
Kafka as a Message Broker
...
ActiveMQ implements the functionality specified in the Java Message Service (JMS) API, which defines a standard for creating, sending, and receiving messages. ActiveMQ client applications—producers and consumers—written in Java can use the JMS API to send and receive messages. Additionally, both Classic and Artemis support non-JMS clients written in Node.js, Ruby, PHP, Python, and other languages, which can connect to the ActiveMQ broker via the AMQP, MQTT, and STOMP protocols.
Artemis easier to maintain, which is important if you're basing a commercial product on it. The smaller feature set means a smaller overall implementation, which fits well with developing microservices.
Implementing Apache ActiveMQ-style broker meshes with Apache Artemis
Artemis easier to maintain, which is important if you're basing a commercial product on it. The smaller feature set means a smaller overall implementation, which fits well with developing microservices.
This article describes subtleties that can lead to lost messages in an Artemis active-active mesh. That architecture consists of multiple message brokers interconnected in a mesh, each broker with its own message storage, where all are simultaneously accepting messages from publishers and distributing them to subscribers. ActiveMQ and Artemis use different policies for message distribution. I will explain the differences and show a few ways to make Artemis work more like ActiveMQ in an active-active scenario.
Artemis supports broker discovery for message meshes
The final vital piece of configuration assembles the various broker connectors into a mesh. Artemis provides various discovery mechanisms by which brokers can find one another in the network. However, if you're more familiar with ActiveMQ, you're probably used to specifying the mesh members explicitly.
Artemis provides a number of discovery mechanisms, allowing clients to determine the network topology without additional configuration. These don't work with all wire protocols (notably, there is no discovery mechanism for Advanced Message Queuing Protocol), and ActiveMQ users are probably familiar with configuring the client's connection targets explicitly. The usual mechanism is to list all the brokers in the mesh in the client's connection URI.
you should be able to connect consumers to all the nodes, produce messages to any node, and have them routed to the appropriate consumer. However, this mesh won't behave exactly like ActiveMQ, because Artemis mesh operation is not governed by client demand.
Message forwarding in Artemis is more flexible than Classic ActiveMQ
In ActiveMQ, network connectors are described as "demand forwarding." This means that messages are accepted on a particular broker and remain there until a particular client requests them. If there are no clients for a particular queue, messages remain on the original broker until that situation changes.
...
Written in Erlang, the RabbitMQ server is built on the Open Telecom Platform framework for clustering and failover. Client libraries to interface with the broker are available for all major programming languages. The source code is released under the Mozilla Public License.
Message Broker
A message broker (also known as an integration broker or interface engine[1]) is an intermediary computer program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver. Central reasons for using a message-based communications protocol include its ability to store (buffer), route, or transform messages while conveying them from senders to receivers.
The following represent other examples of actions that might be handled by the broker:[2][3]
- Route messages to one or more destinations
- Transform messages to an alternative representation
- Perform message aggregation, decomposing messages into multiple messages and sending them to their destination, then recomposing the responses into one message to return to the user
- Interact with an external repository to augment a message or store it
- Invoke web services to retrieve data
- Respond to events or errors
- Provide content and topic-based message routing using the publish–subscribe pattern
The Advanced Message Queuing Protocol (AMQP)
...
Streaming Text Oriented Message Protocol (STOMP)
Simple (or Streaming) Text Oriented Message Protocol (STOMP), formerly known as TTMP, is a simple text-based protocol, designed for working with message-oriented middleware (MOM). It provides an interoperable wire format that allows STOMP clients to talk with any message broker supporting the protocol.
- RabbitMQ (message broker, has support for STOMP)
MQTT is a lightweight, publish-subscribe network protocol that transports messages between devices. The protocol usually runs over TCP/IP, however, any network protocol that provides ordered, lossless, bi-directional connections can support MQTT.[1] It is designed for connections with remote locations where resource constraints exist or the network bandwidth is limited. The protocol is an open OASIS standard and an ISO recommendation (ISO/IEC 20922).
Compare Kafka and Rabbit MQ
...
https://www.npmjs.com/package/mitt
// listen to an event emitter.on('foo', e => console.log('foo', e) ) // listen to all events emitter.on('*', (type, e) => console.log(type, e) ) // fire an event emitter.emit('foo', { a: 'b' })
Simple Java Events - use Property Beans for objects in the SAME process - see groovy event example
...
Microsoft Azure Event Grid is a fully managed event routing service provided by Microsoft Azure. It provides a highly scalable and highly available event infrastructure for event-driven architectures and serverless applications. Event Grid supports a variety of event sources and can route events to multiple event handlers in real-time
it charges based on the number of events processed and the number of operations performed
may not provide the level of customization and flexibility required for some scenarios.
can limit interoperability with other platforms and technologies that use different event processing solutions.
Google Cloud Pub/Sub
https://cloud.google.com/pubsub
Google Cloud Pub/Sub is a fully managed messaging service provided by Google Cloud, that allows applications to publish and subscribe to messages in a scalable and reliable manner. Pub/Sub supports a variety of programming languages and platforms and provides powerful features for handling events, such as dead-letter topics, push and pull delivery, and message filtering.
it charges based on the number of messages processed and the amount of data stored.
can limit interoperability with other platforms and technologies
Amazon Simple Notification Service (SNS)
...
Amazon Simple Notification Service (SNS) is a fully managed messaging service provided by Amazon Web Services (AWS), that enables applications to send and receive notifications. It provides a simple and scalable messaging infrastructure for event-driven applications. SNS supports a variety of event sources and can route events to multiple event handlers, including email, SMS, and other AWS services.
it charges based on the number of messages processed and the amount of data stored.
can limit interoperability with other platforms and technologies
Apache EventMesh
https://github.com/apache/eventmesh
Event Mesh on Low-code App Development Solutions swtc page
Apache EventMesh is a new generation serverless event middleware for building distributed event-driven applications.
Apache EventMesh itself has four primary components:
- eventmesh-runtime : an middleware to transmit events between event producers and consumers, support cloud native apps and microservices.
- eventmesh-sdk-java : currently supports HTTP and TCP protocols.
- eventmesh-connector-api : an api layer based on OpenMessaging api and SPI pluggin, which can be implemented by popular EventStores such as IMDG, Messaging Engine and OSS etc.
- eventmesh-connector-rocketmq : an implementation of eventmesh-connector-api, pub event to or sub event from RocketMQ as EventStore.
Apache EventMesh provides standard protocol such as cloud events; interface such as http, tcp etc.; and pluggable storage engine with the Apache RocketMQ as the default back-end storage. Compared to other mesh-based applications in the market, our platform supports fancy features like extreme low latency and stability, and cloud-native architecture
need to address complexity brought in by heavy clients based on the pull model; event-based streaming; and, the openness of event metadata and mixed media storage
Apache EventMesh has a vast amount of features to help users achieve their goals. Let us share with you some of the key features EventMesh has to offer:
...
Event Mesh - open-source cloud-native eventing infrastructure that decouples the application and backend middleware layer,
http://eventmesh.incubator.apache.org/
Features
- Communication Protocol: EventMesh could communicate with clients with TCP, HTTP, or gRPC.
- CloudEvents: EventMesh supports the CloudEvents specification as the format of the events. CloudEvents is a specification for describing event data in common formats to provide interoperability across services, platforms, and systems.
- Schema Registry: EventMesh implements a schema registry that receives and stores schemas from clients and provides an interface for other clients to retrieve schemas.
- Observability: EventMesh exposed a range of metrics, such as the average latency of the HTTP protocol and the number of delivered messages. The metrics could be collected and analyzed with Prometheus or OpenTelemetry.
- Event Workflow Orchestration: EventMesh Workflow could receive an event and decide which command to trigger next based on the workflow definitions and the current workflow state. The workflow definition could be written with the Serverless Workflow DSL.
...
Apache EventMesh (Incubating) is a dynamic event-driven application multi-runtime used to decouple the application and backend middleware layer, which supports a wide range of use cases that encompass complex multi-cloud, widely distributed topologies using diverse technology stacks.
EventMesh roadmap
https://eventmesh.apache.org/docs/roadmap/
...
-------------------------------
data services - Node-RED
-------------------------------
Node-RED basics video
https://www.youtube.com/watch?v=3AR432bguOY
Node-RED best pracices video
https://www.youtube.com/watch?v=3AR432bguOYNode-RED best pracices video.youtube.com/watch?v=V0SmNcIYCtQ
Apache Pulsar - is an open-source, distributed messaging and streaming platform built for the cloud.
m Apache Data Services#ApachePulsar
https://wwwpulsar.youtubeapache.comorg/watch?v=V0SmNcIYCtQ
Solace Message Service and Event Hub with EDA - Event Driven Architecture ( compare to EventMesh, Kafka, CouchDb )
...
Potential Value Opportunities
Compare Pulsar, Solace & Artemis for Event Mesh and Streaming Apps
detail comparison on popularity, features, benefits, challenges, case studis of apache pulsar, solace and apache artemis for distributed event mesh and streaming applications?
Conceptual Value Chain Services
...
distributed network work flows with distribution event subscriptions with a large set of service capabilities
see Modern Platform ODP supports VCE , DAN, STSVCE > Value Chain Economies: micro economies for value-chain communities ( VCC )#FSN-FinancialServicesNetworkisaconceptualFinancialMarketInfrastructure(FMI) for more details
Potential Challenges
...
Onix Solutions Java FIX Engine is a simple fully Java compliant tool that will FIX-enable applications written in Java.
The Engine provides the following services:
- manages a network connection
- manages the session layer (for the delivery of application messages)
- manages the application layer (defines business related data content)
- creates (outgoing) messages
- parses (incoming) messages
- validates messages
- persists messages (the ability to log data to a flat file)
- session recovery in accordance with the FIX state model
import biz.onixs.fix.engine.Engine; public class SimpleEngine { public static void main(String[] args) { try { final Engine engine = Engine.init(); // Session-related logic... engine.shutdown(); } catch (Exception ex) System.out.println("Exception: " + ex); } } }
Message Format
The general format of a FIX message is a stream of <tag>=<value> fields with a field delimiter (<SOH>) between fields in the stream (so-called "tag-value FIX format").
Step-by-step guide for Example
...