In this article we will be looking at a new pipeline offering called Webhook Pipelines
and explore specific use cases for using this new technology.
Until now, Enzo offered Replication Pipelines that can be used to replicate data changes from a source system to one or more
destination systems, providing both Change Data Capture (CDC) and replay capabilities.
With the introduction of Webhook Pipelines, clients can push messages into the pipeline
architecture so that messages sent by client applications or systems can be forwarded to
one or more listeners, with full replay capability.
Use cases for Webhook Pipelines
Application Development
Webhook pipelines can simplify application development by abstracting
the destination messaging system architecture and complexity, since
sending messages is done through simple HTTP requests. In addition,
abstracting the underlying messaging architecture reduces long-term
maintenance costs when the destination messaging architecture needs to change.
For example you can build an application that leverages
Azure Queues (from the Azure Storage Service), but may choose to leverage
the Azure Service Bus in the future. Using webhook pipelines allows you
to add/remove listeners without changing a single line of application code.
Multicast Architecture
The pipeline architecture provides the ability to send messages to any number of listeners
with an eventual consistency model. This allows an application to send a single message, and
forward the message to multiple destination systems. For example, if an application sends a
warning in the form of an HTTP post to a webhook pipeline, the message could be forwarded to
two or more listeners without changing the application code. This is also very useful for
transparently forwarding HTTP messages sent from vendor hosted systems.
Change Data Capture & Replay
A key feature of the Enzo Pipeline architecture is the ability to store messages
into a scalable CDC Store, providing guaranteed delivery of messages to all listeners.
In addition, each listener can choose to replay a portion of the CDC store if necessary.
This provides destination systems the ability to receive some or all of the messages
from the CDC store if necessary.