Lessons from the Trenches: Event-Driven Systems

There is no dearth of material on Event-Driven Architectures (EDA) in the context of DDD and Microservices, with some great blogs by Martin Fowler & Mathias Verraes, in addition to some brilliant books and videos, courses etc. ( No affiliation to any of these).

The idea for this post arose from a discussion in an internal ThoughtWorks technical chat thread on EDAs. It really made me think hard of the mistakes I have made in designing such systems.

However, just like any other technical concept, there are two phases of learning: one is reading and the other is building. And both are critical to the process of learning and doing.

Digression: I have realized that “learning” as an activity unto itself is sometimes an escape hatch that I use to avoid the activities that cause Resistance, aka tasks that matter and need Action, but are psychologically perceived painful. I end up wasting time watching technical videos and reading blogs that make me feel productive, but are not in fact.

I was fortunate enough to get an opportunity to implement EDAs with one of my projects. It was a large-scale & high-traffic system with critical business data and stringent SLAs. In this post, I’ll skip the “ Why” of adopting an asynchronous design and will rather share the “What”, “How” and “What Not” from the experience.

This is a subjective post and I expect (and hope for) some disagreements from the readers.

Kafka event-driven architecture diagram Source: blog.ippon.tech

Learnings

Consider using Event Schemas to ensure their evolution is handled in a versioned way. On the other hand, avoid schemas if the variation of the types of commands/events is small.
Consider Event Registries in advanced cases ( e.g. when the types of events are in hundreds and the number of consumers is very high). The highly decoupled nature of event-driven systems can sometimes lead to reduced traceability of dependencies and event data flow graph. Registries act like bookkeepers of publisher/consumer contracts. But like point 1 above, it’s an overkill for less complex systems.
Consider idempotency conditions, since Exactly Once semantics are not reliable. Kafka fans will tout its Exactly Once guarantees. However for critical processes involving financial or logistical systems, idempotency is a responsibility of the consumer systems. The CEO won’t be thrilled if double spending is commonplace, or if the same customer order is shipped twice.
Consider side effects of eventual consistency across systems. In more complex cases, the systems need to behave transactionally and/or atomically. In such cases, the likes of Saga Pattern or Outbox Pattern could be useful. For a non-trivial & multi-step transactionality, a central orchestrator with an async workflow is a better fit ( e.g. A complex Tax related workflow that combines automated, manual and batch operations). This opens the Pandora’s box of the BPM tools. Use them at your own risk.
Consider the eventuality of replaying events or commands— Sometimes systems reject events with incorrect acknowledgments and they may have to be re-sent. An event store that persists the event data and whether it succeeded or not could be useful. Most message brokers have this functionality built-in.
Consider pull vs push model of event handling. ( e.g. Kafka is pull-based, while GCP Pub/Sub provides both). Pull-based systems are good for more control & easy back-pressure configuration ( since the consumer dictates the rate of pulling messages), but it causes coupling between the broker and consumers, and a lot of onus falls upon the consumer.

Push-based brokers are more decoupled from consumers who only need to expose an API to the broker. But push-based systems need rate limiting for effective back pressure. This adds some complexity, and any 429s may be treated as errors.

In my experience, I have found the push-based model more reliable and robust.
Consider retrying with Exponential Back-offs and configuring Dead Letter Topics/Queues (DLQs) as a default practice. Also have intelligent retry jobs from DLQ.

This is a no-brainer in any non-trivial EDA system, especially those with high throughput.
Consider a business facing reporting functionality for the number of events received vs consumed vs failed. Most brokers provide this out of the box, along with alerting rules. In many cases a human intervention is imperative, and such reports are vital to make decisions. It saved our team’s time when we could present numbers to executives.
Explore the AsyncAPI standard for event driven APIs ( https://www.asyncapi.com). Although I haven’t been able to use it hands on, it looks promising with support for generating Specifications (like OpenAPI), Code generation, Documenting Events, Governance etc.
Evaluate the decision on the contents of your event payload with great deliberation. It’s a Type-1 decision and has caused me a great deal of pain. Balance between API chattiness and bulky events.
And lastly, YAGNI. Consider lightweight queuing mechanisms instead of Kafka or heavyweight brokers. You likely don’t operate at LinkedIn scale. ( e.g. Postgres, Redis, RabbitMQ etc. can act as suitable queues for many use cases) — Postgres as a Queue, Redis Pub/Sub

25-Oct-2024 EDIT: Avoid Redis Pub/Sub for applications needing strong guarantees of delivery.

These are some of the lessons I could recollect from my experience. I am sure there would be a lot more Whats and What-Nots. Hope this helps!