In this article, guest blogger Gigi Sayfan explores event-driven programming: its benefits and shortcomings, how it works, and what useful patterns it brings to the table. We’ll also dive into some fun examples.
Gigi Sayfan
|RSS FeedGuest blogger
Subscribe for the latest news and insights on open source, Aiven offerings, and more.
Event-driven programming is a great approach for building complex systems. It embodies the divide-and-conquer principle while allowing you to continue using other approaches like synchronous calls.
When discussing event-based systems, several different terms often refer to the same concept. For simplicity, we’ll primarily use the terms listed below in bold:
Event-driven programming, or event-oriented programming, is a paradigm where entities (objects, services, and so on) communicate indirectly by sending messages to one another through an intermediary. The messages are typically stored in a queue before being handled by the consumers.
Unlike in using direct calls, event-driven programming completely decouples the producer from the consumer, leading to some noteworthy benefits. For example, multiple producers and multiple consumers can all collaborate to process incoming requests. Retrying failed operations and maintaining an event history are also simplified. Event-driven programming also makes it easier to scale large systems, adding capacity simply by adding consumers.
Let's look at the actors of event-based systems: events, producers, consumers, and message queues.
Events are pieces of data that are sent by producers and eventually consumed by consumers.
For example, a mouse down event typically contains the following information:
Events are usually structured but can sometimes just be blobs of data (such as a chunk of JSON) that consumers should know how to parse and handle.
Producers are entities that generate events and send them to a message queue. Consumers either subscribe to receive new events or poll periodically from the queue. Key to the event-driven paradigm is this main point: Producers and consumers are blissfully unaware of one another and interact through the message queue only.
Learn the basics of Apache Kafka Producers and Consumers through building an interactive notebook in Python
Try it outMessage queues are repositories of messages (events). The messages may be stored in memory and/or durable storage. Message queues are typically partitioned into topics. Producers and consumers send and receive messages for particular topics separately.
A message broker is responsible for making sure messages that are sent to the queue are delivered to all subscribed consumers. Some message brokers in popular use today include RabbitMQ, Redis, and Apache Kafka.
Consumers use pull or push mode to consume messages. In pull mode, the message queue keeps all the messages for all the consumers. When a message has been received by all consumers, that message may be removed from the queue (depending on retention configuration). Some message queues allow consumers to pull historical messages.
In push mode, the message queue pushes any new message to all current subscribers.
Let's explore some of the properties of the event-driven programming paradigm and see what makes it cool—and what makes it tricky. Our focus will be on push-based systems.
Traditional call-based workflows are tightly coupled. If object A has some information X that is relevant to object B, then object A needs to satisfy the following requirements to deliver this information successfully to B:
That’s a lot of requirements.
To further complicate matters, consider a system in which B is not the only one that is interested in X. Maybe C and D are also interested in X, and they have different interfaces for receiving X. Now, A needs to get references to C and D, along with their respective interfaces. A also needs to consider what to do in case of partial failure (for example, if B and C received X successfully, but D failed).
Now, let's look at the requirements on object A for a loosely-coupled, event-based system:
That’s it. Now B, C, and D can all subscribe to receive event X—if they are interested. The event framework managing the queue can have policies about error handling and retries, and these can often be specified by consumers.
The downside of loosely-coupled workflows, however, is that these interested parties are difficult to see in the code or in a call graph and workflow.
The request-response approach to communication is typical of traditional web development. For example, a user navigates to a new URL, the browser requests a web page, and the server responds with information. This is a typical pattern of remote procedure call (RPC) systems.
The publish-subscribe approach (PubSub for short) tackles a different use case: fire and forget. The publisher simply publishes an event to the queue without needing any confirmation or response.
Both approaches have their place. While it is possible to implement asynchronous request-response using events—as we will see later—doing so is cumbersome.
Synchronous calls are your garden variety function calls. You call a function with some arguments, and you get the result on the same thread. The thread will be blocked until the function call returns.
Asynchronous calls return immediately and don't wait for the computation to complete. The result will arrive later on a separate thread. Event-based systems that rely on PubSub often use asynchronous calls to receive the events. The consumer subscribes to specific topics or events, and the events are handled in a separate thread.
When you make a call to another object or service, there is always the possibility of an error. In general, the caller can take any of several actions:
Often, the event framework can be configured to retry sending the events to consumers, making the retry action easy to implement.
However, if a consumer receives an event but fails while processing it, there is no direct way to communicate that failure to the producer. Within event-driven architectures and programming, this is a non-issue because the producer is unaware of the consumers and is not responsible for handling their failures.
The event framework managing the queue can retry several times in case the failure is intermittent. If the failure persists, the event can be stored for later processing and proper error messages can be logged.
In event-based programming, determining the state and performance of the system is very straightforward. If the number of events in your queue—also known as queue depth—grows, then you need more consumers. If your consumers are sitting idle, waiting to process events, then your producers are not sending events.
You can find issues by checking which queues grow uncontrollably and which queues starve. Sometimes, it is more useful to track the age of messages in the queue rather than the number of messages.
In PubSub systems, there are primarily three different models for delivery: exactly once, at most once, and at least once.
In this delivery model, every consumer is guaranteed to receive exactly one copy of every event. Actually, this model is very difficult to accomplish. When a consumer fails or disconnects from the event queue, it’s often unknown if the consumer received the event (but failed to acknowledge) or if it didn't even receive the event at all.
Here, some consumers may not get all events, but they will never get duplicates. This model is the easiest to implement. Events are just shotgunned out one time to all of the consumers. Either the consumers get the events or they don’t. No retries.
In this most common model, every consumer is guaranteed to receive every event, but some consumers might also get duplicates. The event framework will retry several times until it gets a confirmation from the consumer. Since duplicate events are possible, the consumer should be idempotent, meaning it can safely process duplicate events.
In pull-based systems, the consumers are responsible for pulling messages from the queue. If the queue retains historical messages, then consumers can pull messages as many times as they want.
Let's examine an event queue with multiple producers and multiple consumers.
The producers generate events and send them to a specific topic. In the diagram, you can see that we have two producers. Producer 1 sends Event A to Topic 1. Producer 2 sends Events B and C to Topics 2 and 3.
Each topic may have a group of subscribers (consumers). This is relevant mostly for PubSub systems. Consumers 1 and 2 subscribe to Topic 1, which creates Group 1 and Group 2 to represent the subscriber list. Consumer 2 also subscribes to Topic 2, which creates Group 3.
When Producer 1 sends Event A to Topic 1, Consumers 1 and 2 will receive it through their subscriber groups. When Producer 2 sends Event B to topic 2, Consumer 2 will receive it through Subscriber Group 3.
When Producer 2 sends Event C, no consumer will receive it since no consumer has subscribed to it. The event may be discarded, or it may be stored in Topic 3, waiting for consumers to subscribe.
Some queues immediately dispose of events once they are delivered to all subscribers. Other queues might keep events around. For those queues, it is useful to configure how long to keep events. This can be done based on event age, the number of events in the queue, or the total size of queued events.
Event-based programming opens the door to several interesting and useful patterns.
The simplest configuration for events is a single producer and single consumer. In this pattern, the producer and consumer are isolated from one another, free to operate independently.
A single producer with multiple consumers is useful when handling an event takes longer than generating an event. In this case, multiple consumers can share the load to handle the events together. A load balancer is a good example of this pattern.
When an event fails to be processed even after several retries, it is common to remove it from the main queue and post it to a dedicated DLQ. Engineers can analyze these events later to determine the root cause.
Systems that generate many events may want to keep a history, but not forever. Generally speaking, the value of data degrades with time. For example, when debugging a production issue, you may want to see the stream of events since the last deployment, but probably not events from five years ago. A common pattern is to assign a TTL to each stream of events or each topic, and those events will be discarded when their TTL expires.
Event handlers typically don't return results. In some cases, it is useful to perform a query or an action asynchronously and then get the result later. Polling is common in these situations, but can introduce a lot of overhead.
An event-based alternative is to establish request topics and response topics. The caller sends a request to the request topic and subscribes to the response topic. When the response is ready, the handler of the original request sends the response to the response topic, and the original caller can process the response.
Event splitting is useful when the same event needs to be handled by multiple entities. This is different from the single producer/multiple consumers pattern, in which one consumer handles each event. For example, consider an event that needs to be processed for storage as well as aggregated for later analytics. One consumer can store the event in the proper data store and another consumer can do the aggregation. This division of responsibilities is the hallmark of loosely-coupled systems. The storage code is unaware of the aggregation code even though both operate on the same data.
The following examples of event-driven programming are from a blocks puzzle game I developed with my sons. You can find the source code for the project at https://github.com/the-gigi/blocktser.
Blocktser is a simple game that runs in a browser. It is implemented in TypeScript, the Canvas API, and the phaser.io framework. In Blocktser, you drag shapes from the staging area at the bottom and move them around the screen to eventually drop them in the main area. You get points when you complete rows or columns. If you can't place a shape, the game is over.
Here is a screenshot:
The shapes are Phaser Game objects. Conveniently, Phaser already translates raw Canvas API mouse events to higher-level events.
To receive and handle events, we need to take several actions. First, we make our shapes draggable by setting the container object to be interactive (setting the mouse cursor) and draggable.
Loading code...
Then, we call the scene input object's on method to subscribe to the drag, dragstart, and dragend events. Triggering (producing) the relevant event calls the provided event handling function (the consumer).
Loading code...
The subscription event handlers delegate the actual logic to the onDragging, onDragStart, and onDragEnd methods of the Shape class.
The Shape's onStartDrag method ignores drag start events on other objects. Then, it performs some logic to scale the shape and update it. (Dragging the shape changes its size.) Finally, it calls onDragStart on a list of dragHandlers. This accomplishes the event splitting pattern we discussed earlier.
Loading code...
The Shape's onDragging method is called whenever the mouse is moving during a drag operation. The event handler here updates the shape's position according to the delta since the last position. Then, it invokes the onDragging method of each of the dragHandlers members.
Loading code...
Last but not least is the Shape's onDragEnd method. Here, we see the familiar structure of ignoring the event if it's coming from a different object, then resizing the shape and invoking the drag handlers.
Loading code...
The event handlers we've seen in the previous section were defined by the Phaser framework, and they were in terms of Phaser game objects and their containers. In Blocktser, there is another layer of abstraction that operates in terms of Blocktser shapes. The interfaces.ts file defines the following interfaces:
Loading code...
Let's focus on the onDrop event. It is triggered when a shape ends its dragging over the main area. First, the onDragEnd method is called:
Loading code...
The onDragEnd method calls the settleShape method, which determines whether or not the shape was dropped properly. Then, it calls the onDrop method with either true or false.
Loading code...
The onDrop handler operates at a higher abstraction level of the scene, which includes both the main area and the staging area of the game. It understands the meaning of a shape being dropped. If the shape was dropped correctly in the main area, then it takes several actions:
If the shape was dropped improperly, then it simply returns to the staging area.
Let's follow the chain of events from the OS level all the way to Blocktser's shape drop event:
Event-based programming is an incredibly useful paradigm, with use cases ranging from UI in a single-process system to communication between services in large-scale distributed systems. The loosely coupled nature of event-based programming provides many benefits, though it may require a shift in how you think about inter-object communication.
With the trend of software development moving away from monolith applications toward distributed systems and decoupled microservices, an event-driven paradigm for architectures and programming is certainly here to stay. The event queue solution you choose to use—whether that be Kafka or any other message broker—is one that requires wise consideration. At the very least, it’s time to consider whether your fleet of microservices will get a boost by shifting toward the event driven development.
Try Aiven's fully managed and hosted Apache Kafka® now with a free 30-day trial and $300 credits to play with.
Get a free trialAbout Gigi Sayfan
Gigi Sayfan is the DevOps team manager at Helix — a bioinformatics and genomics start-up and author of several books and hundreds of technical articles.
Gigi has been developing software professionally for more than 20 years in domains as diverse as instant messaging, morphing, multimedia applications for game consoles, brain-inspired machine learning, custom browser development, web services for 3D distributed game platforms, IoT sensors and virtual reality.
--
Not using Aiven services yet? Sign up now for your free trial at https://console.aiven.io/signup!
In the meantime, make sure you follow our changelog and blog RSS feeds or our LinkedIn and Twitter accounts to stay up-to-date with product and feature-related news.
this._container.setInteractive({ cursor: 'pointer' })
scene.input.setDraggable(this._container);scene.input.on('drag', function(pointer, gameObject, dragX, dragY) {
self.onDragging(self, pointer, gameObject, dragX, dragY)
})
scene.input.on('dragstart', function (pointer, gameObject) {
self.onDragStart(self, pointer, gameObject)
})
scene.input.on('dragend', function (pointer, gameObject) {
self.onDragEnd(self, pointer, gameObject)
})onDragStart(shape, pointer, gameObject) {
if (shape._container !== gameObject) {
return
}
shape._unit *= shape.dragScale
shape.updateShape(true)
shape.dragHandlers.forEach((h) => h.onDragStart(shape))
}onDragging(shape, pointer, gameObject, dragX, dragY) {
if (shape._container !== gameObject) {
return
}
const dx = dragX - gameObject.x
const dy = dragY - gameObject.y
shape._container.x += dx
shape._container.y += dy
shape.dragHandlers.forEach((h) => h.onDragging(shape))
}if (shape._container !== gameObject) {
return
}
shape._unit /= shape.dragScale
shape.updateShape(false)
shape.dragHandlers.forEach((h) => h.onDragEnd(shape))export default interface ShapeDragHandler {
onDragStart: (shape: Shape) => void
onDragEnd: (shape: Shape) => void
onDragging: (shape: Shape) => void
}
export default interface MainEventHandler {
onDrop: (shape: Shape, ok: boolean) => void
}onDragEnd(shape: Shape) {
this.settleShape(shape)
this.destroyPhantom()
}settleShape(shape: Shape) {
// bail out if not on grid
if (!this.isOnGrid(shape)) {
this.shapeEventHandler.onDrop(shape, false)
return
}
// bail out if shape intersects with any occupied cell
const [row, col] = this.findGridLocation(shape)
if (!this.canShapeSettle(shape, row, col)) {
this.shapeEventHandler.onDrop(shape, false)
return
}
// Populate cells with the shape's image
...
this.shapeEventHandler.onDrop(shape, true)
}