Aiven for event streaming

What is event streaming and why is it important?

Event streaming works by moving the events we described above one data point at a time. Instead of treating the data as a whole set (often described as ‘batch’), event streaming will rather ensure that there is a continuous flow of generated data that is being made readily available to your organization.

An event driven architecture breaks everything down into separate microservices that create constant streams of data events. Events, at their core, are simply things that happen divided into topics: an online order, the creation of a user ID, an action in a video game, a shipping update, an IoT sensor reading... These events can be processed (and delivered) in real time.

Go from "what was trending yesterday" to "what’s trending today"

Let's say that the marketing team in your online store needs information on the best selling items every day so they can adjust their communications, and the supply chain team needs it too to order the right quantities from their suppliers on a daily basis. The information needs to be available every morning at 8.00 AM, at the start of the business day.

With a monolithic architecture, you would collect the data from the previous day, store it in a database and then process it overnight for delivery to your teams.

This is not ideal in many ways.

1. The information available to your teams is always up to 24 hours old, potentially making it out of date by the time it’s delivered.

2. If more data is generated overnight (like during discount campaigns or holiday seasons) your system might overload and fail during processing. This may result in delays delivering data to your teams, as well as unhappy suppliers with orders coming in too late.

3. There is no way to find out what products are selling best right now, and order more of them from suppliers.

Let's bring event streaming and an event driven architecture to the rescue.

Instead of waiting for the data to be collected, stored, standardized and processed overnight, at the end of each day, we implement separate loosely connected services for the collection, storing, and processing of the data in real time.

Service 1: Collects the data as soon as it comes in - in 'near real time' - and ensures that is standardized to the formats that the teams need.

Service 2: Stores the data in a persistent way, ensuring the data is imminently available for the team (or teams) that might need it the most.

Service 3: Processes the data in real time--for example, removing personal identifiable information--before it is passed on to the internal teams or the suppliers in order to complete the ordering process.

Benefits of the event streaming model

You now know what’s selling best in near real time.
You can easily make the data available for suppliers to send us the quantities we need, when we need them.
You can bring more sales from ‘what’s trending now’ instead of ‘what was trending yesterday’.

Who can use event streaming architecture?

The event streaming paradigm is perfect for a wide range of modern data applications: services can plug into the stream and access whatever information they need asynchronously, whenever they need it.

Industries like retail and e-commerce, gaming, social media, IoT and many others use event streaming to benefit from a flexible, responsive, scalable and powerful data architecture. The smartest businesses everywhere are building event driven architectures for systems that can handle the increase of real-time data and the needs of modern data infrastructures. This keeps them ready for future growth.

Building blocks of event driven architecture

When an event occurs, it is emitted. A producer creates a record of it and sends it off to the stream where it's logged by an event broker (such as Aiven for Apache Kafka®).

Typically there are two key technology components to event driven architectures:

1. The technology that stores and transports events between producers and consumers

2. The technology used to process the events and perform additional actions.

Events might - or might not - be processed in the stream with a stream processing framework (like Aiven for Apache Flink®) before they are picked up by interested consumers.

infographic-event-data-driven-producers-event-processor-consumer.png

Why event-driven architecture matters

Building better systems by breaking things apart

Event driven architecture isn’t just for streaming data. It brings added efficiency, flexibility, and security to your systems, making them more maintainable, scalable, and robust.

Scalability, efficiency and flexibility

Event-driven architecture is distributed, which helps you build elastic, scalable applications and microservices. You can use your favorite languages and CLIs.

Asynchronicity lets services get on with their work no matter what’s happening elsewhere in the system. During high loads, you can add parallel consumers to take up the strain.

The event stream is the beating heart – the single source of truth for your microservices and applications; a solid, reliable foundation for whatever you build on top.

Processing event streams in real time

Event-driven architecture is perfect for processing and analytics.

Because events are produced as a stream, multiple different consumers can quickly access that information and analyze it, process it, and react to it in real time – or whenever they need to. Consumers can also produce processed data events for consumers further downstream.

It’s also possible to turn static data from more traditional databases into streamed data using Kafka connectors or Change Data Capture (CDC) – so you can get the best of both worlds.

Monoliths vs. microservices

In a traditional monolithic architecture, one database might handle everything. As a result, a small failure can potentially bring the entire system down.

In an event driven architecture, everything is broken up into microservices. They are smaller, independent pieces that communicate with each other and pass information as needed.

When you decouple the producers and consumers from the events, events are always available in the event stream for any microservice that needs them.

▶

"We’re really, really happy with the approach that we took, and that we chose Aiven as the provider."

Nicolas Chiu

Lead Software Developer at JobCloud

Read case study

Aiven for event driven architecture

Open source tools for better applications

Aiven offers a complete ecosystem of technologies around event streaming and Apache Kafka®. You can use them in combination to create a stable, secure, and responsive event streaming framework and event-driven architecture for any application.

Use our fully managed services to build microservices that are flexible, scalable, and easy to integrate.

Scalability

Increase (or decrease) your storage, switch providers and regions, and add services at will – and only pay for what you use.

Familiar toolset

Build your EDA solution using the open source tools and languages you already use

Hassle-free infrastructure

Offload your infrastructure concerns to Aiven and focus on implementing your EDA and developing apps

Secure and stable

With 99.99% uptime, full compliance, and backups built in, your systems – and your users’ data – is safe.

Building event driven architecture – with Aiven

Aiven’s complete ecosystem of technologies around event streaming and Apache Kafka help you build your ideal event-driven architecture.

Place Aiven for Apache Kafka as the core event streaming framework. Then throw in Aiven for Apache Flink for real-time stream processing on your data streams; and Aiven for Apache Kafka Connect to connect your event driven architecture to external sources and systems in your organizations.

With Aiven for Apache Kafka MirrorMaker 2 you can set up cluster to cluster data replication, disaster recovery and geo proximity across multiple regions.

And finally, with Klaw you can implement granular data security and watertight data governance standards across teams.

Streaming engine

Apache Kafka®
Apache Kafka forms the beating heart of your event stream, with integrated management and operational tools like Terraform, Kubernetes, CLI and API support.

Real-time event processing

Apache Flink®
Build an event-streaming powerhouse with Kafka and Flink, and benefit from real-time stream processing and analytics with familiar SQL language support.

Event sourcing / sink connections

Apache Kafka Connect®
With over 30 open source connectors, bring data into Apache Kafka from many popular external sources, or sink your data streams to many other systems (like OpenSearch and more).

Event replication

Apache Kafka MirrorMaker 2®
Easily enable cluster to cluster data replication, disaster recovery and geo proximity across multiple regions.

Data governance

Klaw
Implement granular security and data governance standards across multiple teams in your organization.

Want to know more about Apache Kafka®? We've got what you need.

Apache Kafka® is the perfect reference tool when building event driven architecture and fitting microservices into the big picture. Here's everything you always wanted to know, but were afraid to ask.

Get our free ebook

Store, search and analyze

Our supported connectors allow you to deploy PostgreSQL, MySQL or Cassandra as sources, and OpenSearch, Cassandra, InfluxDB and Redis as sinks. You can also connect to OpenSearch Dashboards to visualize your data.

Our fully managed services are based on open source technologies:

Open source services

Aiven for

Apache Cassandra®

Aiven for

MySQL

Aiven for

OpenSearch®

Aiven for

PostgreSQL®

Aiven for

Redis®

Aiven for

InfluxDB®

Event collaboration

Event-driven architecture for retail

Retail and eCommerce are examples of industries where multiple microservices interact collaboratively as part of a single business workflow.

Some services (orders) create events, and others (payments, shipping) consume them. But the consumers can also process data and create new events for handling down the line (e.g. sales recommendations).
Asynchronicity ensures that no single service controls the entire process; events can be handled as and when needed by independent services, resulting in a faster and more orchestrated architecture.

Events can be associated with each other through the use of topics – for example, orders, payments, or shipping. In this way it might be possible to combine multiple orders or shipments, and reduce costs.
Maintenance or updating of services is simplified because individual microservices can be updated or modified independently, without taking the entire system offline.

Event processing

Fraudulent transaction detection and alerting

The power of event-driven architecture lies in event processing – the ability to pick data from the stream on the fly, process it, store it for later, or deliver real-time analytics. These processes are essential for the real-time detection of fraudulent activities.

Aiven for Apache Kafka forms the core of the system, creating an event stream of all customer banking transactions.
Aiven for Apache Flink grabs customer details from PostgreSQL, and identifies potential fraudulent activities in real time.

OpenSearch stores fraudulent transactions, M3 creates a transaction history, and notification apps alert customers at risk.
OpenSearch Dashboards used to visualize fraudulent transactions and help identify patterns for further investigation.