Aiven for Apache Flink® concepts

Service architecture

At a high level, Flink has a runtime architecture consisting of two types of processes: a JobManager and one or more TaskManager.

Limitations

Because Aiven for Apache Flink is a fully managed service, there are differences between Aiven for Apache Flink and an Apache Flink service you run yourself.

Applications

An Aiven for Apache Flink® Application is an abstraction layer that simplifies building data processing pipelines in Apache Flink.

Built-in SQL editor

The built-in Table SQL editor in the [Aiven

Custom JARs

Aiven for Apache Flink enables you to upload, deploy, and manage your own Java code as custom JARs within a JAR application.

Tables

With Aiven for Apache Flink®, you can create and manage data pipelines using Flink tables.

Checkpoints

Checkpoints in Aiven for Apache Flink® are a key feature for ensuring resiliency and fault tolerance in stateful functions.

Savepoints

Savepoints in Aiven for Apache Flink® are snapshots of the current state

Event time refers to when events occur, and processing time is when a system observes or processes these events. Understanding the difference between these two is essential for data processing and streaming. It affects data handling, analysis, and storage.

Watermarks

Apache Flink® uses watermarks to synchronize and process events in data streams accurately. These watermarks are timestamps embedded in the data stream that track the progression of event time.

Windows

Apache Flink® uses the concept of windows to manage the continuous flow of data in streams by segmenting it into manageable chunks. This approach is essential due to the continuous and unbounded nature of data streams, where waiting for all data to arrive is impractical.

Standard and upsert connectors for Apache Kafka®

In addition to integration with Apache Kafka® through a standard

Settings for Apache Kafka® connectors

Explore the necessary settings for standard and upsert Kafka connectors in Aiven for Apache Flink®.