Skip to main content

Get started with Aiven for Apache Kafka®

Start using Aiven for Apache Kafka® by setting up and configuring a service, connecting to it, and managing your data.

Prerequisites​

Before you begin, ensure you have access to the following tools:

Create an Aiven for Apache Kafka® service​

You can create your Aiven for Apache Kafka service using the Aiven Console, the Aiven CLI, or automate the process with Terraform.

  1. From your project, in the Services page, click Create service.

  2. From the Select service page, click Apache Kafka®.

  3. Select the cloud provider and region to host your service on.

    note

    The pricing for the same service can vary between different providers and regions. The service summary shows you the pricing for your selected options.

  4. Select a service plan.

    note

    This determines the number of servers and the memory, CPU, and disk resources allocated to your service. See Plans & Pricing.

  5. Optional: Add disk storage.

  6. Enter a name for your service.

    important

    You cannot change the name after you create the service.

  7. Optional: Add tags.

  8. Click Create service.

The Overview page of the service opens. It shows the connection parameters for your service, its current status, and the configuration options.

The status of the service is Rebuilding during its creation. When the status becomes Running, you can start using the service. This typically takes couple of minutes and can vary between cloud providers and regions.

Create an Apache Kafka® topic​

Once your service is created, you can add topics for organizing and managing your messages.

  1. Log in to the Aiven Console and select the Aiven for Apache Kafka® service.
  2. Click Topics in the sidebar.
  3. Click Create topic and enter a name for the topic.
  4. If required, set the advanced configuration option to Yes.
  5. Set properties like replication factor, number of partitions, and other settings. These can be changed later.
  6. Click Create topic.

Connect to your Aiven for Apache Kafka service​

You can connect to your Aiven for Apache Kafka service to interact with Apache Kafka topics, allowing you to produce and consume messages. Use Quick connect in the Aiven Console to guide the process.

  1. Log in to the Aiven Console, select your project, and the Aiven for Apache Kafka service.

  2. On the Overview page, click Quick connect.

  3. In the Connect window, select your preferred tool or language from the drop-down list.

  4. Follow the connection instructions. You may need to download:

    • CA certificate: Required for secure communication.
    • Access certificate and Access key: Needed for client certificate authentication.

  5. Select your authentication method based on your environment:

    • Client certificate: Provides secure communication using SSL/TLS certificates. For information on downloading CA certificates, see TLS/SSL certificates.
    • SASL: Provides authentication using Simple Authentication and Security Layer (SASL).

  6. Click Done.

Produce and consume messages​

After connecting, use your preferred Apache Kafka client to produce and consume messages.

Produce messages​

To produce messages using Python:

from kafka import KafkaProducer

# Set up the Kafka producer
producer = KafkaProducer(
    bootstrap_servers='your-kafka-host:port',  # Replace with your Kafka service host and port
    security_protocol='SSL',
    ssl_cafile='path/to/ca.pem',
    ssl_certfile='path/to/service.cert',
    ssl_keyfile='path/to/service.key'
)

# Send a test message to a topic
producer.send('your-topic', b'This is a test message')
producer.flush()
producer.close()

Consume messages​

To consume messages using Python:

from kafka import KafkaConsumer

# Set up the Kafka consumer
consumer = KafkaConsumer(
    'your-topic',  # Replace with your topic name
    bootstrap_servers='your-kafka-host:port',  # Replace with your Kafka service host and port
    security_protocol='SSL',
    ssl_cafile='path/to/ca.pem',
    ssl_certfile='path/to/service.cert',
    ssl_keyfile='path/to/service.key',
    auto_offset_reset='earliest'
)

# Consume and print messages from the topic
for message in consumer:
    print(f"Received message: {message.value}")

Integrate Aiven for Apache Kafka with external systems​

Aiven for Apache Kafka supports a wide range of integrations, allowing you to connect with external systems. Key integration options include:

You can configure these integrations in the Aiven Console. In your project, click Integration endpoints to efficient connect your Apache Kafka service with other platforms and tools.

Configure Aiven for Apache Kafka® Connect​

Aiven for Apache Kafka® Connect integrates your Apache Kafka service with various data sources and sinks, enabling you to build and manage data pipelines.

note

Make sure your Aiven for Apache Kafka service is running on a business or premium plan.

You can deploy Aiven for Apache Kafka Connect in two ways:

Enable Karapace​

Karapace, an Aiven-built open-source schema registry for Apache Kafka®, provides schema management and REST APIs for interacting with Aiven for Apache Kafka.

  1. Enable Karapace schema registry and REST APIs in your Kafka service.
  2. Use Karapace to register, update, and version control your schemas.

Learn more about the Karapace schema registry.

View topic catalog Early availability​

The Aiven for Apache Kafka topic catalog provides a centralized interface to view, request, and manage topics across projects. It simplifies topic management and ensures governance within your Aiven for Apache Kafka infrastructure.

  1. Log in to the Aiven Console.
  2. Click Tools.
  3. Select Apache Kafka topic catalog.

Learn more about the Aiven for Apache Kafka® topic catalog.

Enable governance Limited availability​

Governance in Aiven for Apache Kafka provides a secure and efficient way to manage your Apache Kafka clusters. It centralizes control through policies and roles, ensuring compliance and enforcing standards across your Aiven for Apache Kafka environments.

note

You need super admin permissions to enable governance.

  1. Access the Aiven Console and click Admin.
  2. On the organization page, click Apache Kafka governance.
  3. Click Enable governance.

Learn more about the Aiven for Apache Kafka® governance overview.

Related pages