Create a stream reactor source connector from Apache Cassandra® to Apache Kafka®
The Apache Cassandra® stream reactor source connector enables you to move data from a Apache Cassandra® database to an Aiven for Apache Kafka® cluster. The Lenses.io implementation enables you to write KCQL transformations on the topic data before sending it to the Apache Kafka cluster.
See the full set of available parameters and configuration options in the connector's documentation.
Prerequisites
To setup a Apache Cassandra source connector, you need an Aiven for Apache Kafka service with Kafka Connect enabled or a dedicated Aiven for Apache Kafka Connect cluster.
Also collect the following information about the target Cassandra database upfront:
CASSANDRA_HOSTNAME
: The Cassandra hostnameCASSANDRA_PORT
: The Cassandra portCASSANDRA_USERNAME
: The Cassandra usernameCASSANDRA_PASSWORD
: The Cassandra passwordCASSANDRA_SSL
: The Cassandra SSL setting, can betrue
,false
ordefault
CASSANDRA_KEYSTORE
: The path to the Keystore containing the CA certificate, used when connection is secured via SSLCASSANDRA_KEYSTORE_PASSWORD
: The Keystore password, used when connection is secured via SSL
If you're using Aiven for Apache Cassandra, you can use the following keystore values
CASSANDRA_TRUSTSTORE
:/run/aiven/keys/public.truststore.jks
CASSANDRA_TRUSTSTORE_PASSWORD
:password
-
CASSANDRA_KEYSPACE
: The Cassandra keyspace to use to source the data from -
KCQL_TRANSFORMATION
: The KCQL syntax to parse the topic data, should be in the format:INSERT INTO APACHE_KAFKA_TOPIC
SELECT LIST_OF_FIELDS
FROM CASSANDRA_TABLE
[PK CASSANDRA_TABLE_COLUMN]
[INCREMENTAL_MODE=MODE]warningBy default the connector acts in bulk mode, extracting all the rows from the Cassandra table on a polling interval and pushing them to the Apache Kafka topic. You can however define incremental options by defining the incremental mode and primary key.
-
APACHE_KAFKA_HOST
: The hostname of the Apache Kafka service, only needed when using Avro as data format -
SCHEMA_REGISTRY_PORT
: The Apache Kafka's schema registry port, only needed when using Avro as data format -
SCHEMA_REGISTRY_USER
: The Apache Kafka's schema registry username, only needed when using Avro as data format -
SCHEMA_REGISTRY_PASSWORD
: The Apache Kafka's schema registry user password, only needed when using Avro as data format
If you're using Aiven for Cassandra and Aiven for Apache Kafka, the
above details are available in the Aiven
console service Overview tab or via the
dedicated avn service get
command with the
Aiven CLI.
The SCHEMA_REGISTRY
related parameters are available in the Aiven for
Apache Kafka® service page, Overview tab, and Schema Registry subtab
As of version 3.0, Aiven for Apache Kafka no longer supports Confluent Schema Registry. For more information, read the article describing the replacement, Karapace
Setup an Apache Cassandra source connector with Aiven Console
The following example demonstrates how to setup an Apache Cassandra source connector for Apache Kafka using the Aiven Console.
Define a Kafka Connect configuration file
Define the connector configurations in a file (we'll refer to it with
the name cassandra_source.json
) with the following content, creating a
file is not strictly necessary but allows to have all the information in
one place before copy/pasting them in the Aiven
Console:
{
"name":"CONNECTOR_NAME",
"connector.class": "com.datamountaineer.streamreactor.connect.cassandra.source.CassandraSourceConnector",
"connect.cassandra.host": "CASSANDRA_HOSTNAME",
"connect.cassandra.port": "CASSANDRA_PORT",
"connect.cassandra.username": "CASSANDRA_USERNAME",
"connect.cassandra.password": "CASSANDRA_PASSWORD",
"connect.cassandra.ssl.enabled": "CASSANDRA_SSL",
"connect.cassandra.trust.store.path": "CASSANDRA_TRUSTSTORE",
"connect.cassandra.trust.store.password": "CASSANDRA_TRUSTSTORE_PASSWORD",
"connect.cassandra.key.space": "CASSANDRA_KEYSPACE",
"connect.cassandra.kcql": "KCQL_TRANSFORMATION",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "https://APACHE_KAFKA_HOST:SCHEMA_REGISTRY_PORT",
"key.converter.basic.auth.credentials.source": "USER_INFO",
"key.converter.schema.registry.basic.auth.user.info": "SCHEMA_REGISTRY_USER:SCHEMA_REGISTRY_PASSWORD",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "https://APACHE_KAFKA_HOST:SCHEMA_REGISTRY_PORT",
"value.converter.basic.auth.credentials.source": "USER_INFO",
"value.converter.schema.registry.basic.auth.user.info": "SCHEMA_REGISTRY_USER:SCHEMA_REGISTRY_PASSWORD"
}
The configuration file contains the following entries:
name
: the connector name, replaceCONNECTOR_NAME
with the name to give to the connector.connect.cassandra.*
: source parameters collected in the prerequisite phase.key.converter
andvalue.converter
: defines the messages data format in the Apache Kafka topic. Theio.confluent.connect.avro.AvroConverter
converter translates messages from the Avro format. To retrieve the messages schema we use Aiven's Karapace schema registry as specified by theschema.registry.url
parameter and related credentials.
The key.converter
and value.converter
sections define how the topic
messages will be parsed and needs to be included in the connector
configuration.
When using Avro as source data format, set following parameters:
value.converter.schema.registry.url
: pointing to the Aiven for Apache Kafka schema registry URL in the form ofhttps://APACHE_KAFKA_HOST:SCHEMA_REGISTRY_PORT
with theAPACHE_KAFKA_HOST
andSCHEMA_REGISTRY_PORT
parameters retrieved in the previous step.value.converter.basic.auth.credentials.source
: to the valueUSER_INFO
, since you're going to login to the schema registry using username and password.value.converter.schema.registry.basic.auth.user.info
: passing the required schema registry credentials in the form ofSCHEMA_REGISTRY_USER:SCHEMA_REGISTRY_PASSWORD
with theSCHEMA_REGISTRY_USER
andSCHEMA_REGISTRY_PASSWORD
parameters retrieved in the previous step.
Create a Kafka Connect connector with the Aiven Console
To create a Kafka Connect connector:
-
Log in to the Aiven Console and select the Aiven for Apache Kafka® or Aiven for Apache Kafka Connect® service where the connector needs to be defined.
-
Select Connectors from the left sidebar.
-
Select Create New Connector.
noteIt enabled only for services with Kafka Connect enabled.
-
Select Stream Reactor Cassandra Source.
-
In the Common tab, locate the Connector configuration text box and select on Edit.
-
Paste the connector configuration (stored in the
cassandra_source.json
file) in the form. -
Select Apply.
noteThe Aiven Console parses the configuration file and fills the relevant UI fields. You can review the UI fields across the various tab and change them if necessary. The changes will be reflected in JSON format in the Connector configuration text box.
-
After all the settings are correctly configured, select Create new connector.
-
Verify the connector status under the Connectors screen.
-
Verify the presence of the data in the target Cassandra service.
You can also create connectors using the Aiven CLI command.
Example: Create a Cassandra source connector
If you have a Cassandra table named students
in the
students_keyspace
keyspace, with four columns (id
, name
, age
and
timestamp_added
) and you want to load incrementally an Apache Kafka
topic called students_topic
, you can use the following connector
configuration, after replacing the placeholders for CASSANDRA_HOST
,
CASSANDRA_PORT
, CASSANDRA_USERNAME
, CASSANDRA_PASSWORD
,
CASSANDRA_KEYSTORE
, CASSANDRA_KEYSTORE_PASSWORD
,
CASSANDRA_TRUSTSTORE
, CASSANDRA_TRUSTSTORE_PASSWORD
,
CASSANDRA_KEYSPACE
:
{
"name": "my-cassandra-source",
"connector.class": "com.datamountaineer.streamreactor.connect.cassandra.source.CassandraSinkConnector",
"topics": "TOPIC_LIST",
"connect.cassandra.host": "CASSANDRA_HOSTNAME",
"connect.cassandra.port": "CASSANDRA_PORT",
"connect.cassandra.username": "CASSANDRA_USERNAME",
"connect.cassandra.password": "CASSANDRA_PASSWORD",
"connect.cassandra.ssl.enabled": "CASSANDRA_SSL",
"connect.cassandra.trust.store.path": "CASSANDRA_TRUSTSTORE",
"connect.cassandra.trust.store.password": "CASSANDRA_TRUSTSTORE_PASSWORD",
"connect.cassandra.key.space": "students_keyspace",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"connect.cassandra.kcql": "INSERT INTO students_topic SELECT id, name, age, timestamp_added FROM students PK timestamp_added INCREMENTALMODE=TIMESTAMP"
}
The configuration file contains the following peculiarities:
"topics": "students"
: setting the topic to source"connect.cassandra"
: the connection parameters placeholders"value.converter": "org.apache.kafka.connect.json.JsonConverter"
: the topic will be populated in JSON format"connect.cassandra.kcql": "INSERT INTO students_topic SELECT id, name, age, timestamp_added FROM students PK timestamp_added INCREMENTALMODE=TIMESTAMP"
: the connector logic is to insert every row in a new topic message and use the columntimestamp_added
to check for new rows compared to the previous poll.
Once the connector is created successfully, you should see one message
per row in the Cassandra table appearing in the target Apache Kafka
students_topic
topic.
If your Aiven for Apache Kafka instance doesn't have the
automatic creation of topic enabled, you might need to create the students_topic
topic upfront
before starting the connector