Periodic high consumer lag from OpenSearch Connector for Apache Kafka?

Aiven Community
1

Hi,

we are leveraging the Aiven-Open OpenSearch Connector for Apache Kafka to push messages from Kafka topics to their OpenSearch ‘counterpart’ indices.

Burrow (external lag reporter) as well as our Kafka brokers report periodic high consumer group lag for the OpenSearch connector consumer group as seen in the attached

screenshot
screenshot1762×658 58.7 KB
.

Our connector config looks like this:

  config:
    batch.size: 1000
    behavior.on.malformed.documents: warn
    behavior.on.null.values: delete
    behavior.on.version.conflict: warn
    connection.password: <redacted>
    connection.url: https://opensearch-nodes.opensearch.svc:9200
    connection.username: <redacted>
    errors.deadletterqueue.context.headers.enable: true
    errors.deadletterqueue.topic.name: <redacted>.index.dl
    errors.deadletterqueue.topic.replication.factor: 3
    errors.tolerance: all
    flush.timeout.ms: 10000
    index.write.method: upsert
    key.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    key.ignore: false
    key.ignore.id.strategy: record.key
    max.buffered.records: 25000
    max.in.flight.requests: 1
    read.timeout.ms: 7500
    schema.ignore: "true"
    topics: <redacted>.person.document,<redacted>.folder.document
    transforms: extractKey
    transforms.extractKey.field: id
    transforms.extractKey.type: org.apache.kafka.connect.transforms.ExtractField$Key
    type.name: _doc
    value.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter.schemas.enable: false
  tasksMax: 12

Is the observed consumer lag somehow related to batching and / or buffering? Is it actually an issue or rather normal / expected?

I’d be grateful for any hints.