Skip to main content

Advanced parameters for Apacheh Kafka® Connect

See the configuration options available for Apache Kafka® Connect:

Parameter

ip_filter

array

  • default: 0.0.0.0/0

IP filter

Allow incoming connections from CIDR address block, e.g. '10.20.0.0/16'

service_log

boolean,null

Service logging

Store logs for the service so that they are available in the HTTP API and console.

static_ips

boolean

Static IP addresses

Use static public IP addresses

kafka_connect

object

Kafka Connect configuration values

kafka_connect.connector_client_config_override_policy

string

Client config override policy

Defines what client configurations can be overridden by the connector. Default is None

kafka_connect.consumer_auto_offset_reset

string

Consumer auto offset reset

What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server. Default is earliest

kafka_connect.consumer_fetch_max_bytes

integer

  • min: 1048576
  • max: 104857600

The maximum amount of data the server should return for a fetch request

Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not a absolute maximum.

kafka_connect.consumer_isolation_level

string

Consumer isolation level

Transaction read isolation level. read_uncommitted is the default, but read_committed can be used if consume-exactly-once behavior is desired.

kafka_connect.consumer_max_partition_fetch_bytes

integer

  • min: 1048576
  • max: 104857600

The maximum amount of data per-partition the server will return.

Records are fetched in batches by the consumer.If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress.

kafka_connect.consumer_max_poll_interval_ms

integer

  • min: 1
  • max: 2147483647

The maximum delay between polls when using consumer group management

The maximum delay in milliseconds between invocations of poll() when using consumer group management (defaults to 300000).

kafka_connect.consumer_max_poll_records

integer

  • min: 1
  • max: 10000

The maximum number of records returned by a single poll

The maximum number of records returned in a single call to poll() (defaults to 500).

kafka_connect.offset_flush_interval_ms

integer

  • min: 1
  • max: 100000000

The interval at which to try committing offsets for tasks

The interval at which to try committing offsets for tasks (defaults to 60000).

kafka_connect.offset_flush_timeout_ms

integer

  • min: 1
  • max: 2147483647

Offset flush timeout

Maximum number of milliseconds to wait for records to flush and partition offset data to be committed to offset storage before cancelling the process and restoring the offset data to be committed in a future attempt (defaults to 5000).

kafka_connect.producer_batch_size

integer

  • max: 5242880

The batch size in bytes the producer will attempt to collect for the same partition before publishing to broker

This setting gives the upper bound of the batch size to be sent. If there are fewer than this many bytes accumulated for this partition, the producer will 'linger' for the linger.ms time waiting for more records to show up. A batch size of zero will disable batching entirely (defaults to 16384).

kafka_connect.producer_buffer_memory

integer

  • min: 5242880
  • max: 134217728

The total bytes of memory the producer can use to buffer records waiting to be sent to the broker

The total bytes of memory the producer can use to buffer records waiting to be sent to the broker (defaults to 33554432).

kafka_connect.producer_compression_type

string

The default compression type for producers

Specify the default compression type for producers. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'none' which is the default and equivalent to no compression.

kafka_connect.producer_linger_ms

integer

  • max: 5000

Wait for up to the given delay to allow batching records together

This setting gives the upper bound on the delay for batching: once there is batch.size worth of records for a partition it will be sent immediately regardless of this setting, however if there are fewer than this many bytes accumulated for this partition the producer will 'linger' for the specified time waiting for more records to show up. Defaults to 0.

kafka_connect.producer_max_request_size

integer

  • min: 131072
  • max: 67108864

The maximum size of a request in bytes

This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests.

kafka_connect.scheduled_rebalance_max_delay_ms

integer

  • max: 600000

The maximum delay of rebalancing connector workers

The maximum delay that is scheduled in order to wait for the return of one or more departed workers before rebalancing and reassigning their connectors and tasks to the group. During this period the connectors and tasks of the departed workers remain unassigned. Defaults to 5 minutes.

kafka_connect.session_timeout_ms

integer

  • min: 1
  • max: 2147483647

The timeout used to detect failures when using Kafka’s group management facilities

The timeout in milliseconds used to detect failures when using Kafka’s group management facilities (defaults to 10000).

secret_providers

array

Kafka Connect secret providers

Configure external secret providers in order to reference external secrets in connector configuration. Currently Hashicorp Vault (provider: vault, auth_method: token) and AWS Secrets Manager (provider: aws, auth_method: credentials) are supported. Secrets can be referenced in connector config with ${<provider_name>:<secret_path>:<key_name>}

private_access

object

Allow access to selected service ports from private networks

private_access.kafka_connect

boolean

Allow clients to connect to kafka_connect with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations

private_access.prometheus

boolean

Allow clients to connect to prometheus with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations

privatelink_access

object

Allow access to selected service components through Privatelink

privatelink_access.jolokia

boolean

Enable jolokia

privatelink_access.kafka_connect

boolean

Enable kafka_connect

privatelink_access.prometheus

boolean

Enable prometheus

public_access

object

Allow access to selected service ports from the public Internet

public_access.kafka_connect

boolean

Allow clients to connect to kafka_connect from the public internet for service nodes that are in a project VPC or another type of private network

public_access.prometheus

boolean

Allow clients to connect to prometheus from the public internet for service nodes that are in a project VPC or another type of private network