Advanced parameters for Aiven for Apache Kafka® Inkless

View the configuration options for Aiven for Apache Kafka® Inkless. These configurations apply to Inkless Kafka on Aiven Cloud.

backup_interval_hours integer,null min: 3 max: 24 enum: 3,4,6,8,12,24,null Interval in hours between automatic backups. Minimum value is 3 hours. Must be a divisor of 24 (3, 4, 6, 8, 12, 24). (Applicable to ACU plans only)

backup_retention_days integer,null min: 1 max: 30 Number of days to retain automatic backups. Backups older than this value will be automatically deleted. (Applicable to ACU plans only)

custom_domain string,null maxLength: 255 Serve the web frontend using a custom CNAME pointing to the Aiven DNS name. When you set a custom domain for a service deployed in a VPC, the service certificate is only created for the public-* hostname and the custom domain.

ip_filter array[string,object] maxItems: 8000 default: 0.0.0.0/0,::/0 Allow incoming connections from CIDR address block, e.g. '10.20.0.0/16'

ip_filter.[0].description string maxLength: 1024 Description for IP filter list entry

ip_filter.[0].network string maxLength: 43 CIDR address block

service_log boolean,null Store logs for the service so that they are available in the HTTP API and console.

static_ips boolean Use static public IP addresses

single_zone object Single-zone configuration

single_zone.enabled boolean Whether to allocate nodes on the same Availability Zone or spread across zones available. By default service nodes are spread across different AZs. The single AZ support is best-effort and may temporarily allocate nodes in different AZs e.g. in case of capacity limitations in one AZ.

single_zone.availability_zone string maxLength: 40 The availability zone to use for the service. This is only used when enabled is set to true. If not set the service will be allocated in random AZ.The AZ is not guaranteed, and the service may be allocated in a different AZ if the selected AZ is not available. Zones will not be validated and invalid zones will be ignored, falling back to random AZ selection. Common availability zones include: AWS (euc1-az1, euc1-az2, euc1-az3), GCP (europe-west1-a, europe-west1-b, europe-west1-c), Azure (germanywestcentral/1, germanywestcentral/2, germanywestcentral/3).

private_access object Allow access to selected service ports from private networks

private_access.kafka boolean Allow clients to connect to kafka with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations

private_access.kafka_connect boolean Allow clients to connect to kafka_connect with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations

private_access.kafka_rest boolean Allow clients to connect to kafka_rest with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations

private_access.prometheus boolean Allow clients to connect to prometheus with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations

private_access.schema_registry boolean Allow clients to connect to schema_registry with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations

public_access object Allow access to selected service ports from the public Internet

public_access.kafka boolean Allow clients to connect to kafka from the public internet for service nodes that are in a project VPC or another type of private network

public_access.kafka_connect boolean Allow clients to connect to kafka_connect from the public internet for service nodes that are in a project VPC or another type of private network

public_access.kafka_rest boolean Allow clients to connect to kafka_rest from the public internet for service nodes that are in a project VPC or another type of private network

public_access.prometheus boolean Allow clients to connect to prometheus from the public internet for service nodes that are in a project VPC or another type of private network

public_access.schema_registry boolean Allow clients to connect to schema_registry from the public internet for service nodes that are in a project VPC or another type of private network

privatelink_access object Allow access to selected service components through Privatelink

privatelink_access.jolokia boolean Enable jolokia

privatelink_access.kafka boolean Enable kafka

privatelink_access.kafka_connect boolean Enable kafka_connect

privatelink_access.kafka_rest boolean Enable kafka_rest

privatelink_access.prometheus boolean Enable prometheus

privatelink_access.schema_registry boolean Enable schema_registry

letsencrypt_sasl boolean,null Use a Let's Encrypt certificate authority (CA) for Kafka SASL authentication. (Default: False)

letsencrypt_sasl_privatelink boolean,null Use a Let's Encrypt certificate authority (CA) for Kafka SASL authentication via Privatelink. (Default: False)

kafka object default: [object Object] Kafka broker configuration values

kafka.message_max_bytes integer min: 0 max: 20971520 The maximum size of message that the server can receive. (Default: 1048588 bytes (1 mebibyte + 12 bytes))

kafka.log_local_retention_ms integer min: 900000 The number of milliseconds to keep the local log segments before it gets eligible for deletion. If set to -2, the value of log.retention.ms is used. The effective value should always be less than or equal to log.retention.ms value. (Default: -2)

kafka.log_local_retention_bytes integer min: 5368709120 The maximum size of local log segments that can grow for a partition before it gets eligible for deletion. If set to -2, the value of log.retention.bytes is used. The effective value should always be less than or equal to log.retention.bytes value. (Default: -2)

kafka.log_retention_bytes integer min: -1 max: 9223372036854775807 The maximum size of the log before deleting messages (Default: -1)

kafka.log_retention_hours integer min: -1 max: 9223372036854775807 The number of hours to keep a log file before deleting it. Use -1 for unlimited retention or 1 or higher. Setting 0 is invalid and prevents Kafka from starting. (Default: 168 hours, or 1 week)

kafka.log_retention_ms integer range: -1 or between 60000 and 9223372036854775807 min: -1 The number of milliseconds to keep a log file before deleting it (in milliseconds), If not set, the value in log.retention.minutes is used. If set to -1, no time limit is applied. (Default: null, log.retention.hours applies)

kafka.auto_create_topics_enable boolean Enable auto-creation of topics. (Default: false)

kafka.num_partitions integer min: 1 max: 2048 Number of partitions for auto-created topics (Default: 1)

kafka.default_replication_factor integer min: 1 max: 3 Replication factor for auto-created topics (Default: 3)

kafka.sasl_oauthbearer_expected_audience string The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. (Default: null)

kafka.sasl_oauthbearer_expected_issuer string Optional setting for the broker to use to verify that the JWT was created by the expected issuer.(Default: null)

kafka.sasl_oauthbearer_jwks_endpoint_url string OIDC JWKS endpoint URL. By setting this the SASL SSL OAuth2/OIDC authentication is enabled. See also other options for SASL OAuth2/OIDC. (Default: null)

kafka_authentication_methods object Kafka authentication methods

kafka_authentication_methods.certificate boolean default: true Enable certificate/SSL authentication

kafka_authentication_methods.sasl boolean Enable SASL authentication

kafka_sasl_mechanisms object Kafka SASL mechanisms

kafka_sasl_mechanisms.plain boolean default: true Enable PLAIN mechanism

kafka_sasl_mechanisms.scram_sha_256 boolean default: true Enable SCRAM-SHA-256 mechanism

kafka_sasl_mechanisms.scram_sha_512 boolean default: true Enable SCRAM-SHA-512 mechanism

follower_fetching object Enable follower fetching

follower_fetching.enabled boolean Whether to enable the follower fetching functionality

kafka_connect boolean Enable Kafka Connect service

kafka_connect_config object Kafka Connect configuration values

kafka_connect_config.prefer_ipv6_address_enable boolean When enabled, connectors will automatically resolve IPv6 addresses from external server names configured with dual-stack.

kafka_connect_config.connector_client_config_override_policy string enum: None,All Defines what client configurations can be overridden by the connector. Default is None

kafka_connect_config.consumer_auto_offset_reset string enum: earliest,latest What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server. Default is earliest

kafka_connect_config.consumer_fetch_max_bytes integer min: 1048576 max: 104857600 Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not a absolute maximum.

kafka_connect_config.consumer_isolation_level string enum: read_uncommitted,read_committed Transaction read isolation level. read_uncommitted is the default, but read_committed can be used if consume-exactly-once behavior is desired.

kafka_connect_config.consumer_max_partition_fetch_bytes integer min: 1048576 max: 104857600 Records are fetched in batches by the consumer.If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress.

kafka_connect_config.consumer_max_poll_interval_ms integer min: 1 max: 2147483647 The maximum delay in milliseconds between invocations of poll() when using consumer group management (defaults to 300000).

kafka_connect_config.consumer_max_poll_records integer min: 1 max: 10000 The maximum number of records returned in a single call to poll() (defaults to 500).

kafka_connect_config.offset_flush_interval_ms integer min: 1 max: 100000000 The interval at which to try committing offsets for tasks (defaults to 60000).

kafka_connect_config.offset_flush_timeout_ms integer min: 1 max: 2147483647 Maximum number of milliseconds to wait for records to flush and partition offset data to be committed to offset storage before cancelling the process and restoring the offset data to be committed in a future attempt (defaults to 5000).

kafka_connect_config.producer_batch_size integer min: 0 max: 5242880 This setting gives the upper bound of the batch size to be sent. If there are fewer than this many bytes accumulated for this partition, the producer will 'linger' for the linger.ms time waiting for more records to show up. A batch size of zero will disable batching entirely (defaults to 16384).

kafka_connect_config.producer_buffer_memory integer min: 5242880 max: 134217728 The total bytes of memory the producer can use to buffer records waiting to be sent to the broker (defaults to 33554432).

kafka_connect_config.producer_compression_type string enum: gzip,snappy,lz4,zstd,none Specify the default compression type for producers. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'none' which is the default and equivalent to no compression.

kafka_connect_config.producer_linger_ms integer min: 0 max: 5000 This setting gives the upper bound on the delay for batching: once there is batch.size worth of records for a partition it will be sent immediately regardless of this setting, however if there are fewer than this many bytes accumulated for this partition the producer will 'linger' for the specified time waiting for more records to show up. Defaults to 0.

kafka_connect_config.producer_max_request_size integer min: 131072 max: 67108864 This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests.

kafka_connect_config.scheduled_rebalance_max_delay_ms integer min: 0 max: 600000 The maximum delay that is scheduled in order to wait for the return of one or more departed workers before rebalancing and reassigning their connectors and tasks to the group. During this period the connectors and tasks of the departed workers remain unassigned. Defaults to 5 minutes.

kafka_connect_config.session_timeout_ms integer min: 1 max: 2147483647 The timeout in milliseconds used to detect failures when using Kafka’s group management facilities (defaults to 10000).

kafka_connect_plugin_versions array[object] The plugin selected by the user

kafka_connect_plugin_versions.[0].plugin_name string maxLength: 128 The name of the plugin

kafka_connect_plugin_versions.[0].version string maxLength: 128 The version of the plugin

kafka_connect_secret_providers array[object] Configure external secret providers in order to reference external secrets in connector configuration. Currently Hashicorp Vault (provider: vault, auth_method: token) and AWS Secrets Manager (provider: aws, auth_method: credentials) are supported. Secrets can be referenced in connector config with <provider_name>:<secret_path>:<key_name>

kafka_connect_secret_providers.[0].name string Name of the secret provider. Used to reference secrets in connector config.

kafka_connect_secret_providers.[0].vault object required: auth_method,address Vault secret provider configuration

kafka_connect_secret_providers.[0].vault.auth_method string enum: token An enumeration.

kafka_connect_secret_providers.[0].vault.address string maxLength: 65536 Address of the Vault server

kafka_connect_secret_providers.[0].vault.engine_version integer enum: 1,2 An enumeration.

kafka_connect_secret_providers.[0].vault.prefix_path_depth integer Prefix path depth of the secrets Engine. Default is 1. If the secrets engine path has more than one segment it has to be increased to the number of segments.

kafka_connect_secret_providers.[0].vault.token string maxLength: 256 Token used to authenticate with vault and auth method `token`.

kafka_connect_secret_providers.[0].vault.server_pem string maxLength: 4096 PEM encoded certificate of the Vault server. Required if the vault server uses a self-signed certificate.

kafka_connect_secret_providers.[0].aws object required: auth_method,region AWS secret provider configuration

kafka_connect_secret_providers.[0].aws.auth_method string enum: credentials An enumeration.

kafka_connect_secret_providers.[0].aws.access_key string maxLength: 128 Access key used to authenticate with aws

kafka_connect_secret_providers.[0].aws.secret_key string maxLength: 128 Secret key used to authenticate with aws

kafka_connect_secret_providers.[0].aws.region string maxLength: 64 Region used to lookup secrets with AWS SecretManager

kafka_connect_secret_providers.[0].env object required: secrets ENV secret provider configuration

kafka_connect_secret_providers.[0].env.secrets object Key/value map of secrets for ENV secret provider

kafka_diskless object required: enabled Kafka Diskless configuration values

kafka_diskless.enabled boolean Whether to enable the Diskless functionality

inkless object required: enabled Inkless configuration values

inkless.enabled boolean Whether to enable the Inkless functionality

sasl_oauthbearer_allowed_urls array[string] List of allowed URLs for SASL OAUTHBEARER authentication. Only HTTPS URLs are allowed for security reasons.

gcp_auth_allowed_urls array[string] Allow-list of HTTPS URLs used to validate GCP credential_source requests for Kafka Connect.

kafka_rest boolean Enable Kafka-REST service

kafka_version string,null enum: 3.8,3.9,4.0,4.1 Kafka major version

schema_registry boolean Enable Schema-Registry service

kafka_rest_authorization boolean Enable authorization in Kafka-REST service

kafka_rest_config object Kafka REST configuration

kafka_rest_config.producer_acks string enum: all,-1,0,1 default: 1 The number of acknowledgments the producer requires the leader to have received before considering a request complete. If set to 'all' or '-1', the leader will wait for the full set of in-sync replicas to acknowledge the record.

kafka_rest_config.producer_compression_type string enum: gzip,snappy,lz4,zstd,none Specify the default compression type for producers. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'none' which is the default and equivalent to no compression.

kafka_rest_config.producer_linger_ms integer min: 0 max: 5000 default: 0 Wait for up to the given delay to allow batching records together

kafka_rest_config.producer_max_request_size integer min: 0 max: 2147483647 default: 1048576 The maximum size of a request in bytes. Note that Kafka broker can also cap the record batch size.

kafka_rest_config.consumer_enable_auto_commit boolean default: true If true the consumer's offset will be periodically committed to Kafka in the background

kafka_rest_config.consumer_idle_disconnect_timeout integer min: 0 max: 2147483647 default: 0 Specifies the maximum duration (in seconds) a client can remain idle before it is deleted. If a consumer is inactive, it will exit the consumer group, and its state will be discarded. A value of 0 (default) indicates that the consumer will not be disconnected automatically due to inactivity.

kafka_rest_config.consumer_request_max_bytes integer min: 0 max: 671088640 default: 67108864 Maximum number of bytes in unencoded message keys and values by a single request

kafka_rest_config.consumer_request_timeout_ms integer min: 1000 max: 30000 enum: 1000,15000,30000 default: 1000 The maximum total time to wait for messages for a request if the maximum number of messages has not yet been reached

kafka_rest_config.name_strategy string enum: topic_name,record_name,topic_record_name default: topic_name Name strategy to use when selecting subject for storing schemas

kafka_rest_config.name_strategy_validation boolean default: true If true, validate that given schema is registered under expected subject name by the used name strategy when producing messages.

kafka_rest_config.simpleconsumer_pool_size_max integer min: 10 max: 250 default: 25 Maximum number of SimpleConsumers that can be instantiated per broker

tiered_storage object Tiered storage configuration

tiered_storage.enabled boolean Whether to enable the tiered storage functionality

schema_registry_config object Schema Registry configuration

schema_registry_config.topic_name string maxLength: 249 The durable single partition topic that acts as the durable log for the data. This topic must be compacted to avoid losing data due to retention policy. Please note that changing this configuration in an existing Schema Registry / Karapace setup leads to previous schemas being inaccessible, data encoded with them potentially unreadable and schema ID sequence put out of order. It's only possible to do the switch while Schema Registry / Karapace is disabled. Defaults to `_schemas`.

schema_registry_config.leader_eligibility boolean If true, Karapace / Schema Registry on the service nodes can participate in leader election. It might be needed to disable this when the schemas topic is replicated to a secondary cluster and Karapace / Schema Registry there must not participate in leader election. Defaults to `true`.

schema_registry_config.schema_reader_strict_mode boolean If enabled, causes the Karapace schema-registry service to shutdown when there are invalid schema records in the `_schemas` topic. Defaults to `false`.

schema_registry_config.retriable_errors_silenced boolean If enabled, kafka errors which can be retried or custom errors specified for the service will not be raised, instead, a warning log is emitted. This will denoise issue tracking systems, i.e. sentry. Defaults to `true`.

aiven_kafka_topic_messages boolean Allow access to read Kafka topic messages in the Aiven Console and REST API.

enable_ipv6 boolean Register AAAA DNS records for the service, and allow IPv6 packets to service ports