Advanced parameters for Aiven for Apache Kafka® Free tier
View the configuration options for Aiven for Apache Kafka® Free tier.
Broker parameters
Interval in hours between automatic backups. Minimum value is 3 hours. Must be a divisor of 24 (3, 4, 6, 8, 12, 24). (Applicable to ACU plans only) |
Number of days to retain automatic backups. Backups older than this value will be automatically deleted. (Applicable to ACU plans only) |
Serve the web frontend using a custom CNAME pointing to the Aiven DNS name. When you set a custom domain for a service deployed in a VPC, the service certificate is only created for the public-* hostname and the custom domain. |
Allow incoming connections from CIDR address block, e.g. '10.20.0.0/16' |
Description for IP filter list entry |
CIDR address block |
Store logs for the service so that they are available in the HTTP API and console. |
Use static public IP addresses |
Single-zone configuration |
Whether to allocate nodes on the same Availability Zone or spread across zones available. By default service nodes are spread across different AZs. The single AZ support is best-effort and may temporarily allocate nodes in different AZs e.g. in case of capacity limitations in one AZ. |
The availability zone to use for the service. This is only used when enabled is set to true. If not set the service will be allocated in random AZ.The AZ is not guaranteed, and the service may be allocated in a different AZ if the selected AZ is not available. Zones will not be validated and invalid zones will be ignored, falling back to random AZ selection. Common availability zones include: AWS (euc1-az1, euc1-az2, euc1-az3), GCP (europe-west1-a, europe-west1-b, europe-west1-c), Azure (germanywestcentral/1, germanywestcentral/2, germanywestcentral/3). |
Allow access to selected service ports from private networks |
Allow clients to connect to kafka with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations |
Allow clients to connect to kafka_connect with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations |
Allow clients to connect to kafka_rest with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations |
Allow clients to connect to prometheus with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations |
Allow clients to connect to schema_registry with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations |
Allow access to selected service ports from the public Internet |
Allow clients to connect to kafka from the public internet for service nodes that are in a project VPC or another type of private network |
Allow clients to connect to kafka_connect from the public internet for service nodes that are in a project VPC or another type of private network |
Allow clients to connect to kafka_rest from the public internet for service nodes that are in a project VPC or another type of private network |
Allow clients to connect to prometheus from the public internet for service nodes that are in a project VPC or another type of private network |
Allow clients to connect to schema_registry from the public internet for service nodes that are in a project VPC or another type of private network |
Allow access to selected service components through Privatelink |
Enable jolokia |
Enable kafka |
Enable kafka_connect |
Enable kafka_rest |
Enable prometheus |
Enable schema_registry |
Use a Let's Encrypt certificate authority (CA) for Kafka SASL authentication. (Default: False) |
Use a Let's Encrypt certificate authority (CA) for Kafka SASL authentication via Privatelink. (Default: False) |
Kafka broker configuration values |
Specify the final compression type for a given topic. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'uncompressed' which is equivalent to no compression; and 'producer' which means retain the original compression codec set by the producer.(Default: producer) |
The amount of time, in milliseconds, the group coordinator will wait for more consumers to join a new group before performing the first rebalance. A longer delay means potentially fewer rebalances, but increases the time until processing begins. The default value for this is 3 seconds. During development and testing it might be desirable to set this to 0 in order to not delay test execution time. (Default: 3000 ms (3 seconds)) |
The minimum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures. (Default: 6000 ms (6 seconds)) |
The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures. Default: 1800000 ms (30 minutes) |
Idle connections timeout: the server socket processor threads close the connections that idle for longer than this. (Default: 600000 ms (10 minutes)) |
The maximum number of incremental fetch sessions that the broker will maintain. (Default: 1000) |
The maximum size of message that the server can receive. (Default: 1048588 bytes (1 mebibyte + 12 bytes)) |
Log retention window in minutes for offsets topic (Default: 10080 minutes (7 days)) |
How long are delete records retained? (Default: 86400000 (1 day)) |
The maximum amount of time message will remain uncompacted. Only applicable for logs that are being compacted. (Default: 9223372036854775807 ms (Long.MAX_VALUE)) |
The minimum time a message will remain uncompacted in the log. Only applicable for logs that are being compacted. (Default: 0 ms) |
The default cleanup policy for segments beyond the retention window (Default: delete) |
The number of messages accumulated on a log partition before messages are flushed to disk (Default: 9223372036854775807 (Long.MAX_VALUE)) |
The maximum time in ms that a message in any topic is kept in memory (page-cache) before flushed to disk. If not set, the value in log.flush.scheduler.interval.ms is used (Default: null) |
The interval with which Kafka adds an entry to the offset index (Default: 4096 bytes (4 kibibytes)) |
The maximum size in bytes of the offset index (Default: 10485760 (10 mebibytes)) |
The number of milliseconds to keep the local log segments before it gets eligible for deletion. If set to -2, the value of log.retention.ms is used. The effective value should always be less than or equal to log.retention.ms value. (Default: -2) |
The maximum size of local log segments that can grow for a partition before it gets eligible for deletion. If set to -2, the value of log.retention.bytes is used. The effective value should always be less than or equal to log.retention.bytes value. (Default: -2) |
Define whether the timestamp in the message is message create time or log append time. (Default: CreateTime) |
The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message. If message.timestamp.type=CreateTime, a message will be rejected if the difference in timestamp exceeds this threshold. Applies only for messages with timestamps earlier than the broker's timestamp. (Default: 9223372036854775807 (Long.MAX_VALUE)) |
The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message. If message.timestamp.type=CreateTime, a message will be rejected if the difference in timestamp exceeds this threshold. Applies only for messages with timestamps later than the broker's timestamp. (Default: 9223372036854775807 (Long.MAX_VALUE)) |
Should pre allocate file when create new segment? (Default: false) |
The maximum size of the log before deleting messages (Default: -1) |
The number of hours to keep a log file before deleting it. Use -1 for unlimited retention or 1 or higher. Setting 0 is invalid and prevents Kafka from starting. (Default: 168 hours, or 1 week) |
The number of milliseconds to keep a log file before deleting it (in milliseconds), If not set, the value in log.retention.minutes is used. If set to -1, no time limit is applied. (Default: null, log.retention.hours applies) |
The maximum jitter to subtract from logRollTimeMillis (in milliseconds). If not set, the value in log.roll.jitter.hours is used (Default: null) |
The maximum time before a new log segment is rolled out (in milliseconds). (Default: null, log.roll.hours applies (Default: 168, 7 days)) |
The maximum size of a single log file (Default: 1073741824 bytes (1 gibibyte)) |
The amount of time to wait before deleting a file from the filesystem (Default: 60000 ms (1 minute)) |
Enable auto-creation of topics. (Default: false) |
When a producer sets acks to 'all' (or '-1'), min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. (Default: 1) |
Number of partitions for auto-created topics (Default: 1) |
Replication factor for auto-created topics (Default: 3) |
The number of bytes of messages to attempt to fetch for each partition . This is not an absolute maximum, if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. (Default: 1048576 bytes (1 mebibytes)) |
Maximum bytes expected for the entire fetch response. Records are fetched in batches, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. As such, this is not an absolute maximum. (Default: 10485760 bytes (10 mebibytes)) |
The maximum number of connections allowed from each ip address (Default: 2147483647). |
The purge interval (in number of requests) of the producer request purgatory (Default: 1000). |
The maximum number of bytes in a socket request (Default: 104857600 bytes). |
The transaction topic segment bytes should be kept relatively small in order to facilitate faster log compaction and cache loads (Default: 104857600 bytes (100 mebibytes)). |
The interval at which to remove transactions that have expired due to transactional.id.expiration.ms passing (Default: 3600000 ms (1 hour)). |
Enable verification that checks that the partition has been added to the transaction before writing transactional records to the partition. (Default: true) |
Kafka authentication methods |
Enable certificate/SSL authentication |
Enable SASL authentication |
Kafka SASL mechanisms |
Enable PLAIN mechanism |
Enable SCRAM-SHA-256 mechanism |
Enable SCRAM-SHA-512 mechanism |
Enable follower fetching |
Whether to enable the follower fetching functionality |
Enable Kafka Connect service |
Kafka Connect configuration values |
When enabled, connectors will automatically resolve IPv6 addresses from external server names configured with dual-stack. |
Defines what client configurations can be overridden by the connector. Default is None |
What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server. Default is earliest |
Records are fetched in batches by the consumer, and if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that the consumer can make progress. As such, this is not a absolute maximum. |
Transaction read isolation level. read_uncommitted is the default, but read_committed can be used if consume-exactly-once behavior is desired. |
Records are fetched in batches by the consumer.If the first record batch in the first non-empty partition of the fetch is larger than this limit, the batch will still be returned to ensure that the consumer can make progress. |
The maximum delay in milliseconds between invocations of poll() when using consumer group management (defaults to 300000). |
The maximum number of records returned in a single call to poll() (defaults to 500). |
The interval at which to try committing offsets for tasks (defaults to 60000). |
Maximum number of milliseconds to wait for records to flush and partition offset data to be committed to offset storage before cancelling the process and restoring the offset data to be committed in a future attempt (defaults to 5000). |
This setting gives the upper bound of the batch size to be sent. If there are fewer than this many bytes accumulated for this partition, the producer will 'linger' for the linger.ms time waiting for more records to show up. A batch size of zero will disable batching entirely (defaults to 16384). |
The total bytes of memory the producer can use to buffer records waiting to be sent to the broker (defaults to 33554432). |
Specify the default compression type for producers. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'none' which is the default and equivalent to no compression. |
This setting gives the upper bound on the delay for batching: once there is batch.size worth of records for a partition it will be sent immediately regardless of this setting, however if there are fewer than this many bytes accumulated for this partition the producer will 'linger' for the specified time waiting for more records to show up. Defaults to 0. |
This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. |
The maximum delay that is scheduled in order to wait for the return of one or more departed workers before rebalancing and reassigning their connectors and tasks to the group. During this period the connectors and tasks of the departed workers remain unassigned. Defaults to 5 minutes. |
The timeout in milliseconds used to detect failures when using Kafka’s group management facilities (defaults to 10000). |
The plugin selected by the user |
The name of the plugin |
The version of the plugin |
kafka_connect_secret_providers
Configure external secret providers in order to reference external secrets in connector configuration. Currently Hashicorp Vault (provider: vault, auth_method: token) and AWS Secrets Manager (provider: aws, auth_method: credentials) are supported. Secrets can be referenced in connector config with <provider_name>:<secret_path>:<key_name> |
Name of the secret provider. Used to reference secrets in connector config. |
Vault secret provider configuration |
An enumeration. |
Address of the Vault server |
An enumeration. |
Prefix path depth of the secrets Engine. Default is 1. If the secrets engine path has more than one segment it has to be increased to the number of segments. |
Token used to authenticate with vault and auth method `token`. |
PEM encoded certificate of the Vault server. Required if the vault server uses a self-signed certificate. |
AWS secret provider configuration |
An enumeration. |
Access key used to authenticate with aws |
Secret key used to authenticate with aws |
Region used to lookup secrets with AWS SecretManager |
ENV secret provider configuration |
Key/value map of secrets for ENV secret provider |
Kafka Diskless configuration values |
Whether to enable the Diskless functionality |
Inkless configuration values |
Whether to enable the Inkless functionality |
List of allowed URLs for SASL OAUTHBEARER authentication. Only HTTPS URLs are allowed for security reasons. |
Allow-list of HTTPS URLs used to validate GCP credential_source requests for Kafka Connect. |
Enable Kafka-REST service |
Kafka major version |
Enable Schema-Registry service |
Enable authorization in Kafka-REST service |
Kafka REST configuration |
The number of acknowledgments the producer requires the leader to have received before considering a request complete. If set to 'all' or '-1', the leader will wait for the full set of in-sync replicas to acknowledge the record. |
Specify the default compression type for producers. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'none' which is the default and equivalent to no compression. |
Wait for up to the given delay to allow batching records together |
The maximum size of a request in bytes. Note that Kafka broker can also cap the record batch size. |
If true the consumer's offset will be periodically committed to Kafka in the background |
Specifies the maximum duration (in seconds) a client can remain idle before it is deleted. If a consumer is inactive, it will exit the consumer group, and its state will be discarded. A value of 0 (default) indicates that the consumer will not be disconnected automatically due to inactivity. |
Maximum number of bytes in unencoded message keys and values by a single request |
The maximum total time to wait for messages for a request if the maximum number of messages has not yet been reached |
Name strategy to use when selecting subject for storing schemas |
If true, validate that given schema is registered under expected subject name by the used name strategy when producing messages. |
Maximum number of SimpleConsumers that can be instantiated per broker |
Tiered storage configuration |
Whether to enable the tiered storage functionality |
Schema Registry configuration |
The durable single partition topic that acts as the durable log for the data. This topic must be compacted to avoid losing data due to retention policy. Please note that changing this configuration in an existing Schema Registry / Karapace setup leads to previous schemas being inaccessible, data encoded with them potentially unreadable and schema ID sequence put out of order. It's only possible to do the switch while Schema Registry / Karapace is disabled. Defaults to `_schemas`. |
If true, Karapace / Schema Registry on the service nodes can participate in leader election. It might be needed to disable this when the schemas topic is replicated to a secondary cluster and Karapace / Schema Registry there must not participate in leader election. Defaults to `true`. |
If enabled, causes the Karapace schema-registry service to shutdown when there are invalid schema records in the `_schemas` topic. Defaults to `false`. |
If enabled, kafka errors which can be retried or custom errors specified for the service will not be raised, instead, a warning log is emitted. This will denoise issue tracking systems, i.e. sentry. Defaults to `true`. |
Allow access to read Kafka topic messages in the Aiven Console and REST API. |
Register AAAA DNS records for the service, and allow IPv6 packets to service ports |
Topic parameters
Specify the final compression type for a given topic. This configuration accepts the standard compression codecs ('gzip', 'snappy', 'lz4', 'zstd'). It additionally accepts 'uncompressed' which is equivalent to no compression; and 'producer' which means retain the original compression codec set by the producer. |
When a producer sets acks to 'all' (or '-1'), this configuration specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend). When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of 'all'. This will ensure that the producer raises an exception if a majority of replicas do not receive a write. |
The amount of time to retain delete tombstone markers for log compacted topics. This setting also gives a bound on the time in which a consumer must complete a read if they begin from offset 0 to ensure that they get a valid snapshot of the final stage (otherwise delete tombstones may be collected before they complete their scan). |
The time to wait before deleting a file from the filesystem. |
This setting allows specifying an interval at which we will force an fsync of data written to the log. For example if this was set to 1 we would fsync after every message; if it were 5 we would fsync after every five messages. In general we recommend you not set this and use replication for durability and allow the operating system's background flush capabilities as it is more efficient. |
This setting allows specifying a time interval at which we will force an fsync of data written to the log. For example if this was set to 1000 we would fsync after 1000 ms had passed. In general we recommend you not set this and use replication for durability and allow the operating system's background flush capabilities as it is more efficient. |
This setting controls how frequently Kafka adds an index entry to its offset index. The default setting ensures that we index a message roughly every 4096 bytes. More indexing allows reads to jump closer to the exact position in the log but makes the index larger. You probably don't need to change this. |
This configuration controls the maximum bytes tiered storage will retain segment files locally before it will discard old log segments to free up space. If set to -2, the limit is equal to overall retention time. If set to -1, no limit is applied but it's possible only if overall retention is also -1. |
This configuration controls the maximum time tiered storage will retain segment files locally before it will discard old log segments to free up space. If set to -2, the time limit is equal to overall retention time. If set to -1, no time limit is applied but it's possible only if overall retention is also -1. |
The maximum time a message will remain ineligible for compaction in the log. Only applicable for logs that are being compacted. |
The largest record batch size allowed by Kafka (after compression if compression is enabled). If this is increased and there are consumers older than 0.10.2, the consumers' fetch size must also be increased so that the they can fetch record batches this large. In the latest message format version, records are always grouped into batches for efficiency. In previous message format versions, uncompressed records are not grouped into batches and this limit only applies to a single record in that case. |
The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message. If message.timestamp.type=CreateTime, a message will be rejected if the difference in timestamp exceeds this threshold. Applies only for messages with timestamps later than the broker's timestamp. |
The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message. If message.timestamp.type=CreateTime, a message will be rejected if the difference in timestamp exceeds this threshold. Applies only for messages with timestamps earlier than the broker's timestamp. |
Define whether the timestamp in the message is message create time or log append time. |
The minimum time a message will remain uncompacted in the log. Only applicable for logs that are being compacted. |
True if we should preallocate the file on disk when creating a new log segment. |
This configuration controls the maximum size a partition (which consists of log segments) can grow to before we will discard old log segments to free up space if we are using the 'delete' retention policy. By default there is no size limit only a time limit. Since this limit is enforced at the partition level, multiply it by the number of partitions to compute the topic retention in bytes. |
This configuration controls the maximum time we will retain a log before we will discard old log segments to free up space if we are using the 'delete' retention policy. This represents an SLA on how soon consumers must read their data. If set to -1, no time limit is applied. |
This configuration controls the segment file size for the log. Retention and cleaning is always done a file at a time so a larger segment size means fewer files but less granular control over retention. Setting this to a very low value has consequences, and the Aiven management plane ignores values less than 10 megabytes. |
This configuration controls the size of the index that maps offsets to file positions. We preallocate this index file and shrink it only after log rolls. You generally should not need to change this setting. |
The maximum random jitter subtracted from the scheduled segment roll time to avoid thundering herds of segment rolling |
This configuration controls the period of time after which Kafka will force the log to roll even if the segment file isn't full to ensure that retention can delete or compact old data. Setting this to a very low value has consequences, and the Aiven management plane ignores values less than 10 seconds. |