Skip to main content

Advanced parameters for Aiven for OpenSearch®

See the configuration options available for Aiven for OpenSearch®:

Parameter

Additional Cloud Regions for Backup Replication

OpenSearch version

Disable automatic replication factor adjustment for multi-node services. By default, Aiven ensures all indexes are replicated at least to two nodes. Note: Due to potential data loss in case of losing a service node, this setting can not be activated unless specifically allowed for the project.

custom_domain

string,null

Serve the web frontend using a custom CNAME pointing to the Aiven DNS name. When you set a custom domain for a service deployed in a VPC, the service certificate is only created for the public-* hostname and the custom domain.

Allow to register object storage repositories in OpenSearch

Allow to register custom keystores in OpenSearch

  • default: 0.0.0.0/0,::/0

IP filter

Allow incoming connections from CIDR address block, e.g. '10.20.0.0/16'

service_log

boolean,null

Service logging

Store logs for the service so that they are available in the HTTP API and console.

static_ips

boolean

Use static public IP addresses

saml

object

OpenSearch SAML configuration

  • default: true

Enable or disable OpenSearch SAML authentication

Enables or disables SAML-based authentication for OpenSearch. When enabled, users can authenticate using SAML with an Identity Provider.

The URL of the SAML metadata for the Identity Provider (IdP). This is used to configure SAML-based authentication with the IdP.

The unique identifier for the Identity Provider (IdP) entity that is used for SAML authentication. This value is typically provided by the IdP.

The unique identifier for the Service Provider (SP) entity that is used for SAML authentication. This value is typically provided by the SP.

saml.subject_key

string,null

Optional. Specifies the attribute in the SAML response where the subject identifier is stored. If not configured, the NameID attribute is used by default.

saml.roles_key

string,null

Optional. Specifies the attribute in the SAML response where role information is stored, if available. Role attributes are not required for SAML authentication, but can be included in SAML assertions by most Identity Providers (IdPs) to determine user access levels or permissions.

This parameter specifies the PEM-encoded root certificate authority (CA) content for the SAML identity provider (IdP) server verification. The root CA content is used to verify the SSL/TLS certificate presented by the server.

openid

object

OpenSearch OpenID Connect Configuration

  • default: true

Enable or disable OpenSearch OpenID Connect authentication

Enables or disables OpenID Connect authentication for OpenSearch. When enabled, users can authenticate using OpenID Connect with an Identity Provider.

The URL of your IdP where the Security plugin can find the OpenID Connect metadata/configuration settings.

openid.roles_key

string,null

The key in the JSON payload that stores the user’s roles. The value of this key must be a comma-separated list of roles. Required only if you want to use roles in the JWT

The key in the JSON payload that stores the user’s name. If not defined, the subject registered claim is used. Most IdP providers use the preferred_username claim. Optional.

The HTTP header that stores the token. Typically the Authorization header with the Bearer schema: Authorization: Bearer <token>. Optional. Default is Authorization.

URL JWT token.

If the token is not transmitted in the HTTP header, but as an URL parameter, define the name of the parameter here. Optional.

  • min: 10
  • max: 9223372036854776000
  • default: 10

The maximum number of unknown key IDs in the time frame. Default is 10. Optional.

  • min: 10000
  • max: 9223372036854776000
  • default: 10000

The time frame to use when checking the maximum number of unknown key IDs, in milliseconds. Optional.Default is 10000 (10 seconds).

The ID of the OpenID Connect client configured in your IdP. Required.

The client secret of the OpenID Connect client configured in your IdP. Required.

The scope of the identity token issued by the IdP. Optional. Default is openid profile email address phone.

  • default: Authorization

HTTP header name of the JWT token. Optional. Default is Authorization.

jwt

object

OpenSearch JWT Configuration

Enable or disable OpenSearch JWT authentication

Enables or disables JWT-based authentication for OpenSearch. When enabled, users can authenticate using JWT tokens.

JWT signing key

The secret key used to sign and verify JWT tokens. This should be a secure, randomly generated key HMAC key or public RSA/ECDSA key.

jwt.jwt_header

string,null

  • default: Authorization

The HTTP header name where the JWT token is transmitted. Typically 'Authorization' for Bearer tokens.

If the JWT token is transmitted as a URL parameter instead of an HTTP header, specify the parameter name here.

jwt.subject_key

string,null

The key in the JWT payload that contains the user's subject identifier. If not specified, the 'sub' claim is used by default.

jwt.roles_key

string,null

JWT claim key for roles

The key in the JWT payload that contains the user's roles. If specified, roles will be extracted from the JWT for authorization.

Required JWT audience

If specified, the JWT must contain an 'aud' claim that matches this value. This provides additional security by ensuring the JWT was issued for the expected audience.

Required JWT issuer

If specified, the JWT must contain an 'iss' claim that matches this value. This provides additional security by ensuring the JWT was issued by the expected issuer.

  • max: 300
  • default: 20

JWT clock skew tolerance in seconds

The maximum allowed time difference in seconds between the JWT issuer's clock and the OpenSearch server's clock. This helps prevent token validation failures due to minor time synchronization issues.

Azure migration settings

The snapshot name to restore from

If true, restore the cluster state. Defaults to false

Include aliases

Whether to restore aliases alongside their associated indexes. Default is true.

A comma-delimited list of indices to restore from the snapshot. Multi-index syntax is supported.

The path to the repository data within its container. The value of this setting should not start or end with a /

when set to true metadata files are stored in compressed format

Chunk size

Big files can be broken down into chunks during snapshotting if needed. Should be the same as for the 3rd party repository

Account name

Azure account secret key. One of key or sas_token should be specified

SAS token

A shared access signatures (SAS) token. One of key or sas_token should be specified

Azure container name

Endpoint suffix

Defines the DNS suffix for Azure Storage endpoints.

  • default: true

Whether the repository is read-only.

Google Cloud Storage migration settings

The snapshot name to restore from

If true, restore the cluster state. Defaults to false

Include aliases

Whether to restore aliases alongside their associated indexes. Default is true.

A comma-delimited list of indices to restore from the snapshot. Multi-index syntax is supported.

The path to the repository data within its container. The value of this setting should not start or end with a /

when set to true metadata files are stored in compressed format

Chunk size

Big files can be broken down into chunks during snapshotting if needed. Should be the same as for the 3rd party repository

Google Cloud Storage bucket name

The path to the repository data within its container

Google Cloud Storage credentials file content

  • default: true

Whether the repository is read-only.

AWS S3 / AWS S3 compatible migration settings

The snapshot name to restore from

If true, restore the cluster state. Defaults to false

Include aliases

Whether to restore aliases alongside their associated indexes. Default is true.

A comma-delimited list of indices to restore from the snapshot. Multi-index syntax is supported.

The path to the repository data within its container. The value of this setting should not start or end with a /

when set to true metadata files are stored in compressed format

Chunk size

Big files can be broken down into chunks during snapshotting if needed. Should be the same as for the 3rd party repository

AWS Access key

AWS secret key

S3 bucket name

S3 region

The S3 service endpoint to connect to. If you are using an S3-compatible service then you should set this to the service’s endpoint

When set to true files are encrypted on server side

  • default: true

Whether the repository is read-only.

Index patterns

  • max: 9223372036854776000

Maximum index count

DEPRECATED: use index_patterns instead

Don't reset index.refresh_interval to the default value

Aiven automation resets index.refresh_interval to default value for every index to be sure that indices are always visible to search. If it doesn't fit your case, you can disable this by setting up this flag to true.

OpenSearch Dashboards settings

  • default: true

Enable or disable OpenSearch Dashboards

  • min: 64
  • max: 4096
  • default: 128

Limits the maximum amount of memory (in MiB) the OpenSearch Dashboards process can use. This sets the max_old_space_size option of the nodejs running the OpenSearch Dashboards. Note: the memory reserved by OpenSearch Dashboards is not available for OpenSearch.

  • min: 5000
  • max: 120000
  • default: 30000

Timeout in milliseconds for requests made by OpenSearch Dashboards towards OpenSearch

  • default: true

Enable or disable multiple data sources in OpenSearch Dashboards

Index rollup settings

  • min: 1

plugins.rollup.search.backoff_millis

The backoff time between retries for failed rollup jobs. Defaults to 1000ms.

  • min: 1

plugins.rollup.search.backoff_count

How many retries the plugin should attempt for failed rollup jobs. Defaults to 5.

plugins.rollup.search.all_jobs

Whether OpenSearch should return all jobs that match all specified search terms. If disabled, OpenSearch returns just one, as opposed to all, of the jobs that matches the search terms. Defaults to false.

plugins.rollup.dashboards.enabled

Whether rollups are enabled in OpenSearch Dashboards. Defaults to true.

plugins.rollup.enabled

Whether the rollup plugin is enabled. Defaults to true.

OpenSearch settings

reindex_remote_whitelist

Whitelisted addresses for reindexing. Changing this value will cause all OpenSearch instances to restart.

  • min: 1
  • max: 2147483647

http.max_content_length

Maximum content length for HTTP requests to the OpenSearch HTTP API, in bytes.

  • min: 1024
  • max: 262144

http.max_header_size

The max size of allowed headers, in bytes

  • min: 1024
  • max: 65536

http.max_initial_line_length

The max length of an HTTP URL, in bytes

  • min: 64
  • max: 4096

indices.query.bool.max_clause_count

Maximum number of clauses Lucene BooleanQuery can have. The default value (1024) is relatively high, and increasing it may cause performance issues. Investigate other approaches first before increasing this value.

  • min: 1
  • max: 1000000

search.max_buckets

Maximum number of aggregation buckets allowed in a single response. OpenSearch default value is used when this is not defined.

  • min: 3
  • max: 100

indices.fielddata.cache.size

Relative amount. Maximum amount of heap memory used for field data cache. This is an expert setting; decreasing the value too much will increase overhead of loading field data; too much memory used for field data cache will decrease amount of heap available for other operations.

  • min: 3
  • max: 40

indices.memory.index_buffer_size

Percentage value. Default is 10%. Total amount of heap used for indexing buffer, before writing segments to disk. This is an expert setting. Too low value will slow down indexing; too high value will increase indexing performance but causes performance issues for query performance.

  • min: 3
  • max: 2048

indices.memory.min_index_buffer_size

Absolute value. Default is 48mb. Doesn't work without indices.memory.index_buffer_size. Minimum amount of heap used for query cache, an absolute indices.memory.index_buffer_size minimal hard limit.

  • min: 3
  • max: 2048

indices.memory.max_index_buffer_size

Absolute value. Default is unbound. Doesn't work without indices.memory.index_buffer_size. Maximum amount of heap used for query cache, an absolute indices.memory.index_buffer_size maximum hard limit.

  • min: 3
  • max: 40

indices.queries.cache.size

Percentage value. Default is 10%. Maximum amount of heap used for query cache. This is an expert setting. Too low value will decrease query performance and increase performance for other operations; too high value will cause issues with other OpenSearch functionality.

  • min: 40
  • max: 400

indices.recovery.max_bytes_per_sec

Limits total inbound and outbound recovery traffic for each node. Applies to both peer recoveries as well as snapshot recoveries (i.e., restores from a snapshot). Defaults to 40mb

  • min: 2
  • max: 5

indices.recovery.max_concurrent_file_chunks

Number of file chunks sent in parallel for each recovery. Defaults to 2.

action.auto_create_index

Explicitly allow or block automatic creation of indices. Defaults to true

plugins.alerting.filter_by_backend_roles

Enable or disable filtering of alerting by backend roles. Requires Security plugin. Defaults to false

  • max: 100

knn.memory.circuit_breaker.limit

Maximum amount of memory in percentage that can be used for the KNN index. Defaults to 50% of the JVM heap size. 0 is used to set it to null which can be used to invalidate caches.

  • Service restart

knn.memory.circuit_breaker.enabled

Enable or disable KNN memory circuit breaker. Defaults to true.

Opensearch Security Plugin Settings

internal_authentication_backend_limiting.type

The type of rate limiting

internal_authentication_backend_limiting.authentication_backend

The internal backend. Enter internal

  • min: 1
  • max: 32767

internal_authentication_backend_limiting.allowed_tries

The number of login attempts allowed before login is blocked

  • max: 2147483647

internal_authentication_backend_limiting.time_window_seconds

The window of time in which the value for allowed_tries is enforced

  • max: 2147483647

internal_authentication_backend_limiting.block_expiry_seconds

The duration of time that login remains blocked after a failed login

  • max: 2147483647

internal_authentication_backend_limiting.max_blocked_clients

The maximum number of blocked IP addresses

  • max: 2147483647

internal_authentication_backend_limiting.max_tracked_clients

The maximum number of tracked IP addresses that have failed login

Enable/Disable security audit

Enable/Disable snapshot API for custom repositories, this requires security management to be enabled

  • min: 1
  • max: 128

search thread pool size

Size for the thread pool. See documentation for exact details. Do note this may have maximum value depending on CPU count - value is automatically lowered if set to higher than maximum value.

  • min: 1
  • max: 128

search_throttled thread pool size

Size for the thread pool. See documentation for exact details. Do note this may have maximum value depending on CPU count - value is automatically lowered if set to higher than maximum value.

  • min: 1
  • max: 128

Size for the thread pool. See documentation for exact details. Do note this may have maximum value depending on CPU count - value is automatically lowered if set to higher than maximum value.

  • min: 1
  • max: 128

analyze thread pool size

Size for the thread pool. See documentation for exact details. Do note this may have maximum value depending on CPU count - value is automatically lowered if set to higher than maximum value.

  • min: 1
  • max: 128

write thread pool size

Size for the thread pool. See documentation for exact details. Do note this may have maximum value depending on CPU count - value is automatically lowered if set to higher than maximum value.

  • min: 1
  • max: 128

force_merge thread pool size

Size for the thread pool. See documentation for exact details. Do note this may have maximum value depending on CPU count - value is automatically lowered if set to higher than maximum value.

  • min: 10
  • max: 2000

search thread pool queue size

Size for the thread pool queue. See documentation for exact details.

  • min: 10
  • max: 2000

search_throttled thread pool queue size

Size for the thread pool queue. See documentation for exact details.

  • min: 10
  • max: 2000

Size for the thread pool queue. See documentation for exact details.

  • min: 10
  • max: 2000

analyze thread pool queue size

Size for the thread pool queue. See documentation for exact details.

  • min: 10
  • max: 2000

write thread pool queue size

Size for the thread pool queue. See documentation for exact details.

Require explicit index names when deleting

  • min: 100
  • max: 10000

cluster.max_shards_per_node

Controls the number of shards allowed in the cluster per data node

compatibility.override_main_response_version

Compatibility mode sets OpenSearch to report its version as 7.10 so clients continue to work. Default is false

Script max compilation rate - circuit breaker to prevent/minimize OOMs

Script compilation circuit breaker limits the number of inline script compilations within a period of time. Default is use-context

  • min: 2
  • max: 16

How many concurrent incoming/outgoing shard recoveries (normally replicas) are allowed to happen on a node. Defaults to node cpu count * 2.

This should be identical to the Sender name defined in Opensearch dashboards

Sender username for Opensearch alerts

Sender password for Opensearch alerts to authenticate with SMTP server

Specifies whether ISM is enabled or not

Specifies whether audit history is enabled or not. The logs from ISM are automatically indexed to a logs document.

  • min: 1
  • max: 2147483647

The maximum age before rolling over the audit history index in hours

  • min: 1
  • max: 9223372036854776000

The maximum number of documents before rolling over the audit history index.

  • min: 1
  • max: 2147483647

The time between rollover checks for the audit history index in hours.

  • min: 1
  • max: 2147483647

How long audit history indices are kept in days.

Search Backpressure Settings

The search backpressure mode. Valid values are monitor_only, enforced, or disabled. Default is monitor_only

Node duress settings

  • max: 1

The CPU usage threshold (as a percentage) required for a node to be considered to be under duress. Default is 0.9

  • max: 1

The heap usage threshold (as a percentage) required for a node to be considered to be under duress. Default is 0.7

  • min: 1

The number of successive limit breaches after which the node is considered to be under duress. Default is 3

Search task settings

  • min: 1

The maximum number of search tasks to cancel in a single iteration of the observer thread. Default is 5.0

The maximum number of search tasks to cancel per millisecond of elapsed time. Default is 0.003

  • max: 1

The maximum number of search tasks to cancel, as a percentage of successful search task completions. Default is 0.1

The CPU usage threshold (in milliseconds) required for an individual parent task before it is considered for cancellation. Default is 30000

The elapsed time threshold (in milliseconds) required for an individual parent task before it is considered for cancellation. Default is 45000

The window size used to calculate the rolling average of the heap usage for the completed parent tasks. Default is 10

  • max: 1

The heap usage threshold (as a percentage) required for an individual parent task before it is considered for cancellation. Default is 0.2

The heap usage variance required for an individual parent task before it is considered for cancellation. A task is considered for cancellation when taskHeapUsage is greater than or equal to heapUsageMovingAverage * variance. Default is 2.0

  • max: 1

The heap usage threshold (as a percentage) required for the sum of heap usages of all search tasks before cancellation is applied. Default is 0.5

Search shard settings

  • min: 1

The maximum number of search tasks to cancel in a single iteration of the observer thread. Default is 10.0

The maximum number of tasks to cancel per millisecond of elapsed time. Default is 0.003

  • max: 1

The maximum number of tasks to cancel, as a percentage of successful task completions. Default is 0.1

The CPU usage threshold (in milliseconds) required for a single search shard task before it is considered for cancellation. Default is 15000

The elapsed time threshold (in milliseconds) required for a single search shard task before it is considered for cancellation. Default is 30000

The number of previously completed search shard tasks to consider when calculating the rolling average of heap usage. Default is 100

  • max: 1

The heap usage threshold (as a percentage) required for a single search shard task before it is considered for cancellation. Default is 0.5

The minimum variance required for a single search shard task’s heap usage compared to the rolling average of previously completed tasks before it is considered for cancellation. Default is 2.0

  • max: 1

The heap usage threshold (as a percentage) required for the sum of heap usages of all search shard tasks before cancellation is applied. Default is 0.5

Shard indexing back pressure settings

Enable or disable shard indexing backpressure. Default is false

Run shard indexing backpressure in shadow mode or enforced mode. In shadow mode (value set as false), shard indexing backpressure tracks all granular-level metrics, but it doesn’t actually reject any indexing requests. In enforced mode (value set as true), shard indexing backpressure rejects any requests to the cluster that might cause a dip in its performance. Default is false

Primary parameter

Node soft limit

Define the percentage of the node-level memory threshold that acts as a soft indicator for strain on a node. Default is 0.7

Shard min limit

Specify the minimum assigned quota for a new shard in any role (coordinator, primary, or replica). Shard indexing backpressure increases or decreases this allocated quota based on the inflow of traffic for the shard. Default is 0.001

Operating factor

Specify the lower occupancy limit of the allocated quota of memory for the shard. If the total memory usage of a shard is below this limit, shard indexing backpressure decreases the current allocated memory for that shard. Default is 0.75

Specify the optimal occupancy of the allocated quota of memory for the shard. If the total memory usage of a shard is at this level, shard indexing backpressure doesn’t change the current allocated memory for that shard. Default is 0.85

Specify the upper occupancy limit of the allocated quota of memory for the shard. If the total memory usage of a shard is above this limit, shard indexing backpressure increases the current allocated memory for that shard. Default is 0.95

Top N queries monitoring by CPU

Enable or disable top N query monitoring by the metric

  • min: 1

Specify the value of N for the top N queries by the metric

Configure the window size of the top N queries

Top N queries monitoring by latency

Enable or disable top N query monitoring by the metric

  • min: 1

Specify the value of N for the top N queries by the metric

Configure the window size of the top N queries

Top N queries monitoring by memory

Enable or disable top N query monitoring by the metric

  • min: 1

Specify the value of N for the top N queries by the metric

Configure the window size of the top N queries

cluster.routing.allocation.balance.prefer_primary

When set to true, OpenSearch attempts to evenly distribute the primary shards between the cluster nodes. Enabling this setting does not always guarantee an equal number of primary shards on each node, especially in the event of a failover. Changing this setting to false after it was set to true does not invoke redistribution of primary shards. Default is false.

Watermark settings

Low watermark (percentage)

The low watermark for disk usage.

The high watermark for disk usage.

The flood stage watermark for disk usage.

Segment Replication Backpressure Settings

segrep.pressure.enabled

Enables the segment replication backpressure mechanism. Default is false.

  • default: 5m

The maximum amount of time that a replica shard can take to copy from the primary shard. Once segrep.pressure.time.limit is breached along with segrep.pressure.checkpoint.limit, the segment replication backpressure mechanism is initiated. Default is 5 minutes.

  • default: 4

The maximum number of indexing checkpoints that a replica shard can fall behind when copying from primary. Once segrep.pressure.checkpoint.limit is breached along with segrep.pressure.time.limit, the segment replication backpressure mechanism is initiated. Default is 4 checkpoints.

  • max: 1
  • default: 0.5

The maximum number of stale replica shards that can exist in a replication group. Once segrep.pressure.replica.stale.limit is breached, the segment replication backpressure mechanism is initiated. Default is .5, which is 50% of a replication group.

The default value of the translog buffer interval used when performing periodic translog updates. This setting is only effective when the index setting index.remote_store.translog.buffer_interval is not present. Defaults to 650ms.

  • min: 100
  • max: 2147483647

Sets the maximum number of open translog files for remote-backed indexes. This limits the total number of translog files per shard. After reaching this limit, the remote store flushes the translog files. Default is 1000. The minimum required is 100.

The amount of time to wait for the cluster state upload to complete. Defaults to 20s.

The amount of time to wait for the manifest file upload to complete. The manifest file contains the details of each of the files uploaded for a single cluster state, both index metadata files and global metadata files. Defaults to 20s.

Enables remote segment backpressure. Default is true

  • min: 1
  • max: 2147483647

The minimum consecutive failure count for activating remote segment backpressure. Defaults to 5.

  • min: 1

The variance factor that is used together with the moving average to calculate the dynamic bytes lag threshold for activating remote segment backpressure. Defaults to 10.

  • min: 1

The variance factor that is used together with the moving average to calculate the dynamic time lag threshold for activating remote segment backpressure. Defaults to 10.

  • max: 100

Defines a limit of how much total remote data can be referenced as a ratio of the size of the disk reserved for the file cache. This is designed to be a safeguard to prevent oversubscribing a cluster. Defaults to 0.

Defines a limit of how much total remote data can be referenced as a ratio of the size of the disk reserved for the file cache. This is designed to be a safeguard to prevent oversubscribing a cluster. Defaults to 5gb. Requires restarting all OpenSearch nodes.

  • default: trace

Log level

Debug threshold for total request took time. The value should be in the form count and unit, where unit one of (s,m,h,d,nanos,ms,micros) or -1. Default is -1

Info threshold for total request took time. The value should be in the form count and unit, where unit one of (s,m,h,d,nanos,ms,micros) or -1. Default is -1

Trace threshold for total request took time. The value should be in the form count and unit, where unit one of (s,m,h,d,nanos,ms,micros) or -1. Default is -1

Warning threshold for total request took time. The value should be in the form count and unit, where unit one of (s,m,h,d,nanos,ms,micros) or -1. Default is -1

Enable remote-backed storage

Enable searchable snapshots

Template settings for all new indexes

  • max: 100000

(DEPRECATED) index.mapping.nested_objects.limit

The maximum number of nested JSON objects that a single document can contain across all nested types. This limit helps to prevent out of memory errors when a document contains too many nested objects. Default is 10000. Deprecated, use an index template instead.

  • min: 1
  • max: 1024

(DEPRECATED) index.number_of_shards

The number of primary shards that an index should have. Deprecated, use an index template instead.

  • max: 29

(DEPRECATED) index.number_of_replicas

The number of replicas each primary shard has. Deprecated, use an index template instead.

Allow access to selected service ports from private networks

Allow clients to connect to opensearch with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations

Allow clients to connect to opensearch_dashboards with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations

Allow clients to connect to prometheus with a DNS name that always resolves to the service's private IP addresses. Only available in certain network locations

Allow access to selected service components through Privatelink

Enable opensearch

Enable opensearch_dashboards

Enable prometheus

Allow access to selected service ports from the public Internet

Allow clients to connect to opensearch from the public internet for service nodes that are in a project VPC or another type of private network

Allow clients to connect to opensearch_dashboards from the public internet for service nodes that are in a project VPC or another type of private network

Allow clients to connect to prometheus from the public internet for service nodes that are in a project VPC or another type of private network

Name of the basebackup to restore in forked service

Name of another service to fork from. This has effect only when a new service is being created.

Name of another project to fork a service from. This has effect only when a new service is being created.

Elasticsearch version