Migrate Elasticsearch data to Aiven for OpenSearch®

To migrate Elasticsearch data to Aiven for OpenSearch®, reindex from a remote Elasticsearch cluster. This method can also be used to migrate data from Aiven for OpenSearch to a self-hosted Elasticsearch service.

tip

To migrate a large number of indexes, consider automating the process with a script.

As Aiven for OpenSearch does not support joining external Elasticsearch servers to the same cluster, online migration is not currently possible.

important

Migrating from Elasticsearch to OpenSearch can impact connectivity between client applications and services. Some clients or tools may check the service version, which can lead to compatibility issues with OpenSearch. For more details, refer to the following OpenSearch resources:

• OpenSearch release notes
• OpenSearch Dashboards release notes
• Frequently asked questions about OpenSearch

Migrate data

1. Create an Aiven for OpenSearch service.

2. Set the reindex.remote.whitelist parameter to point to your source Elasticsearch service using the following Aiven CLI command:

   avn service update your-aiven-service \
     -c 'opensearch.reindex_remote_whitelist=["your-non-aiven-service:port"]'

   Replace port with the port number your source Elasticsearch service is using.

3. Wait for the cluster to restart. This process might take a few minutes as the service attempts a rolling restart to minimize downtime.

4. Start migrating the indexes. For each index:

   1. Stop writes to the index. This step is optional if testing the process.

   2. Export the index mapping from the source Elasticsearch instance. For example, using curl:

      curl https://avnadmin:yourpassword@os-123-demoprj.aivencloud.com:23125/logs-2024-09-21/_mapping > mapping.json

   3. Edit mapping.json:

      With jq

      If you have jq, run:

      jq .[].mappings mapping.json > src_mapping.json

      Manual update

      To edit mapping.json manually:

      • Remove the wrapping {"logs-2024-09-21":{"mappings": ... }}.
      • Keep {"properties":...}}.

   4. Create the empty index on your destination Aiven for OpenSearch service.

      curl -XPUT https://avnadmin:yourpassword@os-123-demoprj.aivencloud.com:23125/logs-2024-09-21

   5. Import the mapping to the destination Aiven for OpenSearch index.

      curl -XPUT https://avnadmin:yourpassword@os-123-demoprj.aivencloud.com:23125/logs-2024-09-21/_mapping \
      -H 'Content-type: application/json' -T src_mapping.json

   6. Submit the reindexing request.

      curl -XPOST https://avnadmin:yourpassword@os-123-demoprj.aivencloud.com:23125/_reindex \
        -H 'Content-type: application/json' \
        -d '{"source":
                {"index": "logs-2024-09-21",
                 "remote":
                     {"username": "your-remote-username",
                      "password": "your-remote-password",
                      "host": "https://your.non-aiven-service.example.com:9200"
                     }
                },
             "dest":
                {"index": "logs-2024-09-21"}
            }'

   7. Wait for the reindexing process to complete. If you receive a response message such as:

      [your.non-aiven-service.example.com:9200] not whitelisted in reindex.remote.whitelist

      Verify the hostname and port match those set earlier. The time required for reindexing can vary depending on the amount of data.

   8. Update clients to use the new index on Aiven for OpenSearch for both read and write operations, then resume any paused write activity.

   9. Delete the source index if necessary.