Migrator from elasticsearch to opensearch

• Introduction
• Compatibility matrix
• Pre requisites
• Previous configuration
• Parameters and configuration
• Manual execution
• Recommendations

Introduction

Ver. 131123

CLI utility to migrate an elastic search environment to opensearch

Download link:  link

Type: utility

Compatibility matrix

Versión de opensearch probada	2.11.0
Tested elastic version	6.4.3  8.10.3
OS systems where tested	CentOS 7 Rocky 8 Ubuntu 22.04
Systems where it should work	Any Linux operating system with a Pandora FMS server installed

Pre requisites

In order to run this utility you need to :

• Connectivity to the elastic search environment.
  
• Connectivity to the opensearch environment.
  
• To have opensearch authentication credentials (in case you have basic authentication configured).

Previous configuration

Check that you have access to Elasticsearch.
From the machine where the migrator will run, run a test with curl to verify connectivity.

curl http://<hostname_or_ip>:9200

Perform the same check, but this time to the opensearch environment, it can be done with the same command and changing the hostname or ip in the case of not having basic authentication.

In case of having authentication you must execute:

curl -X GET -ku 'user:pass' https://<hostname_or_ip>:9200

If the check is correct and we have access to both environments, we can now use the viewpoint.

Parameters and configuration

Parameters

-s SOURCE_URL, --source_url SOURCE_URL	URL of elastic source environment to be migrated
-d DESTINATION_URL, --destination_url DESTINATION_URL	URL of the opensearch target environment
-P INDEX_PATTERN, --index_pattern INDEX_PATTERN	Index pattern to be migrated, by default: pandorafms*.
-a, --use_auth	Flag, use basic authentication
-u USER, --user USER	User for basic authentication, by default: admin
-p PASSWORD, --password PASSWORD	Basic authentication password, default: admin
-t TIMEOUT, --timeout TIMEOUT	Time to wait between index migrations
-f, --force	Deletes the index at the destination if it exists and forces migration
-C, --compare	Compares the source and destination indexes and the status of each.
-v, --verbose	Flag, shows step by step all migration actions, for debugging.

Manual execution

Migration execution:

./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 -a \
-u admin -p admin

Migration execution filtering by a single index.

./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 -a \
-u admin -p admin -f \
-P pandorafms-b8ef1b65799a11ee94810242ac100002-2023.11.02

Execution check:

./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 -a \
-u admin -p admin -C

The check can also be performed by filtering by indexes using a search pattern:

./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 \
-a -u admin -p admin -C \
-P "pandorafms-b8ef1b65799a11ee94810242ac100002*"

Recommendations

To perform a migration it is important to take into account the size of the environments to be migrated and the amount of indexes and documents we have stored. Migration is a demanding process that transfers a lot of data at high speed.

In general we indicate certain recommendations for environments with a large amount of data:

• Perform the migration before connecting the pandorafms environment to the new opensearch.
  
• Perform a staggered migration, year by year, month by month or even index by index, using the pattern filter. Depending on the amount of data stored.
  
• Right-size the environment.
  
• For a large amount of data, use an opensearch cluster instead of a single node.