Migrator from elasticsearch to opensearch

CLI utility to migrate an elasticsearch environment (PandoraFMS v773 and earlier) to opensearch (PandoraFMS v774 onwards)

Introduction

Ver. 131123

CLI utility to migrate an elastic search environment to opensearch

Download link:  link

Type: utility

Compatibility matrix

Versión de opensearch probada

2.11.0

Tested elastic version

6.4.3 

8.10.3

OS systems where tested

CentOS 7

Rocky 8

Ubuntu 22.04

Systems where it should work

Any Linux operating system with a Pandora FMS server installed


Pre requisites

In order to run this utility you need to :

Previous configuration

Check that you have access to Elasticsearch.
From the machine where the migrator will run, run a test with curl to verify connectivity.

curl http://<hostname_or_ip>:9200

image.png

Perform the same check, but this time to the opensearch environment, it can be done with the same command and changing the hostname or ip in the case of not having basic authentication.

In case of having authentication you must execute:

curl -X GET -ku 'user:pass' https://<hostname_or_ip>:9200

image.png

If the check is correct and we have access to both environments, we can now use the viewpoint.

Parameters and configuration

Parameters

-s SOURCE_URL, --source_url SOURCE_URL URL of elastic source environment to be migrated
-d DESTINATION_URL, --destination_url DESTINATION_URL URL of the opensearch target environment
-P INDEX_PATTERN, --index_pattern INDEX_PATTERN Index pattern to be migrated, by default: pandorafms*.
 -a, --use_auth Flag, use basic authentication
-u USER, --user USER User for basic authentication, by default: admin
 -p PASSWORD, --password PASSWORD Basic authentication password, default: admin
-t TIMEOUT, --timeout TIMEOUT Time to wait between index migrations
-f, --force Deletes the index at the destination if it exists and forces migration
 -C, --compare Compares the source and destination indexes and the status of each.
-v, --verbose Flag, shows step by step all migration actions, for debugging.

image.png

Manual execution

Migration execution:

./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 -a \
-u admin -p admin

Migration execution filtering by a single index.

./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 -a \
-u admin -p admin -f \
-P pandorafms-b8ef1b65799a11ee94810242ac100002-2023.11.02

image.png

Execution check:

./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 -a \
-u admin -p admin -C

image.png

The check can also be performed by filtering by indexes using a search pattern:

./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 \
-a -u admin -p admin -C \
-P "pandorafms-b8ef1b65799a11ee94810242ac100002*"

image.png

Recommendations

To perform a migration it is important to take into account the size of the environments to be migrated and the amount of indexes and documents we have stored. Migration is a demanding process that transfers a lot of data at high speed.

In general we indicate certain recommendations for environments with a large amount of data: