Migrator from elasticsearch to opensearch
CLI utility to migrate an elasticsearch environment (PandoraFMS v773 and earlier) to opensearch (PandoraFMS v774 onwards)
- Introduction
- Compatibility matrix
- Pre requisites
- Previous configuration
- Parameters and configuration
- Manual execution
- Recommendations
Introduction
Ver. 131123
CLI utility to migrate an elastic search environment to opensearch
Download link: link
Type: utility
Compatibility matrix
Versión de opensearch probada |
2.11.0 |
Tested elastic version |
6.4.3 8.10.3 |
OS systems where tested |
CentOS 7 Rocky 8 Ubuntu 22.04 |
Systems where it should work |
Any Linux operating system with a Pandora FMS server installed |
Pre requisites
In order to run this utility you need to :
- Connectivity to the elastic search environment.
- Connectivity to the opensearch environment.
- To have opensearch authentication credentials (in case you have basic authentication configured).
Previous configuration
Check that you have access to Elasticsearch.
From the machine where the migrator will run, run a test with curl to verify connectivity.
curl http://<hostname_or_ip>:9200
Perform the same check, but this time to the opensearch environment, it can be done with the same command and changing the hostname or ip in the case of not having basic authentication.
In case of having authentication you must execute:
curl -X GET -ku 'user:pass' https://<hostname_or_ip>:9200
If the check is correct and we have access to both environments, we can now use the viewpoint.
Parameters and configuration
Parameters
-s SOURCE_URL, --source_url SOURCE_URL | URL of elastic source environment to be migrated |
-d DESTINATION_URL, --destination_url DESTINATION_URL | URL of the opensearch target environment |
-P INDEX_PATTERN, --index_pattern INDEX_PATTERN | Index pattern to be migrated, by default: pandorafms*. |
-a, --use_auth | Flag, use basic authentication |
-u USER, --user USER | User for basic authentication, by default: admin |
-p PASSWORD, --password PASSWORD | Basic authentication password, default: admin |
-t TIMEOUT, --timeout TIMEOUT | Time to wait between index migrations |
-f, --force | Deletes the index at the destination if it exists and forces migration |
-C, --compare | Compares the source and destination indexes and the status of each. |
-v, --verbose | Flag, shows step by step all migration actions, for debugging. |
Manual execution
Migration execution:
./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 -a \
-u admin -p admin
Migration execution filtering by a single index.
./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 -a \
-u admin -p admin -f \
-P pandorafms-b8ef1b65799a11ee94810242ac100002-2023.11.02
Execution check:
./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 -a \
-u admin -p admin -C
The check can also be performed by filtering by indexes using a search pattern:
./elastic_to_opensearch -s 192.168.80.44:9200 -d 192.168.50.92:9200 \
-a -u admin -p admin -C \
-P "pandorafms-b8ef1b65799a11ee94810242ac100002*"
Recommendations
To perform a migration it is important to take into account the size of the environments to be migrated and the amount of indexes and documents we have stored. Migration is a demanding process that transfers a lot of data at high speed.
In general we indicate certain recommendations for environments with a large amount of data:
- Perform the migration before connecting the pandorafms environment to the new opensearch.
- Perform a staggered migration, year by year, month by month or even index by index, using the pattern filter. Depending on the amount of data stored.
- Right-size the environment.
- For a large amount of data, use an opensearch cluster instead of a single node.