OpenSearch installation and configuration

• Rocky Linux 8 / RHEL 8 / Ubuntu 22.04 (recommended operating systems).
• Minimum 4 GB RAM (testing, dev), recommended 8 GB of RAM for each OpenSearch instance (minimum base requirements, for each environment and amount of data to be processed and/or stored, the specific requirements must be estimated).
• Disable SWAP on the node or nodes where OpenSearch is.
• Minimum 4 CPU cores (minimum base requirements, for each environment and amount of data to be processed and/or stored, the specific requirements must be estimated).
• 50 GB system storage.
• 100 GB OpenSearch storage (minimum base requirements, for each environment and amount of data to be processed and/or stored, the specific requirements must be estimated).
• Connectivity from Pandora FMS Server and Web Console to the OpenSearch API (default port 9200/TCP) and between cluster nodes (default port 9300/TCP).

A single node environment with these features can store up to 1 GB of data per day and store it for 30 days. In the case of requiring greater data resilience, greater data processing and storage, and fault tolerance, the configuration of an OpenSearch cluster will be necessary (with a minimum of 3 nodes to guarantee data integrity). By switching to a cluster environment, it is also possible to distribute the load between the nodes, doubling (in the case of 3 nodes) the processing capacity of the environment. A load balancing system will be necessary (Keepalived, for example) if you want to work with the different nodes simultaneously.

Official OpenSearch documentation for installation:

https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index /

Before running OpenSearch on your machine, you should disable memory paging and swap on the host to improve performance and increase the number of memory maps available to OpenSearch. See “Important Settings” for more information:

https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/#important-settings

  # Disable memory paging and swapping.
sudo swapoff -a

# Edit the sysctl config file that defines the host's max map count.
sudo vi /etc/sysctl.conf

# Set max map count to the recommended value of 262144.
vm.max_map_count=262144

# Reload the kernel parameters.
sudo sysctl -p

For Rocky Linux 8, installation via RPM package is recommended.

Package listing: https://opensearch.org/downloads.html

Official installation documentation: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/rpm/

Once OpenSearch is installed, access to OpenSearch must be checked from Pandora FMS. Before performing this test you must configure the node or cluster. For this installation check, execute:

curl -X GET https://<ip_opoensearch_box>:9200 -u 'admin:admin' --insecure

You should get a response similar to:

  {
     "name" : "hostname",
     "cluster_name" : "opensearch",
     "cluster_uuid" : "6XNc9m2gTUSIoKDqJit0PA",
     "version" : {
        "distribution" : "opensearch",
        "number" : <version>,
        "build_type" : <build-type>,
        "build_hash" : <build-hash>,
        "build_date" : <build-date>,
        "build_snapshot" : false,
        "lucene_version" : <lucene-version>,
        "minimum_wire_compatibility_version" : "7.10.0",
        "minimum_index_compatibility_version" : "7.0.0"
     },
     "tagline" : "The OpenSearch Project: https://opensearch.org/"
  }

By default, the OpenSearch installation enables SSL, username and password which is a good practice; It is recommended to change the default username and password.

This file contains the configuration of all the parameters of theOpenSearch service; See the official documentation for more information:

https://opensearch.org/docs/latest/install-and-configure/configuration/

Minimum configurations necessary to start the service and its use with Pandora FMS.

• Port number.

# ---------------------------------- Network
# Set the bind address to a specific IP (IPv4 or IPv6):
network.host: 0.0.0.0
# Set a custom port for HTTP:
http.port: 9200
# For more information, consult the network module documentation.

• Location of stored data and logs:

# ----------------------------------- Paths
# Path to directory where to store the data (separate multiple locations by comma):
path.data: /var/lib/opensearch
# Path to log files:
path.logs: /var/log/opensearch

It will also be necessary to uncomment and define the following lines:

cluster.name: pandorafms
node.name: ${HOSTNAME}
network.host: 0.0.0.0

• cluster.name: This will be the name that the group or cluster will receive.
• node.name: To name the node using the ${HOSTNAME} system variable, it will automatically take the name of the host.
• For network.host the value 0.0.0.0 allows OpenSearch to “listen” on all network interfaces (NIC); To use a specific NIC, enter a corresponding specific value.

If you work with a single node, add the line to the configuration file to allow the single node to start:

discovery.type: single-node

In case of working with a cluster you need to complete the discovery.seed_hosts parameter:

discover.seed_hosts : ["ip:port", "ip", "ip"]

To start OpenSearch, execute:

systemctl start opensearch.service

To restart use restart, to stop stop and status to check the status.

Remember that to check the installation and operation of OpenSearch you may run:

curl -X GET https://<node-ip> -u 'admin:admin' --insecure

To configure an OpenSearch cluster, follow the official documentation:

https://opensearch.org/blog/optimize-opensearch-index-shard-size/

To change the default password from admin, a series of steps must be followed. The first thing is to export the variable to use the Java® JDK installed by OpenSearch to use any of the tools:

export OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk

Then, to generate the hashed password to place in the OpenSearch configuration file, the following script is used (replace < password > with the password to use):

/usr/share/opensearch/plugins/opensearch-security/tools/hash.sh -p <password>

For example:

Then open the file /etc/opensearch/opensearch-security/internal_users.yml with the text editor vim or nano to modify the password of the required user or users.

Example file:

To make the changes effective, the following must be executed:

cd /usr/share/opensearch/plugins/opensearch-security/tools

OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk ./securityadmin.sh -cd /etc/opensearch/opensearch-security/ -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/kirk.pem -key /etc/opensearch/kirk-key.pem -icl -nhnv-t internalusers -icl -nhnv -cacert /etc/opensearch/root-ca.pem -cert /etc/opensearch/kirk.pem -key /etc/opensearch/ kirk-key.pem

A final message Done with success should be displayed; to check the new password (following the previous example with pandora used):

For more information on user management in OpenSearch:

• https://opensearch.org/docs/latest/security/configuration/yaml/
• https://opensearch.org/docs/latest/security/access-control/users-roles/#create-users

To configure Pandora FMS with OpenSearch see the topic “Log collection and monitoring”.

Before setting an environment into production, whether it is a single node or a data cluster, it is recommended to apply the corresponding configurations to this node or cluster based on its use. In the case of indexes generated by Pandora FMS, the most effective way to do so is by defining a template to define the configuration of the fields and the stored data.

To create a basic template, you only need to define the following fields:

curl -X PUT -ku 'admin:admin' https://<node_ip>:9200/_index_template/pandorafms -H 'Content-Type: application/json' -d'
{
  "index_patterns": [
    "pandorafms*"
  ],
  "template": {
    "aliases": {
      "pandorafms_logs": {}
    },
    "settings": {
      "number_of_shards": 1,
      "auto_expand_replicas" : "0-1",
      "number_of_replicas": "0"
    },
"mappings" : {
     "properties" : {
       "agent_id" : {
         "type" : "long"
       },
       "group_id" : {
         "type" : "long"
       },
       "group_name" : {
         "type" : "text"
       },
       "logcontent" : {
         "type" : "text"
       },
       "source_id" : {
         "type" : "text"
       },
       "suid" : {
         "type" : "text"
       },
       "type" : {
         "type" : "text"
       },
       "utimestamp" : {
         "type" : "long"
       },
       "@timestamp": {
          "type": "date"
        }
     }
   }
}
}
'

Through Pandora FMS (menu) interface you may upload said template:

• PUT _template/<templatename>: In this example PUT _template/pandorafms .

You may also check the templates through Pandora FMS interface itself:

• GET _template/<templatename>: In this example GET _template/pandorafms .

To define a multinode template, take into account the following information:

• When configuring the template (JSON format), you need to configure as many shards as nodes you have, however to correctly configure the replicas subtract 1 from the number of nodes in the environment.

For example, in a Pandora FMS environment with 3 nodes configured, when you modify the fields number_of_shards and number_of_replicas it should look like this:

{
  "index_patterns": ["pandorafms*"],
  "settings": {
    "number_of_shards": 3,
    "auto_expand_replicas" : "0-1",
    "number_of_replicas" : "2"
  },

From the command line you may list the environment templates by executing:

curl -X GET "localhost:9200/_cat/templates/*?v=true&s=name&pretty"

You may also see the details of a template, for example, created for pandorafms by running:

curl -X GET "localhost:9200/_template/pandorafms*?pretty"

which will return the configuration you defined in JSON format.

You may perform these operations through Pandora FMS interface:

• PUT _template/<template_name> {json_data}: It allows you to enter the data of the template to be created.
• GET _template/><template_name>: It allows you to see the created template.

To configure Pandora FMS with OpenSearch see “Log collection and monitoring”.

Return to Pandora FMS documentation index