Pandora: Documentation en: Virtual environment monitoring

Go back Pandora FMS documentation index

1 Virtual Environment Monitoring

https://wiki.pandorafms.com/index.php?title=Pandora:Documentation_en:Virtual_environment_monitoring

Amazon EC2

VMware

RHEV

OpenNebula

IBM HMC

HPVM

1.1 Amazon EC2

This specific monitor utilizes the CloudWatch API to monitor your instances in an Amazon EC2 environment. You're required to have the CloudWatch service enabled in your instance. Feel free to download the EC2 module from the Pandora FMS Module Library.

The main idea of this remote server plugin is to obtain information from your instances by using the network server plug-in. That means you're required to register the plugin on the server and create different modules to obtain the information from your EC2 Servers.

This is an execution example:

/home/slerena/ec2_plugin.sh -A AKIAILTVCAS26GTKLD4A -S CgmQ6DxUWES05txju+alJLoM57acDudHogkLotWk -i i-9d0b4af1 -n AWS/EC2 -m CPUUtilization

It will return a numeric percentage value of the 'CPU Utilization' metric in the instance named 'i-9d0b4af1'.

To install it, you're required to:

1. Have a running JAVA environment and a JAVA home directory. In the Pandora FMS Appliance (VMware/Image) it's located under '/usr/'.
2. Copy this plugin to a path, change the permissions to '755' and enter the base path on the 'AWS_CLOUDWATCH_HOME' variable which is located among the first lines of the plugin. It is necessary to edit the first lines of the plugin.

The plugin consists of several files:

/ec2_plugin.sh: The plugin itself /bin/*: The components of Amazon CloudWatch command-line (monitoring) tools are included in this bundle. The scripts contained in there are distributed under the Apache License.

Please put the whole package in a directory on the server, e.g:

/usr/share/pandora_server/plugin/ec2 

and change the 'AWS_CLOUDWATCH_HOME' variable to '/usr/share/pandora_server/plugin/ec2'.

If you have any doubts about whether it's correctly installed or not, feel free to execute this command to test it:

/usr/share/pandora_server/plugin/ec2/mon-cmd --version 

It should return something like this:

Amazon CloudWatch CLI version 1.0.9.5 (API 2010-08-01) 

If it returns approximately the same string, you're ready to use the plugin.

If not, you're probably required to install and configure the Amazon CloudWatch command-line monitoring tools properly. Please follow these steps to do so:

1.1.1 Installation

Requirements:

1. Please ensure that a JAVA version from version 1.5 or higher is installed on your system (the command to check this is 'java -version').
   
2. Unzip the installation's zip package.
   
3. Set the following environment variables:
   
   1. 'AWS_CLOUDWATCH_HOME': The directory where the deployment files to check with were copied to:
      

 UNIX: 'ls ${AWS_CLOUDWATCH_HOME}/bin' (should list 'mon-list-metrics')

 Windows: 'dir %AWS_CLOUDWATCH_HOME%\bin' (should list 'mon-list-metrics')

1. 1. JAVA_HOME - Home directory of the Java installation
      
   2. Add '{AWS_CLOUDWATCH_HOME}/bin' to your path (under Windows it's: '%AWS_CLOUDWATCH_HOME%\bin')
      

1.1.2 Configuration

Please provide your AWS user credentials by using the command-line tools. There are two ways to provide the credentials:

1. Using AWS keys.
2. Using X.509 certificates.

1.1.3 Using AWS Keys

1. . Explicitly specify credentials in the command line: --I ACCESS_KEY --S SECRET_KEY
2. . Create a credentials file. The deployment includes a template file:

 ${AWS_CLOUDWATCH_HOME}/credential-file-path.template.

• Edit a copy of this file to add its information.
• In UNIX, limit permissions for the owner of the credentials file:

$ chmod 600 <the file created above>.

Once the file is created, there are several ways to reference it:

• Set the following environment variable:

export AWS_CREDENTIAL_FILE=<credentials file>

• You can also provide the following option with each command

--aws-credential-file <credentials file>

1.1.4 Using X.509 Certificates

1. Please save your certificate and private keys to e.g. 'my-cert.pem' and 'my-pk.pem' files.

2. There are two ways to provide the certificate information to the command line tool:

• Please set the following environment variables:

       EC2_CERT=/path/to/cert/file
       EC2_PRIVATE_KEY=/path/to/key/file

• Please specify the files for every command directly on the command-line:

       <command> --ec2-cert-file-path=/path/to/cert/file --ec2-private-key-file-path=/path/to/key/file

1.1.5 Setting Custom JVM Properties

By setting the environment variable 'SERVICE_JVM_ARGS', you can pass arbitrary JVM properties to the command line. For example, the following line sets the proxy server properties under Linux/UNIX: export SERVICE_JVM_ARGS="-D http.proxyHost=http://my.proxy.com -Dhttp.proxyPort=8080"

1.1.6 Running

1. Please check whether your setup works properly and execute the following command:

  $ mon-cmd --help

You should be able to see the usage page for all monitoring commands by this command:

  $ mon-list-metrics --headers

You should see a header line here. If you have any metrics defined, you should see them as well.

1.2 VMware

Virtual environments are very important for IT architectures, that is why monitoring these environments is crucial for the proper performance of your company. With Pandora FMS Enterprise you're able to install the VMware Monitoring Plug in, which allows you to control VMware architectures easily.

1.2.1 Monitoring VMware Architecture

With this system, it's possible to monitor architectures like the one below.

Pandora FMS monitors ESXi servers, Data stores and Virtual Machines. Pandora FMS utilizes the provided VMware web API to collect the data.

1.2.2 Monitoring by the VMware Monitoring Plug In

VMware monitoring is based on several components:

1. An agent plug in that discovers all the entities of your environment and collects the information to monitor. For Pandora FMS 5.0 and higher versions, the plug in is also able to update some agent parameters required by the extensions. Furthermore, it can copy the events from the VMware vCenter to the Pandora FMS Console.
   
2. A basic configuration extension in order to execute the VMware plug in from the Enterprise image file (ISO) without any problems.
   
3. VMware View and VMware Manager are extensions which allow you to manage virtual machines easily and provide a view of the whole environment.
   

By using this system, you're able to get an agent for every ESXi, Data store and virtual machine found as well as an agent which represents the Data Center. This allows you to manage the entities regardless of the relationship between them within the virtual architecture. Furthermore, each agent possesses the configured modules and is ready to be monitored according to its VMware entity type.

1.2.2.1 Internal Plug-In Execution

For versions 5.0 and above, the agent plug in performs all the features: Entity self-discovery, data collection, copying of events and custom field configuration.

For each discovered entity, the plug in sends an XML file to the Pandora FMS Server to be processed. This entity is going to become a Pandora FMS Agent. The events copied into the console are going to appear in the event view and managed as events generated by the system.

The plug in has some auxiliary files with information which is related to the monitoring configuration: Logs, monitored entities, event monitoring pointer, etc.

Since it certainly would negatively affect the monitoring performance, it's very important not to delete those files.

1.2.3 Plug-In Requirements

These are the requirements to ensure the proper performance of the VMware plug in:

• Installation of Pandora FMS 4.0.3 or higher versions (the extensions only work from Pandora FMS 4.0.3 Patch 130310 and above)
• The Pandora FMS Agent must be installed on the machine.
• A Linux, UNIX or Windows operating system is required.
• The curl command for versions 5.0 or higher has to be available. The Pandora FMS Windows Agent incorporates the (curl) command.
• The ping command has to be available.
• The installation of VMware vSphere SDK for Perl is required.

Note: In the last revisions of the plugin, curl has been replaced by perl-LWP.

1.2.4 Configuring vCenter for Monitoring

The plug in uses vCenter performance counters. Performance counter availability depends on the statistics which collect the levels configured in the vCenter.

These levels could be changed from the vCenter Server Settings menu and the Statistics option. You're going to see a description of the counters, collected by vCenter for each time option and level. The minimum level to use the Pandora FMS Monitoring Plug In is Level 2.

Depending on the vCenter configuration, some modules may not report their data to Pandora FMS. There are three possible reasons for that:

• There's a plug in missing in the vCenter.
• A VMware agent must be installed on the entity.
• The entity (or the ESXi virtual machine) is simply switched off.

Consult the VMware documentation to solve these problems.

Some solutions, such as the hardware status monitoring, may require an advanced configuration of both: The vCenter and the host which supports the ESX.

1.2.5 VMware vSphere SDK for Perl Installation

Get the VMware software by visiting their Download Center.

1.2.5.1 Installing the Linux SDK

This procedure was tested for SDK versions 4.1 and 5.1.

It's always recommended to utilize the SDK version with its corresponding VMware software version, e.g.: The 4.1 VMware software is recommended to be used with the version 4.1 of the SDK.

Decompress the SDK package by the following command first:

# tar -xzvf VMware-vSphere-Perl-SDK-x.x.x-xxxxxx.i386.tar.gz

Then, compile and install the SDK by the following commands:

# perl Makefile.PL
# make
# make install

If the SDK was installed successfully without the appearance of any errors, you can connect with the vCenter by using the following command:

# /usr/lib/vmware-viperl/apps/general/connect.pl --server <vcenter_ip> --username <vcenter_user> --password <vcenter_pass>

The command response should be something like this:

Connection Successful
Server Time : 2013-02-21T16:24:05.213672Z

1.2.5.2 SDK Setup under Windows

The version of Perl which was shipped with the vSphere SDK doesn't work with VMware's PERL libraries. Please follow these steps to fix this problem:

• Install the VMware vSphere SDK.
• Install the Strawberry PERL version 5.12.
• Copy the directory named 'C:\Program Files\VMware\VMware vSphere CLI\Perl\lib\VMware' to 'C:\strawberry\perl\lib'.
• Uninstall the VMware vSphere SDK.

1.2.6 Installing the Plug In by the VMware Settings Extension

This extension is going to be installed by the ISO of the Enterprise Version 5.1 by default.

By the VMware Settings extension, we're able to set up the VMware plug-in execution directly from the Pandora FMS Console. This extension appears under 'Administration' -> 'Setup' and 'VMware'.

Within it, we're able to set the plug-in's path, the config's file path and the parameters 'V-Center IP', the data center's name, 'username', 'password' and the runtime plugin.

The execution of the plug in is added as a new Cron Job which can be executed every 5, 10 or 15 minutes. The execution of the Pandora FMS cron extension, which has to be added in the file '/etc/crontab', is required to be configured as shown below.

*/1 * * * * root wget -q -O http//localhost/pandora_console/enterprise/extensions/cron/cron.php >> /var/www/html/pandora_console/pandora_console.log

This extension requires the installation of Pandora FMS 5.1 to be compatible to the implementation of the Pandora FMS cron job and has to be configured to an interval of one minute. Without this requirement, the extension is not going to work properly.

After configuring the plug in, a new task is added within the cron jobs by the following settings:

This particular cron job can only be set from the VMware Settings extension. Any changes made from a different location or tool within the cron job is going to cause a malfunction of the plug in.

1.2.7 Manual plugin installation

1. Change your working directory to where you extracted VMware plugin files

2. Copy extensions to pandora_console/enterprise/extensions/

sudo -u apache cp -R extensions/vmware* /var/www/html/pandora_console/enterprise/extensions/

3. Copy vmware-plugin.{pl,conf}, to appropriate directory

sudo cp vmware-plugin.pl vmware-plugin.conf /usr/share/pandora_server/util/plugin/
sudo chown pandora:apache /usr/share/pandora_server/util/plugin/vmware-plugin.{pl,conf}
sudo chmod g+w /usr/share/pandora_server/util/plugin/vmware-plugin.conf

4. Edit vmware-plugin.conf

tentacle_ip: IP address of monitoring server    
pandora_url: "http://127.0.0.1/pandora_console" 
server: IP address of vCenter              
datacenter: Center Name                   
user: account for vCenter                
pass: password for vCenter               

server, datacenter, user, pass can be set at Pandora Console

5. Visit "Setting" screen at Pandora Console and setup API password

in example

api password: 1234

6. Copy vmware-plugin.{pl,conf} for Pandora Agent

sudo cp /usr/share/pandora_server/util/plugin/vmware-plugin.{pl,conf} /etc/pandora/plugins/

There is no vmware-plugin-events.conf in the tar, but you can create it by copying the vmware-plugin.conf and modifying 'event_mode' to 1 by hand.

1.2.8 Setup and Commissioning of the Plug-In Agent

In order to install the plug-in agent, please copy the 'vmware-plugin.pl' and 'vmware-plugin.conf' files to the folder '/etc/pandora/plugins' by the following command:

cp vmware-plugin.pl vmware-plugin.con /etc/pandora/plugins

Then add a new module plug-in type to the agent configuration file by the following line:

module_plugin /etc/pandora/plugins/vmware-plugin.pl /etc/pandora/plugins/vmware-plugin.conf

If you also wish to copy events, create another plug-in module with a different configuration file that enables to copy events. The command would be like this:

module_plugin /etc/pandora/plugins/vmware-plugin.pl /etc/pandora/plugins/vmware-plugin-events.conf

Under a Windows system, you're required to specify the interpreter you're intending to use. The command could e.g. look like this:

module_plugin perl "C:\Program Files\pandora_agent\util\vmware-plugin.pl" "C:\Program Files\pandora_agent\util\vmware-plugin.conf"

The following sections explain the parameters of the plug-in configuration file in detail.

Since the VMware plug in uses a very heavy SOAP API, it takes too much time to execute some tasks. On systems with a large number of entities to monitor, it may be necessary to share and distribute the load among various Pandora FMS Software Agents. All relevant information is going to be provided in the sections below.

If you are using Pandora FMS 5.0 or a higher version and you intend to use the plug-in extensions or any event monitoring, you're required to properly configure the Pandora API. You're also required to provide an API password and access to the relevant addresses in the API access list to do so. These fields are defined in the general configuration of the Pandora FMS Console.

1.2.9 Monitoring the VMware Virtual Architecture

To see the result of the plugin's execution go to 'Monitoring' > 'Views' > 'Agent Detail' to do so.

The picture below shows the agents created by the plug in along with the other Pandora FMS agents.

If we click on the name of an agent we will see the Pandora FMS agent view where the modules monitored by the VMware plugin will appear. .

The plug in displays a basic monitoring for every VMware element by default. The default settings for these entities consist of the following:

1.2.9.1 Default Modules for the Data Center

• Ping
• Check 443 port

1.2.9.2 Default Modules for the Data Store

• Capacity
• Free Space
• Disk Overallocation
• Free Space Bytes

1.2.9.3 Default Modules for ESXi

• CPU Usage
• Memory Usage
• Data received
• Data transmitted
• Disk Read Latency
• Disk Write Latency
• Host Alive

1.2.9.4 Default Modules for Virtual Machines

• CPU Usage
• Memory Usage
• Tools Running Status
• Host Alive
• Disk Free
• Disk Read Latency
• Disk Write Latency
• Data received
• Data transmitted

In the following section, all available modules and information reported by them will be explained in detail.

1.2.10 VMware Virtual Architecture Agent Modules

Some modules may not be available, depending on the VMware version and environment settings. In the following tables, the available modules and their features will be described.

The plug in allows you to configure Custom Performance Counters for ESX hosts and virtual machines. The details on how to create those custom counters is described in the sections below, where the contents of the configuration file are described in detail.

1.2.10.1 Modules for the Data Center

Module	Description	API Version	Availability
Ping	ping check ping to the machine which supports vCenter	All	Always
Check port 443	Check the port 443 on the machine that supports vCenter	All	Always

1.2.10.2 Modules for Data Store Agents

Module	Description	API Version	Availability
Capacity	Maximum capacity of the Data Store in bytes	All	Always
Free Space	Percentage of free space on the Data Store	All	Always
Disk over-allocation	Disk over-allocation percentage	≥v4.0	Always
Free Space Bytes	Amount of free disk space in bytes	All	Always

1.2.10.3 Modules for Agents of the ESXi Host Type

Shows whether connected or not

</tr>

Module	Description	API Version	Availability
Boot Time	Last time the host was booted	All	Always
CPU Info [x]	General CPU information (it creates one module for each ESXi CPU)	All	If connected.
Memory Size	Total amount of the host's physical memory in bytes	All	If connected.
Overall CPU Usage	Addition of the use of all CPUs in MHz	All	If connected.
Overall Memory Usage	Used physical memory on the host in MB	All	If connected.
Power State	State of the host's power.	≥v2.5	Always.
SSL Thumbprint	Host SSL print	≥v4.0	If configured.
Uptime	Uptime of the host in seconds	≥v4.1	If configured.
VNIC Info [x]	Information about the host's virtual network interfaces	All	If connected and configured.
Host Alive	Module KeepAlive type. Value is '1' if the ESX is connected and '0' if it's not.	All	Always.
Connection State	State of the host's connection.	All	Always.
Disk Read	Rate of read Kb/s of the disk	All	Stats Level ≥2
Disk Write	Rate of written Kb/s of the disk	All	Stats Level ≥2
Disk Read Latency	Latency of the disk reading in milliseconds	All	Stats Level ≥2
Disk Write Latency	Latency of the disk writing in milliseconds	All	Stats Level ≥2
Data received	Range of host received Kb/s	All	Stats Level ≥2
Data transmitted	Range of host sent Kb/s	All	Stats Level ≥2
Packages Received	Number of packages received in the interval	All	Stats Level ≥2
Packages Transmitted	Number of packages sent in the interval	All	Stats Level ≥2
CPU Usage	Percentage of CPU usage	All	Stats Level ≥2
Memory Usage	Percentage of RAM usage	All	Stats Level ≥2
Net Usage	Sent and received data from all NICs	All	Stats Level ≥2
Disk Rate	Aggregated I/O rate in KB/sec	All	Stats Level ≥2
Max. Disk Latency	Max. latency of all disks	All	Stats Level ≥2
HA Status	Host HA status	≥v5.0	If configured.
Sensor*	Status of the hardware sensors (one module per sensor)	All	ESXi >= 3.5

1.2.10.4 Modules for Virtual Machine-type Agents

These modules provide information from a VMware architecture's point of view. If you wish to monitor other parameters related to virtual machine you're also required to consider other options such as Monitoring with Software Agents or Remote Monitoring.

Module	Description	API Version	Availability
Boot Time	Last date where the virtual machine was started.	All	If connected.
Connection State	Connection state	All	Always.
Consumed Overhead Memory	Memory consumed by the virtual machine in MB.	≥v4.0	If configured.
CPU Allocation	Information about the resources assigned to the virtual machine's CPU.	All	If configured.
Disk Free [x]	Free disk percentage of the virtual machine (there will be one module for each disk the virtual machine contains).	All	If configured.
Guest State	Host's operating system's operating mode.	All	If configured.
Host Info	Information about the VMware host	All	If configured.
Host Alive	Module of 'KeepAlive' type. Value is '1' if the virtual machine is executed and '0' if not.	All	Always.
Host Memory Usage	Consumed memory by the virtual machine in MB.	All	If connected.
Host Name	Name of the host's operating system.	All	If configured.
IP Address [x]	System's IP address (It's going to show one for each available network interface.)	≥v4.1	If configured.
MAC Address [x]	System MAC address (It's going to show one for each available network interface).	All	If configured.
Max. CPU Usage	Maximum limit of the virtual machine's CPU usage.	All	If configured.
Max Memory Usage	Maximum limit of the virtual machine's RAM.	All	If connected.
Memory Allocation	Limit of the resources for the memory	All	If connected.
Memory Overhead	The the virtual machine's used memory above the requirements of the host's operating system in Bytes.	All	If configured.
Overall CPU Demand	Basic statistics on the CPU performance in MHz.	≥v4.0	If connected.
Overall CPU Usage	Basic statistics on the CPU usage in MHz.	All	If connected.
Power State	Current state of the virtual machine's power.	All	Always.
Private Memory	The virtual machine's given memory in MB of non-shared memory.	≥v4.0	If connected.
Shared Memory	The virtual machine's given memory in MB of shared memory.	≥v4.0	If connected.
Tools Running Status	Current state of executed VMWare tools installed on the host's operating system.	≥v4.0	If configured.
Trigger Alarm State	State of the VMware alarms	All	If configured.
Uptime Seconds	Virtual machine uptime in seconds.	≥v4.1	If connected.
Virtual Image Path	Virtual machine configuration file path (.vmx).	All	Always.
Disk Read	Rate of the disk read Kbps	All	Stats Level ≥2
Disk Write	Writing speed of the disk in Kb/s.	All	Stats Level ≥2
Disk Read Latency	Disk reading latency in milliseconds.	All	Stats Level ≥2
Disk Write Latency	Disk writing latency in milliseconds.	All	Stats Level ≥2
Data Received	Host Kb/s received range.	All	Stats Level ≥2
Data Transmitted	Host's sent range in Kb/s.	All	Stats Level ≥2
Packages Received	Number of received packages in the interval.	All	Stats Level ≥2
Packages Transmitted	Number of transmitted packages in the interval.	All	Stats Level ≥2
CPU Usage	CPU usage percentage.	All	Stats Level ≥2
Memory Usage	RAM usage percentage.	All	Stats Level ≥2
Net Usage	Sent and received data of all NICs.	All	Stats Level ≥2
Disk Rate	Aggregated I/O rate in KB/sec.	All	Stats Level ≥2
Max. Disk Latency	Max. latency of all disks.	All	Stats Level ≥2
HeartBeat	Number of virtual machine's heartbeat.	All	Stats Level ≥2
CPU Ready	Percentage of time when machine is ready but not scheduled on a physical CPU.	All	Stats Level ≥2
Number Snapshots	Number of snapshots for the virtual machine (This module affects the monitoring performance. We strongly recommend executing it with a high value, e.g. every hour).	All	If configured.
HA Status	HA status for the virtual machine.	≥v5.0	If configured.

Some modules may require to have the VMware tools installed

1.2.11 VMware Event Monitoring

This feature was created to copy event information from the VMware vCenter to Pandora FMS.

These events belong to the Pandora FMS Event Management work flow and are associated to the agent which represents the vCenter (if any) automatically. The picture below shows an example of the events generated.

The copy process respects all the information and severity degree which VMware assigns to them on event creation. The events with 'critical', 'warning' or 'information' severity levels preserve these levels in Pandora FMS. The following picture is an example of the detailed information under Pandora FMS.

Related to the events in Pandora FMS, you could perform all actions available for event management, e.g. alert creation, filter configuration, incident creation, etc.

1.2.12 VMware Virtual Architecture Management and Visualization

Two extensions are distributed along with the VMWare plug in: The VMware Manager and VMware View. VMware View allows you to easily see all the VMware architecture components. By the VMware Manager, you're also able to manage virtual machines, stopping, starting, resetting or canceling the activity from the Pandora FMS Console. These extensions are optional and are solely going to work in conjunction with Pandora FMS 4.0 or newer versions.

From the plug-in versions 4.1 and above, these extensions are encompassed by a single extension, which in turn is divided into the two above cited, and one last extension called VMware Settings. This latest extension is supported from version 5.1 of Pandora FMS and above only.

1.2.12.1 Using VMware View Extensions

To start using the VMware architecture view, please click on 'View Agents' -> 'VMware View' in the monitoring menu.

The VMware View extension is going to display a map with all the VMware architecture discovered.

The map bears elements of the VMware architecture (virtual machines, ESX, Data Stores and Data Centers) with different icons that identify them and the Pandora FMS agents state that represent each element. Besides the relationship that exists between the virtual machines, ESX and the Data Center are shown. Therefore, you can easily check the state of the VMware architecture at a glance.

This extension comes with some options which might help you to improve the architecture visualization by allowing you to hide elements, enlarge the character size and zoom in and out:

By using the previous options, you could only see the Data Center and the ESX with a font size of '14' and a zoom size of 2x.

1.2.12.2 VMware View Dashboards (5.0 or higher)

For Pandora FMS 5.0 or higher versions, the VMware View extension comes with two additional map views of the virtual architecture topology. The two additional tabs allow you to switch between different views of the VMware View Extension.

The first view is a general dashboard where you're able to see the general virtual architecture in numbers at a glance, e.g. how many virtual machines there are or how many Data Stores or ESXi hosts might have a problem. There are also graphs which are going to show the virtual machines which have the highest memory, CPU, disk and network consumption of the entire virtual architecture. You're also able to easily check for general performance parameters at a glance.

The second view allows you to check for the performance parameters of each ESX host. Using this view, you may choose e.g. an ESX host for which a dashboard with the status of the host and virtual machines, metrics relating to the usage of CPU, memory, disk and network ESXi host will be displayed. This also offers a graphical view of the virtual machines with the highest resource (CPU, memory, disk and network) consumption.

1.2.12.3 Using the VMware Manager Extension

To use the VMware Manager extension, go to the operating view of one agent which corresponds with a virtual machine in the VMware architecture. You can see an icon with the VMware symbol which corresponds to the extension.

The VMware Manager Extension allows you to manage virtual machines from the Pandora FMS Console. The extension shows the current state of the virtual machine with a color code (green=on, orange=off and grey=stopped). It also shows the availability status in a combo and allows you to change the status of the virtual machine by selecting it on the 'Change Status' button.

As shown on the image below, you can stop a running virtual machine by selecting the 'Stop' status by this extension:

It stops the machine and changes the VMware Manage extension view. As you can see on the image below, the machine is stopped:

This extension requires the installation of VMware SDK for Perl on the same machine where Pandora FMS is installed. The extension is not going to work without it.

1.2.13 Plug-In Configuration

The VMware plug in detects all entities and adds the standard checks by default. You're able to setup the monitoring and to decide which variables you intend to monitor by using the configuration file.

The configuration file contains all the information necessary for monitoring, consolidated in the following sections: Configuration, Rename, Reject, Datacenter, Datastore, ESX and VM. Subsequently, each section explains its possible configuration.

All errors related to the configuration file are explained in the Error Log Server and in the Event Viewer of Pandora FMS. You can locate any problems in the configuration file by consulting these sources.

1.2.13.1 Configuration File

1.2.13.1.1 Global Configuration

The general configuration is defined by the token named Configuration and contains the following parameters:

• Server: The vCenter's IP.
• User: The vCenter's user.
• Pass: The vCenter's password.
• Datacenter: The Data Center you intend to monitor.
• Temporal: The temporary directory.
• Logfile: The log file's location.
• entities_list: The file location, containing the list of the monitored entities.
• transfer_mode: The transfer mode for XMLs. It can be 'tentacle' or 'local'.
  • Tentacle: It sends XMLs files to the Pandora FMS Server by using the Tentacle protocol.
  • Local: It copies files found in a local folder. The agent is required to be executed on the same machine on which the local folder is located.
• tentacle_ip: The Pandora FMS Server IP to which the information is sent.
• tentacle_port: The Pandora FMS server port to which the information is sent (default value is '41121').
• tentacle_opts: Some additional options for sending with Tentacle (default value is 'none').
• local_folder: The destination directory to copy XMLs with local mode turned on.
• pandora_url: The Pandora FMS console's URL (e.g. 'http://192.168.70.81/pandora_console').
• api_pass: The Pandora FMS API password.
• api_user: The Pandora FMS Console user.
• api_user_pass: The Pandora FMS Console's user password.
• retry_send: Actives (1) or deactivates (0) the .data files resend.
• event_mode: The flag which enables the event collecting mode. If it's set to '1', the event collecting mode is enabled. If it's set to '0', the event collecting mode is disabled.
• event_pointer_file: The temporary file location which stores the pointer to the collection events.
• Verbosity: The log level (please set it to '0' for errors which prevent the plug-in's operation and to '1' for all errors).
• Threads: The number of plug-in threads (default value is '1').
• Interval: The agent's interval which represents the VMware entities.

An example of this section in the configuration file might be the following:

Configuration
server 192.168.70.249
user Administrator
pass S1stemas
datacenter artica
temporal /tmp
logfile /tmp/vmware_plugin.log
entities_list /tmp/vmware_entities_list.txt
transfer_mode tentacle
tentacle_ip 192.168.70.81
tentacle_port 41121
tentacle_opts
local_folder /var/spool/pandora/data_in
pandora_url http://192.168.70.81/pandora_console
api_pass 1234
api_user admin
api_user_pass pandora
event_mode 0
event_pointer_file /tmp/vmware_events_pointer.txt

If you intend to use the plug in under Windows, you're going to have to change all file paths for a compatible routing.

1.2.13.1.2 Entity Renaming

The token Rename is used to rename the entities discovered by the plug in. By using this feature, the agents created under Pandora FMS are going appear with a newly assigned name. The syntax is shown below:

<current name> TO <new name>

A good configuration example could be like the one below.

#Rename entities
Rename
Debian 11 TO Virtual Machine 1
RedHat 12 TO Web server 
ESX Workstation TO Host Work Sales

1.2.13.1.3 Entity Dismissal

The plug in allows you to dismiss entities by type or individually. Both options are explained below.

The dismiss function uses the token Reject to dismiss entities. In this section, you can dismiss entities according to their type, e.g. all virtual machines or all ESX hosts. The accepted values for this function are the following: all_datastore, all_datacenter, all_esx and all_vm.

A configuration for this section (which would dismiss all the entities) would be like the one shown below:

#Dismissed entities
Reject
all_datastore
all_datacenter
all_esx
all_vm

To dismiss entities individually, you have to delete the entity's file which created by the plug in. The plug in creates a file on the location which is indicated by the parameter entities_list (it's '/tmp/vmware_entities_list.txt' by default). This plug in provides the content of this file in the moment of first execution or creates a list with all the discovered entities (if it doesn't already exist). A good example of this file could be like the one below:

Datacenter
artica
Datastore
datastore_1
datastore2
ESX
192.168.70.252
VM
Pandora FMS 4.0.3
Debian2
Debian3
Debian4
Redhat
debian5
Debian6
Debian8
Debian7
Debian11
Debian10
Debian9
NSM
Pandora
vcenter
suse11.2

The configuration file is divided into several tokens: Datacenter, Datastore, ESX and VM where the different entities are listed. Once the configuration file is created, the plug in is going to read the entities to monitor. If you intend to dismiss a certain entity, you just have to delete it from the folder. If you e.g. don't want to monitor the following entities: Debian2, datastore2, NSM, suse11.2 and 192.168.70.252, the configuration file has to be like the one below:

Datacenter
artica
Datastore
datastore_1
ESX
VM
Pandora FMS 4.0.3
Debian3
Debian4
Redhat
debian5
Debian6
Debian8
Debian7
Debian11
Debian10
Debian9
Pandora
vcenter

This feature allows you to distribute the monitoring load by limiting the number of monitored entities in every plug-in execution. Some more load distribution techniques are explained below:

1.2.13.1.4 Monitoring Configuration

The next file sections configure the created modules for every type of entity. These sections use the Data Center, Data Store, ESX and VM sections. In these sections, you can enable and disable modules to monitor. For the following example, we have created a configuration according to the modules which we'd like to create for the ESX and virtual machines.

...
#ESX Modules
ESX
cpuUsagePercent disabled
diskRead enabled
diskWrite enabled

#VM Modules
VM
diskReadLatency disabled
diskWriteLatency disabled
diskRate enabled
...

Every configuration line is a module. Although in the example above, all the modules are created with default values. You're able to configure the following values: 'Name', 'description' and 'limits' for the 'warning' and 'critical' states. An example of this configuration type would be like the one below:

...
#VM Modules
VM
diskReadLatency disabled
diskWriteLatency disabled
diskRate name = Disk Rate; desc = Lec Rate/Esc disk; limits_warn = 5 10; limits_crit = 0 4
...

The available options for the module configuration are as follows:

• <module> disabled: The module will NOT be created
• <module> enabled: The module WILL be created (with values by default)
• <module> name = <name>; desc = <description>; limits_warn <lim_warn>; limits_crit <lim_crit>: The module will be created along with the given name and description. The module is going to define thresholds for the 'maximum' and 'minimum' as well as for 'Critical' and 'Warning' states.

Please keep in mind that it's very important to respect the structure of the configuration file lines, and above all seeing that the character ; is attached to the name and description of the module. These two lines are NOT EQUIVALENT (see spaces before character ;):

diskRate name = Disk Rate; desc = Lec Rate/Esc Disk; limits_warn = 5 10; limits_crit = 0 4
diskRate name = Disk Rate    ; desc = Lec Rate/Esc disk    ; limits_warn = 5 10; limits_crit    = 0    4

The modules are referenced by their short names or a simpler equivalent name to write it in the command line. The short and full names mapping tables are explained in the next section.

Let's analyze the configuration of the example above. We have configured the Disk Rate module which will be created along with the following values:

* Name: Disk Rate
* Description: Lec Rate/Esc disk
* Min Warning: 5
* Max Warning: 10
* Min Critical: 0
* Max Critical: 4

There are some modules which are dynamically generated, e.g. the modules on disks or network interfaces. For these metrics, the plugin creates a module for each discovered element. These modules bear special names in Pandora FMS, e.g.:

Disk Free [0]
Disk Free [1]
Disk Free [2]
...

Since the name has a dynamic part in these cases, you can use the macro '%s' which is going to be replaced by the variable part of the module name. An example of dynamic module configuration would be as follows:

diskfree name = Disk (% s) free space; desc = Free space for disk; limits_warn = 0 0; limits_crit = 0 0

In this case, the default module name is:

Free Disk [0]

And is going to be renamed as:

Disk (0) free space

From version 5.0 and above, you're able to set text strings for the limits of the 'Warning' and 'Critical' states of the modules. In such a case, the configuration would look like this:

powerState operation name = State; desc = VM operating state; limits_warn =. * suspended. *; limits_crit =. * poweredOff. *

You're also able to configure regular expressions to provide greater flexibility within the setting limits.

1.2.13.1.5 Custom Performance Metrics

In this section, we're going to show how to configure new modules for Performance Counters, Virtual Machines and ESX. To set a new performance module, you're required to use the following structure:

custom_performance type = mem; metric = swapinRate; module_type = generic_data; name = Swap In Rate; desc = Swap In Rate for host; limits_warn = 0 0; limits_crit = 0 0

The parameters to set are the following:

• Type: Type of metrics to monitor. The types of metrics are:
  • Cpu: CPU
  • Mem: Memory
  • Disk: Disk
  • Net: Network
  • Sys: System
• Metric: The metrics to monitor (explained later view metrics where available).
• Module_type: The Pandora FMS module type (e.g. 'generic_data').
• Name: The module's name.
• Desc: The description of the module.
• Limits_warn: The 'Warning' limits for the state.
• Limits_crit: The 'Critical' state-limits.

You're able to check the available metrics for each type in the Performance section of each entity. This view shows performance metrics which can be monitored by the VMware plug in which is located in the vCenter. The image below e.g. shows the Performance View for an ESX host.

To see a complete list of all the metrics sorted by type, please click on the Advanced button and then on the Char option button. A window which contains a list of all metric types and their respective metrics are going to be displayed like the ones on the picture below:

For each type of metric, a number of counters are going to appear. They represent the variables you're able to monitor with Pandora FMS. To monitor a variable, you have to use your internal Name. Furthermore, you have to make sure that the level of statistics which is configured in the vCenter shows the variable you seek by a comparison of the variable with the Collection Level of the metric.

If you e.g. like to see the CPU usage of an ESX host, you should search for CPU-type variables for an ESX and select Utilization. In this case, the line you have to add to the configuration file has to look like the one below:

 custom_performance type = cpu; metric = utilization; module_type = generic_data, name = CPU Utilization, desc = CPU Utilization for ESX; limits_warn = 0 0; limits_crit = 0 0

1.2.13.2 Monitoring of Several Data Centers by the same Agent

Each plugin module configured in the agent monitors a datacenter, so if you want to monitor several datacenter with the same Pandora FMS software agent, it will be necessary to take the following considerations.

• It is necessary to add a module_plugin for each datacenter to be monitored, for example

module_plugin /etc/pandora/plugins/vmware-plugin.pl /etc/pandora/plugins/vmware-plugin-datacenter1.conf
module_plugin /etc/pandora/plugins/vmware-plugin.pl /etc/pandora/plugins/vmware-plugin-datacenter2.conf

• In each configuration file it will be necessary to change the parameters: logfile, entities_list, event_pointer_file. The configuration files would look like this:

vmware-plugin-datacenter1.conf

...
logfile /tmp/vmware_plugin_datacenter1.log
entities_list /tmp/vmware_entities_list_datacenter1.txt
event_pointer_file /tmp/vmware_events_pointer_datacenter1.txt
...

vmware-plugin-datacenter2.conf

...
logfile /tmp/vmware_plugin_datacenter2.log
entities_list /tmp/vmware_entities_list_datacenter2.txt
event_pointer_file /tmp/vmware_events_pointer_datacenter2.txt
...

• If you also want to copy events, you must have two more plugin modules with the respective configuration files and activate the flag event_mode. The configuration of the module_plugin would look like this:

module_plugin /etc/pandora/plugins/vmware-plugin.pl /etc/pandora/plugins/vmware-plugin-datacenter1.conf
module_plugin /etc/pandora/plugins/vmware-plugin.pl /etc/pandora/plugins/vmware-plugin-datacenter1-events.conf
module_plugin /etc/pandora/plugins/vmware-plugin.pl /etc/pandora/plugins/vmware-plugin-datacenter2.conf
module_plugin /etc/pandora/plugins/vmware-plugin.pl /etc/pandora/plugins/vmware-plugin-datacenter2-events.conf

1.2.13.3 Sharing the Monitoring Load between several Pandora FMS Servers

The options of the plugin configuration file easily allow you to distribute the monitoring load between several Pandora FMS Servers.

Just imagine that you've acquired a similar structure in your virtualization environment to the one shown below:

DC
|
|- Datastore_1
|- DataStore_2
|
|- ESX_1
      |- mv1.1
      |- mv1.2
      |- mv1.3

|- ESX_2
      |- mv2.1
      |- mv2.2
      |- mv2.3

You have two Pandora FMS servers to monitor all your devices in your environment. It's an easy way to split the load by monitoring the Data Center, Data Stores and ESX on the first server and all the virtual machines on the second. The configuration file for the Recon Script has to be like the one below:

Server 1:

Reject
all_vm

Server 2:

Reject
all_datacenter
all_datastore
all_esx

Server 1 is going to monitor everything except the virtual machines. Server 2 is only going to monitor the virtual machines.

Another option would be to split the monitoring by ESX server. In this case, the first Pandora FMS Server would monitor all the things related to the first ESX and the second would split everything related to the second ESX. The configuration files would have to look like the one below:

Server 1:

Datacenter
DC
Datastore
Datastore_1
ESX
ESX_1
VM
mv1.1
mv1.2
mv1.3

Server 1 ignores everything related to the second group of VMware entities. It's going to monitor the first part of the environment.

Server 2:

 Datastore
DataStore_2
ESX
ESX_2
VM
mv2.1
mv2.2
mv2.3

Server 2 ignores everything related to the first group of VMware entities plus the Data Center (because these entities are monitored by Server 1).

The feature to reject entities is very flexible and allows you to split the load by assigning a few entities to each Pandora FMS Agent.

1.2.13.4 Example of the Configuration File

1.2.13.4.1 File with all Modules disabled

The lines which start by a '#' character are comments.

#Datacenter Modules
Datacenter
ping disabled
check443 disabled

#Datastore Modules
Datastore
capacity disabled
freeSpace disabled
overallocation disabled
freeSpaceBytes disabled

#ESX Modules
ESX
bootTime disabled
cpuInfo disabled
memorySize disabled
overallCpuUsage disabled
overallMemoryUsage disabled
powerState disabled
sslThumbprint disabled
uptime disabled
vnicInfo disabled
hostAlive disabled
connectionState disabled
diskRead disabled
diskWrite disabled
diskReadLatency disabled
diskWriteLatency disabled
netReceived disabled
netTransmitted disabled
netPkgRx disabled
netPkgTx disabled
cpuUsagePercent disabled
memoryUsagePercent disabled
netUsage disabled
diskRate disabled
maxDiskLatency disabled
systemHealthInfo disabled

#VM Modules
VM
bootTime disabled
connectionState disabled
consumedOverheadMemory disabled 
cpuAllocation disabled
diskFree disabled
guestState disabled
host disabled
hostAlive disabled
hostMemoryUsage disabled
hostName disabled
ipAddress disabled
macAddress disabled
maxCpuUsage disabled
maxMemoryUsage disabled
memoryAllocation disabled
memoryOverhead disabled
overallCpuDemand disabled
overallCpuUsage disabled
powerState disabled
privateMemory disabled
sharedMemory disabled
toolsRunningStatus disabled
triggeredAlarmState disabled
virtualImagePath disabled
uptimeSeconds disabled
diskRead disabled
diskWrite disabled
diskReadLatency disabled
diskWriteLatency disabled
netReceived disabled
netTransmitted disabled
netPkgRx disabled
netPkgTx  disabled
cpuUsagePercent disabled
memoryUsagePercent disabled
netUsage disabled
diskRate disabled
maxDiskLatency disabled
heartbeat disabled
cpuReady disabled

1.2.13.5 Correspondence Table of Short Names

1.2.13.5.1 Data Center

Full name	Short name
Ping	ping
Check port 443	check443

1.2.13.5.2 Data Stores

Full name	Short name
Capacity	capacity
Free Space	freeSpace
Disk Overallocation	overallocation
Free Space Bytes	freeSpaceBytes

1.2.13.5.3 ESX

Full name	Short name
Boot Time	bootTime
CPU Info	cpuInfo
Memory Size	memorySize
Overall CPU Usage	overallCpuUsage
Overall Memory Usage	overallMemoryUsage
Power State	powerState
SSL Thumbprint	sslThumbprint
Uptime	uptime
VNIC Info	vnicInfo
Host Alive	hostAlive
Connection State	connectionState
Disk Read	diskRead
Disk Write	diskWrite
Disk Read Latency	diskReadLatency
Disk Write Latency	diskWriteLatency
Data Received	netReceived
Data Transmitted	netTransmitted
Packages Received	netPkgRx
Packages Transmitted	netPkgTx
CPU Usage	cpuUsagePercent
Memory Usage	memoryUsagePercent
Net Usage	netUsage
Disk Rate	diskRate
Max. Disk Latency	maxDiskLatency
HA Status	haStatus
Sensor*	systemHealthInfo

1.2.13.5.4 Virtual Machines

Full name	Short name
Boot Time	bootTime
Connection State	connectionState
Consumed Overhead Memory	consumedOverheadMemory
CPU Allocation	cpuAllocation
Disk Free	diskFree
Guest State	guestState
Host Info	host
Host Alive	hostAlive
Host Memory Usage	hostMemoryUsage
Host Name	hostName
IP Address	ipAddress
MAC Address	macAddress
Max. CPU Usage	maxCpuUsage
Max. Memory Usage	maxMemoryUsage
Memory Allocation	memoryAllocation
Memory Overhead	memoryOverhead
Overall CPU Demand	overallCpuDemand
Overall CPU Usage	overallCpuUsage
Power State	powerState
Private Memory	privateMemory
Shared Memory	sharedMemory
Tools Running Status	toolsRunningStatus
Trigger Alarm State	triggeredAlarmState
Uptime Seconds	uptimeSeconds
Virtual Image Path	virtualImagePath
Disk Read	diskRead
Disk Write	diskWrite
Disk Read Latency	diskReadLatency
Disk Write Latency	diskWriteLatency
Data Received	netReceived
Data Transmitted	netTransmitted
Packages Received	netPkgRx
Packages Transmitted	netPkgTx
CPU Usage	cpuUsagePercent
Memory Usage	memoryUsagePercent
Net Usage	netUsage
Disk Rate	diskRate
Max. Disk Latency	maxDiskLatency
HeartBeat	heartbeat
CPU Ready	cpuReady
Number of Snapshots	snapshotCounter
HA Status	haStatus

1.2.13.6 Table of Events

This list of events is going to help you to configure Alert Events under Pandora FMS. For a complete and updated reference of all possible events, you may check the VMware Documentation.

Event	Severity	Event type	Group
An account was created on host {host.name}	Informational	System	All
Account {account} was removed on host {host.name}.	Informational	System	All
An account was updated on host {host.name}.	Informational	System	All
The default password for the root user on the host {host.name} has not been changed.	Informational	System	All
Alarm '{alarm.name}' on {entity.name} triggered an action	Informational	System	All
Created alarm '{alarm.name}' on {entity.name}	Informational	System	All
Alarm '{alarm.name}' on {entity.name} sent email to {to}	Informational	System	All
Alarm '{alarm.name}' on {entity.name} cannot send email to {to}	Critical	System	All
Reconfigured alarm '{alarm.name}' on {entity.name}	Informational	System	All
Removed alarm '{alarm.name}' on {entity.name}	Informational	System	All
Alarm '{alarm.name}' on {entity.name} ran script {script}	Informational	System	All
Alarm '{alarm.name}' on {entity.name} did not complete script: {reason.msg}	Critical	System	All
Alarm '{alarm.name}': an SNMP trap for entity {entity.name} was sent	Informational	System	All
Alarm '{alarm.name}' on entity {entity.name} did not send SNMP trap: {reason.msg}	Critical	System	All
Alarm '{alarm.name}' on {entity.name} changed from {[email protected]} to {[email protected]}	Informational	System	All
All running virtual machines are licensed	Informational	System	All
User cannot logon since the user is already logged on	Informational	System	All
Cannot login {userName}@{ipAddress}	Critical	System	All
The operation performed on host {host.name} in {datacenter.name} was canceled	Informational	System	All
Changed ownership of file name {filename} from {oldOwner} to {newOwner} on {host.name} in {datacenter.name}.	Informational	System	All
Cannot change ownership of file name {filename} from {owner} to {attemptedOwner} on {host.name} in {datacenter.name}.	Critical	System	All
Checked cluster for compliance	Informational	System	All
Created cluster {computeResource.name} in {datacenter.name}	Informational	System	All
Removed cluster {computeResource.name} in datacenter {datacenter.name}	Informational	System	All
Insufficient capacity in cluster {computeResource.name} to satisfy resource configuration in {datacenter.name}	Critical	System	All
Reconfigured cluster {computeResource.name} in datacenter {datacenter.name}	Informational	System	All
Configuration status on cluster {computeResource.name} changed from {[email protected]} to {[email protected]} in {datacenter.name}	Informational	System	All
Created new custom field definition {name}	Informational	System	All
Removed field definition {name}	Informational	System	All
Renamed field definition from {name} to {newName}	Informational	System	All
Changed custom field {name} on {entity.name} in {datacenter.name} to {value}	Informational	System	All
Cannot complete customization of VM {vm.name}. See customization log at {logLocation} on the guest OS for details.	Informational	System	All
An error occurred while setting up Linux identity. See log file '{logLocation}' on guest OS for details.	Critical	System	All
An error occurred while setting up network properties of the guest OS. See the log file {logLocation} in the guest OS for details.	Critical	System	All
Started customization of VM {vm.name}. Customization log located at {logLocation} in the guest OS.	Informational	System	All
Customization of VM {vm.name} succeeded. Customization log located at {logLocation} in the guest OS.	Informational	System	All
The version of Sysprep {sysprepVersion} provided for customizing VM {vm.name} does not match the version of guest OS {systemVersion}. See the log file {logLocation} in the guest OS for more information.	Critical	System	All
An error occurred while customizing VM {vm.name}. For details reference the log file {logLocation} in the guest OS.	Critical	System	All
dvPort group {net.name} in {datacenter.name} was added to switch {dvs.name}.	Informational	System	All
dvPort group {net.name} in {datacenter.name} was deleted.	Informational	System	All
	Informational	System	All
dvPort group {net.name} in {datacenter.name} was reconfigured.	Informational	System	All
dvPort group {oldName} in {datacenter.name} was renamed to {newName}	Informational	System	All
HA admission control disabled on cluster {computeResource.name} in {datacenter.name}	Informational	System	All
HA admission control enabled on cluster {computeResource.name} in {datacenter.name}	Informational	System	All
Re-established contact with a primary host in this HA cluster	Informational	System	All
Unable to contact a primary HA agent in cluster {computeResource.name} in {datacenter.name}	Critical	System	All
All hosts in the HA cluster {computeResource.name} in {datacenter.name} were isolated from the network. Check the network configuration for proper network redundancy in the management network.	Critical	System	All
HA disabled on cluster {computeResource.name} in {datacenter.name}	Informational	System	All
HA enabled on cluster {computeResource.name} in {datacenter.name}	Informational	System	All
A possible host failure has been detected by HA on {failedHost.name} in cluster {computeResource.name} in {datacenter.name}	Critical	System	All
Host {isolatedHost.name} has been isolated from cluster {computeResource.name} in {datacenter.name}	Warning	System	All
Created datacenter {datacenter.name} in folder {parent.name}	Informational	System	All
Renamed datacenter from {oldName} to {newName}	Informational	System	All
Datastore {datastore.name} increased in capacity from {oldCapacity} bytes to {newCapacity} bytes in {datacenter.name}	Informational	System	All
Removed unconfigured datastore {datastore.name}	Informational	System	All
Discovered datastore {datastore.name} on {host.name} in {datacenter.name}	Informational	System	All
Multiple datastores named {datastore} detected on host {host.name} in {datacenter.name}	Critical	System	All
<internal>	Informational	System	All
File or directory {sourceFile} copied from {sourceDatastore.name} to {datastore.name} as {targetFile}	Informational	System	All
File or directory {targetFile} deleted from {datastore.name}	Informational	System	All
File or directory {sourceFile} moved from {sourceDatastore.name} to {datastore.name} as {targetFile}	Informational	System	All
Reconfigured Storage I/O Control on datastore {datastore.name}	Informational	System	All
Configured datastore principal {datastorePrincipal} on host {host.name} in {datacenter.name}	Informational	System	All
Removed datastore {datastore.name} from {host.name} in {datacenter.name}	Informational	System	All
Renamed datastore from {oldName} to {newName} in {datacenter.name}	Informational	System	All
Renamed datastore from {oldName} to {newName} in {datacenter.name}	Informational	System	All
Disabled DRS on cluster {computeResource.name} in datacenter {datacenter.name}	Informational	System	All
Enabled DRS on {computeResource.name} with automation level {behavior} in {datacenter.name}	Informational	System	All
DRS put {host.name} into standby mode	Informational	System	All
DRS is putting {host.name} into standby mode	Informational	System	All
DRS cannot move {host.name} out of standby mode	Critical	System	All
DRS moved {host.name} out of standby mode	Informational	System	All
DRS is moving {host.name} out of standby mode	Informational	System	All
DRS invocation not completed	Critical	System	All
DRS has recovered from the failure	Informational	System	All
Unable to apply DRS resource settings on host {host.name} in {datacenter.name}. {reason.msg}. This can significantly reduce the effectiveness of DRS.	Critical	System	All
Resource configuration specification returns to synchronization from previous failure on host '{host.name}' in {datacenter.name}	Informational	System	All
{vm.name} on {host.name} in {datacenter.name} is now compliant with DRS VM-Host affinity rules	Informational	System	All
{vm.name} on {host.name} in {datacenter.name} is violating a DRS VM-Host affinity rule	Informational	System	All
DRS migrated {vm.name} from {sourceHost.name} to {host.name} in cluster {computeResource.name} in {datacenter.name}	Informational	System	All
DRS powered On {vm.name} on {host.name} in {datacenter.name}	Informational	System	All
Virtual machine {macAddress} on host {host.name} has a duplicate IP {duplicateIP}	Informational	System	All
A vNetwork Distributed Switch {dvs.name} was created in {datacenter.name}.	Informational	System	All
vNetwork Distributed Switch {dvs.name} in {datacenter.name} was deleted.	Informational	System	All
vNetwork Distributed Switch event	Informational	System	All
The vNetwork Distributed Switch {dvs.name} configuration on the host was synchronized with that of the vCenter Server.	Informational	System	All
The host {hostJoined.name} joined the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
The host {hostLeft.name} left the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
The host {hostMember.name} changed status on the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
The vNetwork Distributed Switch {dvs.name} configuration on the host differed from that of the vCenter Server.	Warning	System	All
vNetwork Distributed Switch {srcDvs.name} was merged into {dstDvs.name} in {datacenter.name}.	Informational	System	All
dvPort {portKey} was blocked in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
The port {portKey} was connected in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}	Informational	System	All
New ports were created in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
Deleted ports in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
The dvPort {portKey} was disconnected in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
dvPort {portKey} entered passthrough mode in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
dvPort {portKey} exited passthrough mode in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
dvPort {portKey} was moved into the dvPort group {portgroupName} in {datacenter.name}.	Informational	System	All
dvPort {portKey} was moved out of the dvPort group {portgroupName} in {datacenter.name}.	Informational	System	All
The port {portKey} link was down in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}	Informational	System	All
The port {portKey} link was up in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}	Informational	System	All
Reconfigured ports in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
dvPort {portKey} was unblocked in the vNetwork Distributed Switch {dvs.name} in {datacenter.name}.	Informational	System	All
The vNetwork Distributed Switch {dvs.name} in {datacenter.name} was reconfigured.	Informational	System	All
The vNetwork Distributed Switch {oldName} in {datacenter.name} was renamed to {newName}.	Informational	System	All
An upgrade for the vNetwork Distributed Switch {dvs.name} in datacenter {datacenter.name} is available.	Informational	System	All
An upgrade for the vNetwork Distributed Switch {dvs.name} in datacenter {datacenter.name} is in progress.	Informational	System	All
Cannot complete an upgrade for the vNetwork Distributed Switch {dvs.name} in datacenter {datacenter.name}	Informational	System	All
vNetwork Distributed Switch {dvs.name} in datacenter {datacenter.name} was upgraded.	Informational	System	All
Host {host.name} in {datacenter.name} has entered maintenance mode	Informational	System	All
The host {host.name} is in standby mode	Informational	System	All
Host {host.name} in {datacenter.name} has started to enter maintenance mode	Informational	System	All
The host {host.name} is entering standby mode	Informational	System	All
{message}	Critical	System	All
Host {host.name} in {datacenter.name} has exited maintenance mode	Informational	System	All
The host {host.name} could not exit standby mode	Critical	System	All
The host {host.name} is no longer in standby mode	Informational	System	All
The host {host.name} is exiting standby mode	Informational	System	All
Sufficient resources are available to satisfy HA failover level in cluster {computeResource.name} in {datacenter.name}	Informational	System	All
General event: {message}	Informational	System	All
Error detected on {host.name} in {datacenter.name}: {message}	Critical	System	All
Issue detected on {host.name} in {datacenter.name}: {message}	Informational	System	All
Issue detected on {host.name} in {datacenter.name}: {message}	Warning	System	All
User logged event: {message}	Informational	System	All
Error detected for {vm.name} on {host.name} in {datacenter.name}: {message}	Critical	System	All
Issue detected for {vm.name} on {host.name} in {datacenter.name}: {message}	Informational	System	All
Issue detected for {vm.name} on {host.name} in {datacenter.name}: {message}	Warning	System	All
The vNetwork Distributed Switch corresponding to the proxy switches {switchUuid} on the host {host.name} does not exist in vCenter Server or does not contain this host.	Informational	System	All
A ghost proxy switch {switchUuid} on the host {host.name} was resolved.	Informational	System	All
The message changed: {message}	Informational	System	All
{componentName} status changed from {oldStatus} to {newStatus}	Informational	System	All
Cannot add host {hostname} to datacenter {datacenter.name}	Critical	System	All
Added host {host.name} to datacenter {datacenter.name}	Informational	System	All
Administrator access to the host {host.name} is disabled	Warning	System	All
Administrator access to the host {host.name} has been restored	Warning	System	All
Cannot connect {host.name} in {datacenter.name}: cannot configure management account	Critical	System	All
Cannot connect {host.name} in {datacenter.name}: already managed by {serverName}	Critical	System	All
Cannot connect host {host.name} in {datacenter.name} : server agent is not responding	Critical	System	All
Cannot connect {host.name} in {datacenter.name}: incorrect user name or password	Critical	System	All
Cannot connect {host.name} in {datacenter.name}: incompatible version	Critical	System	All
Cannot connect host {host.name} in {datacenter.name}. Did not install or upgrade vCenter agent service.	Critical	System	All
Cannot connect {host.name} in {datacenter.name}: error connecting to host	Critical	System	All
Cannot connect {host.name} in {datacenter.name}: network error	Critical	System	All
Cannot connect host {host.name} in {datacenter.name}: account has insufficient privileges	Critical	System	All
Cannot connect host {host.name} in {datacenter.name}	Critical	System	All
Cannot connect {host.name} in {datacenter.name}: not enough CPU licenses	Critical	System	All
Cannot connect {host.name} in {datacenter.name}: incorrect host name	Critical	System	All
Cannot connect {host.name} in {datacenter.name}: time-out waiting for host response	Critical	System	All
Host {host.name} checked for compliance.	Informational	System	All
Host {host.name} is in compliance with the attached profile	Informational	System	All
Host configuration changes applied.	Informational	System	All
Connected to {host.name} in {datacenter.name}	Informational	System	All
Host {host.name} in {datacenter.name} is not responding	Critical	System	All
dvPort connected to host {host.name} in {datacenter.name} changed status	Informational	System	All
HA agent disabled on {host.name} in cluster {computeResource.name} in {datacenter.name}	Informational	System	All
HA is being disabled on {host.name} in cluster {computeResource.name} in datacenter {datacenter.name}	Informational	System	All
HA agent enabled on {host.name} in cluster {computeResource.name} in {datacenter.name}	Informational	System	All
Enabling HA agent on {host.name} in cluster {computeResource.name} in {datacenter.name}	Warning	System	All
HA agent on {host.name} in cluster {computeResource.name} in {datacenter.name} has an error {message}: {[email protected]}	Critical	System	All
HA agent on host {host.name} in cluster {computeResource.name} in {datacenter.name} is configured correctly	Informational	System	All
Disconnected from {host.name} in {datacenter.name}. Reason: {[email protected]}	Informational	System	All
Cannot restore some administrator permissions to the host {host.name}	Critical	System	All
Host {host.name} has the following extra networks not used by other hosts for HA communication:{ips}. Consider using HA advanced option das.allowNetwork to control network usage	Critical	System	All
Cannot complete command 'hostname -s' on host {host.name} or returned incorrect name format	Critical	System	All
Maximum ({capacity}) number of hosts allowed for this edition of vCenter Server has been reached	Critical	System	All
The virtual machine inventory file on host {host.name} is damaged or unreadable.	Informational	System	All
IP address of the host {host.name} changed from {oldIP} to {newIP}	Informational	System	All
Configuration of host IP address is inconsistent on host {host.name}: address resolved to {ipAddress} and {ipAddress2}	Critical	System	All
Cannot resolve IP address to short name on host {host.name}	Critical	System	All
Host {host.name} could not reach isolation address: {isolationIp}	Critical	System	All
A host license for {host.name} has expired	Critical	System	All
Host {host.name} does not have the following networks used by other hosts for HA communication:{ips}. Consider using HA advanced option das.allowNetwork to control network usage	Critical	System	All
Host monitoring state in {computeResource.name} in {datacenter.name} changed to {[email protected]}	Informational	System	All
Host {host.name} currently has no available networks for HA Communication. The following networks are currently used by HA: {ips}	Critical	System	All
Host {host.name} has no port groups enabled for HA communication.	Critical	System	All
Host {host.name} currently has no management network redundancy	Critical	System	All
Host {host.name} is not in compliance with the attached profile	Critical	System	All
Host {host.name} is not a cluster member in {datacenter.name}	Critical	System	All
Insufficient capacity in host {computeResource.name} to satisfy resource configuration in {datacenter.name}	Critical	System	All
Primary agent {primaryAgent} was not specified as a short name to host {host.name}	Critical	System	All
Profile is applied on the host {host.name}	Informational	System	All
Cannot reconnect to {host.name} in {datacenter.name}	Critical	System	All
Removed host {host.name} in {datacenter.name}	Informational	System	All
Host names {shortName} and {shortName2} both resolved to the same IP address. Check the host's network configuration and DNS entries	Critical	System	All
Cannot resolve short name {shortName} to IP address on host {host.name}	Critical	System	All
Shut down of {host.name} in {datacenter.name}: {reason}	Informational	System	All
Configuration status on host {computeResource.name} changed from {[email protected]} to {[email protected]} in {datacenter.name}	Informational	System	All
Cannot synchronize host {host.name}. {reason.msg}	Critical	System	All
Cannot install or upgrade vCenter agent service on {host.name} in {datacenter.name}	Critical	System	All
The userworld swap is not enabled on the host {host.name}	Warning	System	All
Host {host.name} vNIC {vnic.vnic} was reconfigured to use dvPort {vnic.port.portKey} with port level configuration, which might be different from the dvPort group.	Informational	System	All
WWNs are changed for {host.name}	Warning	System	All
The WWN ({wwn}) of {host.name} conflicts with the currently registered WWN	Critical	System	All
Host {host.name} did not provide the information needed to acquire the correct set of licenses	Critical	System	All
{message}	Informational	System	All
Insufficient resources to satisfy HA failover level on cluster {computeResource.name} in {datacenter.name}	Critical	System	All
The license edition '{feature}' is invalid	Critical	System	All
License {feature.featureName} has expired	Critical	System	All
License inventory is not compliant. Licenses are overused	Critical	System	All
Unable to acquire licenses due to a restriction in the option file on the license server.	Critical	System	All
License server {licenseServer} is available	Informational	System	All
License server {licenseServer} is unavailable	Critical	System	All
Created local datastore {datastore.name} on {host.name} in {datacenter.name}	Informational	System	All
The Local Tech Support Mode for the host {host.name} has been enabled	Informational	System	All
Datastore {datastore} which is configured to back the locker does not exist	Warning	System	All
Locker was reconfigured from {oldDatastore} to {newDatastore} datastore	Informational	System	All
Unable to migrate {vm.name} from {host.name} in {datacenter.name}: {fault.msg}	Critical	System	All
Unable to migrate {vm.name} from {host.name} to {dstHost.name} in {datacenter.name}: {fault.msg}	Critical	System	All
Migration of {vm.name} from {host.name} to {dstHost.name} in {datacenter.name}: {fault.msg}	Warning	System	All
Cannot migrate {vm.name} from {host.name} to {dstHost.name} and resource pool {dstPool.name} in {datacenter.name}: {fault.msg}	Critical	System	All
Migration of {vm.name} from {host.name} to {dstHost.name} and resource pool {dstPool.name} in {datacenter.name}: {fault.msg}	Warning	System	All
Migration of {vm.name} from {host.name} in {datacenter.name}: {fault.msg}	Warning	System	All
Created NAS datastore {datastore.name} on {host.name} in {datacenter.name}	Informational	System	All
Cannot login user {userName}@{ipAddress}: no permission	Critical	System	All
No datastores have been configured on the host {host.name}	Informational	System	All
A required license {feature.featureName} is not reserved	Critical	System	All
Unable to automatically migrate {vm.name} from {host.name}	Informational	System	All
Non-VI workload detected on datastore {datastore.name}	Critical	System	All
Not enough resources to failover {vm.name} in {computeResource.name} in {datacenter.name}	Informational	System	All
The vNetwork Distributed Switch configuration on some hosts differed from that of the vCenter Server.	Warning	System	All
Permission created for {principal} on {entity.name}, role is {role.name}, propagation is {[email protected]}	Informational	System	All
Permission rule removed for {principal} on {entity.name}	Informational	System	All
Permission changed for {principal} on {entity.name}, role is {role.name}, propagation is {[email protected]}	Informational	System	All
Profile {profile.name} attached.	Informational	System	All
Profile {profile.name} was changed.	Informational	System	All
Profile is created.	Informational	System	All
Profile {profile.name} detached.	Informational	System	All
Profile {profile.name} reference host changed.	Informational	System	All
Profile was removed.	Informational	System	All
Remote Tech Support Mode (SSH) for the host {host.name} has been enabled	Informational	System	All
Created resource pool {resourcePool.name} in compute-resource {computeResource.name} in {datacenter.name}	Informational	System	All
Removed resource pool {resourcePool.name} on {computeResource.name} in {datacenter.name}	Informational	System	All
Moved resource pool {resourcePool.name} from {oldParent.name} to {newParent.name} on {computeResource.name} in {datacenter.name}	Informational	System	All
Updated configuration for {resourcePool.name} in compute-resource {computeResource.name} in {datacenter.name}	Informational	System	All
Resource usage exceeds configuration for resource pool {resourcePool.name} in compute-resource {computeResource.name} in {datacenter.name}	Critical	System	All
New role {role.name} created	Informational	System	All
Role {role.name} removed	Informational	System	All
Modifed role {role.name}	Informational	System	All
Task {scheduledTask.name} on {entity.name} in {datacenter.name} completed successfully	Informational	System	All
Created task {scheduledTask.name} on {entity.name} in {datacenter.name}	Informational	System	All
Task {scheduledTask.name} on {entity.name} in {datacenter.name} sent email to {to}	Informational	System	All
Task {scheduledTask.name} on {entity.name} in {datacenter.name} cannot send email to {to}: {reason.msg}	Critical	System	All
Task {scheduledTask.name} on {entity.name} in {datacenter.name} cannot be completed: {reason.msg}	Critical	System	All
Reconfigured task {scheduledTask.name} on {entity.name} in {datacenter.name}	Informational	System	All
Removed task {scheduledTask.name} on {entity.name} in {datacenter.name}	Informational	System	All
Running task {scheduledTask.name} on {entity.name} in {datacenter.name}	Informational	System	All
A vCenter Server license has expired	Critical	System	All
vCenter started	Informational	System	All
A session for user '{terminatedUsername}' has stopped	Informational	System	All
Task: {info.descriptionId}	Informational	System	All
Task: {info.descriptionId} time-out	Informational	System	All
Upgrading template {legacyTemplate}	Informational	System	All
Cannot upgrade template {legacyTemplate} due to: {reason.msg}	Informational	System	All
Template {legacyTemplate} upgrade completed	Informational	System	All
The operation performed on {host.name} in {datacenter.name} timed out	Warning	System	All
There are {unlicensed} unlicensed virtual machines on host {host} - there are only {available} licenses available	Informational	System	All
{unlicensed} unlicensed virtual machines found on host {host}	Informational	System	All
The agent on host {host.name} is updated and will soon restart	Informational	System	All
User {userLogin} was added to group {group}	Informational	System	All
User {userName}@{ipAddress} logged in	Informational	System	All
User {userName} logged out	Informational	System	All
Password was changed for account {userLogin} on host {host.name}	Informational	System	All
User {userLogin} removed from group {group}	Informational	System	All
{message}	Informational	System	All
Created VMFS datastore {datastore.name} on {host.name} in {datacenter.name}	Informational	System	All
Expanded VMFS datastore {datastore.name} on {host.name} in {datacenter.name}	Informational	System	All
Extended VMFS datastore {datastore.name} on {host.name} in {datacenter.name}	Informational	System	All
A vMotion license for {host.name} has expired	Critical	System	All
Cannot uninstall vCenter agent from {host.name} in {datacenter.name}. {[email protected]}	Critical	System	All
vCenter agent has been uninstalled from {host.name} in {datacenter.name}	Informational	System	All
Cannot upgrade vCenter agent on {host.name} in {datacenter.name}. {[email protected]}	Critical	System	All
vCenter agent has been upgraded on {host.name} in {datacenter.name}	Informational	System	All
VIM account password was changed on host {host.name}	Informational	System	All
Remote console to {vm.name} on {host.name} in {datacenter.name} has been opened	Informational	System	All
A ticket for {vm.name} of type {ticketType} on {host.name} in {datacenter.name} has been acquired	Informational	System	All
Invalid name for {vm.name} on {host.name} in {datacenter.name}. Renamed from {oldName} to {newName}	Informational	System	All
Cloning {vm.name} on host {host.name} in {datacenter.name} to {destName} on host {destHost.name}	Informational	System	All
Cloning {vm.name} on host {host.name} in {datacenter.name} to {destName} on host {destHost.name}	Informational	System	All
Creating {vm.name} on host {host.name} in {datacenter.name}	Informational	System	All
Deploying {vm.name} on host {host.name} in {datacenter.name} from template {srcTemplate.name}	Informational	System	All
Migrating {vm.name} from {host.name} to {destHost.name} in {datacenter.name}	Informational	System	All
Relocating {vm.name} from {host.name} to {destHost.name} in {datacenter.name}	Informational	System	All
Relocating {vm.name} in {datacenter.name} from {host.name} to {destHost.name}	Informational	System	All
Cannot clone {vm.name}: {reason.msg}	Critical	System	All
Clone of {sourceVm.name} completed	Informational	System	All
Configuration file for {vm.name} on {host.name} in {datacenter.name} cannot be found	Informational	System	All
Virtual machine {vm.name} is connected	Informational	System	All
Created virtual machine {vm.name} on {host.name} in {datacenter.name}	Informational	System	All
dvPort connected to VM {vm.name} on {host.name} in {datacenter.name} changed status	Informational	System	All
{vm.name} on {host.name} in cluster {computeResource.name} in {datacenter.name} reset by HA. Reason: {[email protected]}	Informational	System	All
{vm.name} on {host.name} in cluster {computeResource.name} in {datacenter.name} reset by HA. Reason: {[email protected]}. A screenshot is saved at {screenshotFilePath}.	Informational	System	All
Cannot reset {vm.name} on {host.name} in cluster {computeResource.name} in {datacenter.name}	Warning	System	All
Unable to update HA agents given the state of {vm.name}	Critical	System	All
HA agents have been updated with the current state of the virtual machine	Informational	System	All
Disconnecting all hosts as the date of virtual machine {vm.name} has been rolled back	Critical	System	All
Cannot deploy template: {reason.msg}	Critical	System	All
Template {srcTemplate.name} deployed on host {host.name}	Informational	System	All
{vm.name} on host {host.name} in {datacenter.name} is disconnected	Informational	System	All
Discovered {vm.name} on {host.name} in {datacenter.name}	Informational	System	All
Cannot create virtual disk {disk}	Critical	System	All
Migrating {vm.name} off host {host.name} in {datacenter.name}	Informational	System	All
End a recording session on {vm.name}	Informational	System	All
End a replay session on {vm.name}	Informational	System	All
Cannot migrate {vm.name} from {host.name} to {destHost.name} in {datacenter.name}	Critical	System	All
Cannot complete relayout {vm.name} on {host.name} in {datacenter.name}: {reason.msg}	Critical	System	All
Cannot complete relayout for virtual machine {vm.name} which has disks on a VMFS2 volume.	Critical	System	All
vCenter cannot start the Secondary VM {vm.name}. Reason: {[email protected]}	Critical	System	All
Cannot power Off {vm.name} on {host.name} in {datacenter.name}: {reason.msg}	Critical	System	All
Cannot power On {vm.name} on {host.name} in {datacenter.name}. {reason.msg}	Critical	System	All
Cannot reboot the guest OS for {vm.name} on {host.name} in {datacenter.name}. {reason.msg}	Critical	System	All
Cannot suspend {vm.name} on {host.name} in {datacenter.name}: {reason.msg}	Critical	System	All
{vm.name} cannot shut down the guest OS on {host.name} in {datacenter.name}: {reason.msg}	Critical	System	All
{vm.name} cannot standby the guest OS on {host.name} in {datacenter.name}: {reason.msg}	Critical	System	All
Cannot suspend {vm.name} on {host.name} in {datacenter.name}: {reason.msg}	Critical	System	All
vCenter cannot update the Secondary VM {vm.name} configuration	Critical	System	All
Failover unsuccessful for {vm.name} on {host.name} in cluster {computeResource.name} in {datacenter.name}. Reason: {reason.msg}	Warning	System	All
Fault Tolerance state on {vm.name} changed from {[email protected]} to {[email protected]}	Informational	System	All
Fault Tolerance protection has been turned off for {vm.name}	Informational	System	All
The Fault Tolerance VM ({vm.name}) has been terminated. {[email protected]}	Informational	System	All
Guest OS reboot for {vm.name} on {host.name} in {datacenter.name}	Informational	System	All
Guest OS shut down for {vm.name} on {host.name} in {datacenter.name}	Informational	System	All
Guest OS standby for {vm.name} on {host.name} in {datacenter.name}	Informational	System	All
VM monitoring state in {computeResource.name} in {datacenter.name} changed to {[email protected]}	Informational	System	All
Assign a new instance UUID ({instanceUuid}) to {vm.name}	Informational	System	All
The instance UUID of {vm.name} has been changed from ({oldInstanceUuid}) to ({newInstanceUuid})	Informational	System	All
The instance UUID ({instanceUuid}) of {vm.name} conflicts with the instance UUID assigned to {conflictedVm.name}	Critical	System	All
New MAC address ({mac}) assigned to adapter {adapter} for {vm.name}	Informational	System	All
Changed MAC address from {oldMac} to {newMac} for adapter {adapter} for {vm.name}	Warning	System	All
The MAC address ({mac}) of {vm.name} conflicts with MAC assigned to {conflictedVm.name}	Critical	System	All
Reached maximum Secondary VM (with FT turned On) restart count for {vm.name} on {host.name} in cluster {computeResource.name} in {datacenter.name}.	Warning	System	All
Reached maximum VM restart count for {vm.name} on {host.name} in cluster {computeResource.name} in {datacenter.name}.	Warning	System	All
Error message on {vm.name} on {host.name} in {datacenter.name}: {message}	Critical	System	All
Message on {vm.name} on {host.name} in {datacenter.name}: {message}	Informational	System	All
Warning message on {vm.name} on {host.name} in {datacenter.name}: {message}	Warning	System	All
Migration of virtual machine {vm.name} from {sourceHost.name} to {host.name} completed	Informational	System	All
No compatible host for the Secondary VM {vm.name}	Critical	System	All
Not all networks for {vm.name} are accessible by {destHost.name}	Warning	System	All
{vm.name} does not exist on {host.name} in {datacenter.name}	Warning	System	All
{vm.name} was powered Off on the isolated host {isolatedHost.name} in cluster {computeResource.name} in {datacenter.name}	Informational	System	All
{vm.name} on {host.name} in {datacenter.name} is powered off	Informational	System	All
{vm.name} on {host.name} in {datacenter.name} is powered on	Informational	System	All
Virtual machine {vm.name} powered On with vNICs connected to dvPorts that have a port level configuration, which might be different from the dvPort group configuration.	Informational	System	All
VM ({vm.name}) failed over to {host.name}. {[email protected]}	Critical	System	All
Reconfigured {vm.name} on {host.name} in {datacenter.name}	Informational	System	All
Registered {vm.name} on {host.name} in {datacenter.name}	Informational	System	All
Relayout of {vm.name} on {host.name} in {datacenter.name} completed	Informational	System	All
{vm.name} on {host.name} in {datacenter.name} is in the correct format and relayout is not necessary	Informational	System	All
{vm.name} on {host.name} reloaded from new configuration {configPath}.	Informational	System	All
{vm.name} on {host.name} could not be reloaded from {configPath}.	Critical	System	All
Cannot relocate virtual machine '{vm.name}' in {datacenter.name}	Critical	System	All
Completed the relocation of the virtual machine	Informational	System	All
Remote console connected to {vm.name} on host {host.name}	Informational	System	All
Remote console disconnected from {vm.name} on host {host.name}	Informational	System	All
Removed {vm.name} on {host.name} from {datacenter.name}	Informational	System	All
Renamed {vm.name} from {oldName} to {newName} in {datacenter.name}	Warning	System	All
{vm.name} on {host.name} in {datacenter.name} is reset	Informational	System	All
Moved {vm.name} from resource pool {oldParent.name} to {newParent.name} in {datacenter.name}	Informational	System	All
Changed resource allocation for {vm.name}	Informational	System	All
Virtual machine {vm.name} was restarted on {host.name} since {sourceHost.name} failed	Informational	System	All
{vm.name} on {host.name} in {datacenter.name} is resumed	Informational	System	All
A Secondary VM has been added for {vm.name}	Informational	System	All
vCenter disabled Fault Tolerance on VM '{vm.name}' because the Secondary VM could not be powered On.	Critical	System	All
Disabled Secondary VM for {vm.name}	Informational	System	All
Enabled Secondary VM for {vm.name}	Informational	System	All
Started Secondary VM for {vm.name}	Informational	System	All
{vm.name} was shut down on the isolated host {isolatedHost.name} in cluster {computeResource.name} in {datacenter.name}: {[email protected]}	Informational	System	All
Start a recording session on {vm.name}	Informational	System	All
Start a replay session on {vm.name}	Informational	System	All
{vm.name} on host {host.name} in {datacenter.name} is starting	Informational	System	All
Starting Secondary VM for {vm.name}	Informational	System	All
The static MAC address ({mac}) of {vm.name} conflicts with MAC assigned to {conflictedVm.name}	Critical	System	All
{vm.name} on {host.name} in {datacenter.name} is stopping	Informational	System	All
{vm.name} on {host.name} in {datacenter.name} is suspended	Informational	System	All
{vm.name} on {host.name} in {datacenter.name} is being suspended	Informational	System	All
Starting the Secondary VM {vm.name} timed out within {timeout} ms	Critical	System	All
Unsupported guest OS {guestId} for {vm.name} on {host.name} in {datacenter.name}	Warning	System	All
Virtual hardware upgraded to version {version}	Informational	System	All
Cannot upgrade virtual hardware	Critical	System	All
Upgrading virtual hardware on {vm.name} in {datacenter.name} to version {version}	Informational	System	All
Assigned new BIOS UUID ({uuid}) to {vm.name} on {host.name} in {datacenter.name}	Informational	System	All
Changed BIOS UUID from {oldUuid} to {newUuid} for {vm.name} on {host.name} in {datacenter.name}	Warning	System	All
BIOS ID ({uuid}) of {vm.name} conflicts with that of {conflictedVm.name}	Critical	System	All
New WWNs assigned to {vm.name}	Informational	System	All
WWNs are changed for {vm.name}	Warning	System	All
The WWN ({wwn}) of {vm.name} conflicts with the currently registered WWN	Critical	System	All
{message}	Warning	System	All
Booting from iSCSI failed with an error. See the VMware Knowledge Base for information on configuring iBFT networking.	Warning	System	All
com.vmware.license.AddLicenseEvent|License {licenseKey} added to VirtualCenter	Informational	System	All
com.vmware.license.AssignLicenseEvent|License {licenseKey} assigned to asset {entityName} with id {entityId}	Informational	System	All
com.vmware.license.DLFDownloadFailedEvent|Failed to download license information from the host {hostname} due to {[email protected]ownloadFailedReason}	Warning	System	All
com.vmware.license.LicenseAssignFailedEvent|License assignment on the host fails. Reasons: {[email protected]}.	Informational	System	All
com.vmware.license.LicenseExpiryEvent|Your host license will expire in {remainingDays} days. The host will be disconnected from VC when its license expires.	Warning	System	All
com.vmware.license.LicenseUserThresholdExceededEvent|Current license usage ({currentUsage} {costUnitText}) for {edition} exceeded the user-defined threshold ({threshold} {costUnitText})	Warning	System	All
com.vmware.license.RemoveLicenseEvent|License {licenseKey} removed from VirtualCenter	Informational	System	All
com.vmware.license.UnassignLicenseEvent|License unassigned from asset {entityName} with id {entityId}	Informational	System	All
com.vmware.vc.HA.ClusterFailoverActionCompletedEvent|HA completed a failover action in cluster {computeResource.name} in datacenter {datacenter.name}	Informational	System	All
com.vmware.vc.HA.ClusterFailoverActionInitiatedEvent|HA initiated a failover action in cluster {computeResource.name} in datacenter {datacenter.name}	Warning	System	All
com.vmware.vc.HA.DasAgentRunningEvent|HA Agent on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} is running	Informational	System	All
com.vmware.vc.HA.DasFailoverHostFailedEvent|HA failover host {host.name} in cluster {computeResource.name} in {datacenter.name} has failed	Critical	System	All
com.vmware.vc.HA.DasHostCompleteDatastoreFailureEvent|All shared datastores failed on the host {hostName} in cluster {computeResource.name} in {datacenter.name}	Critical	System	All
com.vmware.vc.HA.DasHostCompleteNetworkFailureEvent|All VM networks failed on the host {hostName} in cluster {computeResource.name} in {datacenter.name}	Critical	System	All
com.vmware.vc.HA.DasHostFailedEvent|A possible host failure has been detected by HA on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name}	Critical	System	All
com.vmware.vc.HA.DasHostMonitoringDisabledEvent|No virtual machine failover will occur until Host Monitoring is enabled in cluster {computeResource.name} in {datacenter.name}	Warning	System	All
com.vmware.vc.HA.DasTotalClusterFailureEvent|HA recovered from a total cluster failure in cluster {computeResource.name} in datacenter {datacenter.name}	Warning	System	All
com.vmware.vc.HA.HostDasAgentHealthyEvent|HA Agent on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} is healthy	Informational	System	All
com.vmware.vc.HA.HostDasErrorEvent|HA agent on {host.name} in cluster {computeResource.name} in {datacenter.name} has an error: {[email protected]}	Critical	System	All
com.vmware.vc.VCHealthStateChangedEvent|vCenter Service overall health changed from '{oldState}' to '{newState}'	Informational	System	All
com.vmware.vc.cim.CIMGroupHealthStateChanged|Health of [data.group] changed from [data.oldState] to [data.newState].	Informational	System	All
com.vmware.vc.datastore.UpdateVmFilesFailedEvent|Failed to update VM files on datastore {ds.name} using host {hostName}	Critical	System	All
com.vmware.vc.datastore.UpdatedVmFilesEvent|Updated VM files on datastore {ds.name} using host {hostName}	Informational	System	All
com.vmware.vc.datastore.UpdatingVmFilesEvent|Updating VM files on datastore {ds.name} using host {hostName}	Informational	System	All
com.vmware.vc.ft.VmAffectedByDasDisabledEvent|VMware HA has been disabled in cluster {computeResource.name} of datacenter {datacenter.name}. HA will not restart VM {vm.name} or its Secondary VM after a failure.	Warning	System	All
com.vmware.vc.npt.VmAdapterEnteredPassthroughEvent|Network passthrough is active on adapter {deviceLabel} of virtual machine {vm.name} on host {host.name} in {datacenter.name}	Informational	System	All
com.vmware.vc.npt.VmAdapterExitedPassthroughEvent|Network passthrough is inactive on adapter {deviceLabel} of virtual machine {vm.name} on host {host.name} in {datacenter.name}	Informational	System	All
com.vmware.vc.vcp.FtDisabledVmTreatAsNonFtEvent|HA VM Component Protection protects virtual machine {vm.name} on {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} as non-FT virtual machine because the FT state is disabled	Informational	System	All
com.vmware.vc.vcp.FtFailoverEvent|FT Primary VM {vm.name} on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} is going to fail over to Secondary VM due to component failure	Informational	System	All
com.vmware.vc.vcp.FtFailoverFailedEvent|FT virtual machine {vm.name} on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} failed to failover to secondary	Critical	System	All
com.vmware.vc.vcp.FtSecondaryRestartEvent|HA VM Component Protection is restarting FT secondary virtual machine {vm.name} on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} due to component failure	Informational	System	All
com.vmware.vc.vcp.FtSecondaryRestartFailedEvent|FT Secondary VM {vm.name} on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} failed to restart	Critical	System	All
com.vmware.vc.vcp.NeedSecondaryFtVmTreatAsNonFtEvent|HA VM Component Protection protects virtual machine {vm.name} on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} as non-FT virtual machine because it has been in the needSecondary state too long	Informational	System	All
com.vmware.vc.vcp.TestEndEvent|VM Component Protection test ends on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name}	Informational	System	All
com.vmware.vc.vcp.TestStartEvent|VM Component Protection test starts on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name}	Informational	System	All
com.vmware.vc.vcp.VcpNoActionEvent|HA VM Component Protection did not take action on virtual machine {vm.name} on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} due to the feature configuration setting	Informational	System	All
com.vmware.vc.vcp.VmDatastoreFailedEvent|Virtual machine {vm.name} on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} lost access to {datastore}	Critical	System	All
com.vmware.vc.vcp.VmNetworkFailedEvent|Virtual machine {vm.name} on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} lost access to {network}	Critical	System	All
com.vmware.vc.vcp.VmPowerOffHangEvent|HA VM Component Protection could not power off virtual machine {vm.name} on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} successfully after trying {numTimes} times and will keep trying	Critical	System	All
com.vmware.vc.vcp.VmRestartEvent|HA VM Component Protection is restarting virtual machine {vm.name} due to component failure on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name}	Informational	System	All
com.vmware.vc.vcp.VmRestartFailedEvent|Virtual machine {vm.name} affected by component failure on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} failed to restart	Critical	System	All
com.vmware.vc.vcp.VmWaitForCandidateHostEvent|HA VM Component Protection could not find a destination host for virtual machine {vm.name} on host {host.name} in cluster {computeResource.name} in datacenter {datacenter.name} after waiting {numSecWait} seconds and will keep trying	Critical	System	All
com.vmware.vc.vmam.AppMonitoringNotSupported|Application monitoring is not supported on {host.name} in cluster {computeResource.name} in {datacenter.name}	Warning	System	All
com.vmware.vc.vmam.VmAppHealthMonitoringStateChangedEvent|Application heartbeat status changed to {status} for {vm.name} on {host.name} in cluster {computeResource.name} in {datacenter.name}	Warning	System	All
com.vmware.vc.vmam.VmDasAppHeartbeatFailedEvent|Application heartbeat failed for {vm.name} on {host.name} in cluster {computeResource.name} in {datacenter.name}	Warning	System	All
esx.clear.net.connectivity.restored|Network connectivity restored on virtual switch {1}, portgroups: {2}. Physical NIC {3} is up.	Informational	System	All
esx.clear.net.dvport.connectivity.restored|Network connectivity restored on DVPorts: {1}. Physical NIC {2} is up.	Informational	System	All
esx.clear.net.dvport.redundancy.restored|Uplink redundancy restored on DVPorts: {1}. Physical NIC {2} is up.	Informational	System	All
esx.clear.net.redundancy.restored|Uplink redundancy restored on virtual switch {1}, portgroups: {2}. Physical NIC {3} is up.	Informational	System	All
esx.clear.net.vmnic.linkstate.up|Physical NIC {1} linkstate is up.	Informational	System	All
esx.clear.storage.connectivity.restored|Connectivity to storage device {1} (Datastores: {2}) restored. Path {3} is active again.	Informational	System	All
esx.clear.storage.redundancy.restored|Path redundancy to storage device {1} (Datastores: {2}) restored. Path {3} is active again.	Informational	System	All
esx.problem.apei.bert.memory.error.corrected|A corrected memory error occurred in last boot. The following details were reported. Physical Addr: {1}, Physical Addr Mask: {2}, Node: {3}, Card: {4}, Module: {5}, Bank: {6}, Device: {7}, Row: {8}, Column: {9} Error type: {10}	Critical	System	All
esx.problem.apei.bert.memory.error.fatal|A fatal memory error occurred in the last boot. The following details were reported. Physical Addr: {1}, Physical Addr Mask: {2}, Node: {3}, Card: {4}, Module: {5}, Bank: {6}, Device: {7}, Row: {8}, Column: {9} Error type: {10}	Critical	System	All
esx.problem.apei.bert.memory.error.recoverable|A recoverable memory error occurred in last boot. The following details were reported. Physical Addr: {1}, Physical Addr Mask: {2}, Node: {3}, Card: {4}, Module: {5}, Bank: {6}, Device: {7}, Row: {8}, Column: {9} Error type: {10}	Critical	System	All
esx.problem.apei.bert.pcie.error.corrected|A corrected PCIe error occurred in last boot. The following details were reported. Port Type: {1}, Device: {2}, Bus #: {3}, Function: {4}, Slot: {5}, Device Vendor: {6}, Version: {7}, Command Register: {8}, Status Register: {9}.	Critical	System	All
esx.problem.apei.bert.pcie.error.fatal|Platform encounterd a fatal PCIe error in last boot. The following details were reported. Port Type: {1}, Device: {2}, Bus #: {3}, Function: {4}, Slot: {5}, Device Vendor: {6}, Version: {7}, Command Register: {8}, Status Register: {9}.	Critical	System	All
esx.problem.apei.bert.pcie.error.recoverable|A recoverable PCIe error occurred in last boot. The following details were reported. Port Type: {1}, Device: {2}, Bus #: {3}, Function: {4}, Slot: {5}, Device Vendor: {6}, Version: {7}, Command Register: {8}, Status Register: {9}.	Critical	System	All
esx.problem.iorm.nonviworkload|An external I/O activity is detected on datastore {1}, this is an unsupported configuration. Consult the Resource Management Guide or follow the Ask VMware link for more information.	Informational	System	All
esx.problem.net.connectivity.lost|Lost network connectivity on virtual switch {1}. Physical NIC {2} is down. Affected portgroups:{3}.	Critical	System	All
esx.problem.net.dvport.connectivity.lost|Lost network connectivity on DVPorts: {1}. Physical NIC {2} is down.	Critical	System	All
esx.problem.net.dvport.redundancy.degraded|Uplink redundancy degraded on DVPorts: {1}. Physical NIC {2} is down.	Warning	System	All
esx.problem.net.dvport.redundancy.lost|Lost uplink redundancy on DVPorts: {1}. Physical NIC {2} is down.	Warning	System	All
esx.problem.net.e1000.tso6.notsupported|Guest-initiated IPv6 TCP Segmentation Offload (TSO) packets ignored. Manually disable TSO inside the guest operating system in virtual machine {1}, or use a different virtual adapter.	Critical	System	All
esx.problem.net.migrate.bindtovmk|The ESX advanced configuration option /Migrate/Vmknic is set to an invalid vmknic: {1}. /Migrate/Vmknic specifies a vmknic that vMotion binds to for improved performance. Update the configuration option with a valid vmknic. Alternatively, if you do not want vMotion to bind to a specific vmknic, remove the invalid vmknic and leave the option blank.	Warning	System	All
esx.problem.net.proxyswitch.port.unavailable|Virtual NIC with hardware address {1} failed to connect to distributed virtual port {2} on switch {3}. There are no more ports available on the host proxy switch.	Warning	System	All
esx.problem.net.redundancy.degraded|Uplink redundancy degraded on virtual switch {1}. Physical NIC {2} is down. Affected portgroups:{3}.	Warning	System	All
esx.problem.net.redundancy.lost|Lost uplink redundancy on virtual switch {1}. Physical NIC {2} is down. Affected portgroups:{3}.	Warning	System	All
esx.problem.net.uplink.mtu.failed|VMkernel failed to set the MTU value {1} on the uplink {2}.	Warning	System	All
esx.problem.net.vmknic.ip.duplicate|A duplicate IP address was detected for {1} on the interface {2}. The current owner is {3}.	Warning	System	All
esx.problem.net.vmnic.linkstate.down|Physical NIC {1} linkstate is down.	Informational	System	All
esx.problem.net.vmnic.watchdog.reset|Uplink {1} has recovered from a transient failure due to watchdog timeout	Informational	System	All
esx.problem.scsi.device.limitreached|The maximum number of supported devices of {1} has been reached. A device from plugin {2} could not be created.	Critical	System	All
esx.problem.scsi.device.thinprov.atquota|Space utilization on thin-provisioned device {1} exceeded configured threshold. Affected datastores (if any): {2}.	Warning	System	All
esx.problem.scsi.scsipath.limitreached|The maximum number of supported paths of {1} has been reached. Path {2} could not be added.	Critical	System	All
esx.problem.storage.connectivity.devicepor|Frequent PowerOn Reset Unit Attentions are occurring on device {1}. This might indicate a storage problem. Affected datastores: {2}	Warning	System	All
esx.problem.storage.connectivity.lost|Lost connectivity to storage device {1}. Path {2} is down. Affected datastores: {3}.	Critical	System	All
esx.problem.storage.connectivity.pathpor|Frequent PowerOn Reset Unit Attentions are occurring on path {1}. This might indicate a storage problem. Affected device: {2}. Affected datastores: {3}	Warning	System	All
esx.problem.storage.connectivity.pathstatechanges|Frequent path state changes are occurring for path {1}. This might indicate a storage problem. Affected device: {2}. Affected datastores: {3}	Warning	System	All
esx.problem.storage.redundancy.degraded|Path redundancy to storage device {1} degraded. Path {2} is down. Affected datastores: {3}.	Warning	System	All
esx.problem.storage.redundancy.lost|Lost path redundancy to storage device {1}. Path {2} is down. Affected datastores: {3}.	Warning	System	All
esx.problem.vmfs.heartbeat.recovered|Successfully restored access to volume {1} ({2}) following connectivity issues.	Informational	System	All
esx.problem.vmfs.heartbeat.timedout|Lost access to volume {1} ({2}) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.	Informational	System	All
esx.problem.vmfs.heartbeat.unrecoverable|Lost connectivity to volume {1} ({2}) and subsequent recovery attempts have failed.	Critical	System	All
esx.problem.vmfs.journal.createfailed|No space for journal on volume {1} ({2}). Opening volume in read-only metadata mode with limited write support.	Critical	System	All
esx.problem.vmfs.lock.corruptondisk|At least one corrupt on-disk lock was detected on volume {1} ({2}). Other regions of the volume might be damaged too.	Critical	System	All
esx.problem.vmfs.nfs.mount.connect.failed|Failed to mount to the server {1} mount point {2}. {3}	Critical	System	All
esx.problem.vmfs.nfs.mount.limit.exceeded|Failed to mount to the server {1} mount point {2}. {3}	Critical	System	All
esx.problem.vmfs.nfs.server.disconnect|Lost connection to server {1} mount point {2} mounted as {3} ({4}).	Critical	System	All
esx.problem.vmfs.nfs.server.restored|Restored connection to server {1} mount point {2} mounted as {3} ({4}).	Informational	System	All
esx.problem.vmfs.resource.corruptondisk|At least one corrupt resource metadata region was detected on volume {1} ({2}). Other regions of the volume might be damaged too.	Critical	System	All
esx.problem.vmfs.volume.locked|Volume on device {1} locked, possibly because remote host {2} encountered an error during a volume operation and could not recover.	Critical	System	All
vim.event.LicenseDowngradedEvent|License downgrade: {licenseKey} removes the following features: {lostFeatures}	Warning	System	All
vprob.net.connectivity.lost|Lost network connectivity on virtual switch {1}. Physical NIC {2} is down. Affected portgroups:{3}.	Critical	System	All
vprob.net.e1000.tso6.notsupported|Guest-initiated IPv6 TCP Segmentation Offload (TSO) packets ignored. Manually disable TSO inside the guest operating system in virtual machine {1}, or use a different virtual adapter.	Critical	System	All
vprob.net.migrate.bindtovmk|The ESX advanced config option /Migrate/Vmknic is set to an invalid vmknic: {1}. /Migrate/Vmknic specifies a vmknic that vMotion binds to for improved performance. Please update the config option with a valid vmknic or, if you do not want vMotion to bind to a specific vmknic, remove the invalid vmknic and leave the option blank.	Warning	System	All
vprob.net.proxyswitch.port.unavailable|Virtual NIC with hardware address {1} failed to connect to distributed virtual port {2} on switch {3}. There are no more ports available on the host proxy switch.	Warning	System	All
vprob.net.redundancy.degraded|Uplink redundancy degraded on virtual switch {1}. Physical NIC {2} is down. {3} uplinks still up. Affected portgroups:{4}.	Warning	System	All
vprob.net.redundancy.lost|Lost uplink redundancy on virtual switch {1}. Physical NIC {2} is down. Affected portgroups:{3}.	Warning	System	All
vprob.scsi.device.thinprov.atquota|Space utilization on thin-provisioned device {1} exceeded configured threshold.	Warning	System	All
vprob.storage.connectivity.lost|Lost connectivity to storage device {1}. Path {2} is down. Affected datastores: {3}.	Critical	System	All
vprob.storage.redundancy.degraded|Path redundancy to storage device {1} degraded. Path {2} is down. {3} remaining active paths. Affected datastores: {4}.	Warning	System	All
vprob.storage.redundancy.lost|Lost path redundancy to storage device {1}. Path {2} is down. Affected datastores: {3}.	Warning	System	All
vprob.vmfs.heartbeat.recovered|Successfully restored access to volume {1} ({2}) following connectivity issues.	Informational	System	All
vprob.vmfs.heartbeat.timedout|Lost access to volume {1} ({2}) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.	Informational	System	All
vprob.vmfs.heartbeat.unrecoverable|Lost connectivity to volume {1} ({2}) and subsequent recovery attempts have failed.	Critical	System	All
vprob.vmfs.journal.createfailed|No space for journal on volume {1} ({2}). Opening volume in read-only metadata mode with limited write support.	Critical	System	All
vprob.vmfs.lock.corruptondisk|At least one corrupt on-disk lock was detected on volume {1} ({2}). Other regions of the volume may be damaged too.	Critical	System	All
vprob.vmfs.nfs.server.disconnect|Lost connection to server {1} mount point {2} mounted as {3} ({4}).	Critical	System	All
vprob.vmfs.nfs.server.restored|Restored connection to server {1} mount point {2} mounted as {3} ({4}).	Informational	System	All
vprob.vmfs.resource.corruptondisk|At least one corrupt resource metadata region was detected on volume {1} ({2}). Other regions of the volume might be damaged too.	Critical	System	All
vprob.vmfs.volume.locked|Volume on device {1} locked, possibly because remote host {2} encountered an error during a volume operation and could not recover.	Critical	System	All

1.3 RHEV

Red Hat Enterprise Virtualization (RHEV) is one of the most excessively used virtualization technologies by companies with a Data Center based on Red Hat. Pandora FMS offers the possibility to monitor virtual architectures based on RHEV by the RHEV Monitoring Plug in which allows you to easily control all variables related to the RHEV virtual architecture.

1.3.1 Monitoring RHEV Architectures

You're able to monitor the entire RHEV architecture by this plug in, e.g. Data Centers, Host Clusters, Storage Domains, Networks, Hosts and Virtual Machines, offering a global view of the virtual environment status.

Pandora FMS utilizes the official API which is provided by the RHEV virtualization system to accomplish this.

1.3.2 Monitoring by RHEV Monitoring Plug In

RHEV environment monitoring is based on three components:

1. An agent plug in which performs entity auto-discoveries and data collection tasks. This agent plug in sends information to Pandora FMS.
2. A recon script which updates several parameters of entities discovered. This script is required for extensions.
3. Several RHEV View and RHEV Manager extensions. These are extensions which provide an added value to the plug in, allowing you to see all the monitored infrastructure and managing virtual machines (switch on/switch off) by the Pandora FMS Console.

To be able to use any kind of Recon Script, you're required to enable the Recon Server first.

To ensure some API variables return the correct data of any associated virtual machine, you're required to install the RHEV Agent. You can find all information on how to do that by taking a look into the RHEV Documentation.

To monitor an operating system installed on a virtual machine, it's recommended to use a Pandora FMS Agent instead of the RHEV API.

1.3.2.1 Internal operation of the plugin

The RHEV Monitoring Plug ins extract information by the web API of RHEV virtualization environments.

If you just want to monitor, please configure the software agent plug in which performs this task.

The agent plug in performs the device's auto discovery and creates an XML file along with modules for each discovered device. The plug-in configuration allows you to select which elements you want to monitor and to configure the modules. The modules created by the plug in are completely configurable. You're also able to change names and descriptions and to add 'max' and 'min' values for the 'Warning' and 'Critical' states of the module.

The updating of values for the 'Warning' and 'Critical' states by using XML is only available for Pandora FMS 4.0 or higher versions. For earlier versions, you're required to perform this task by using the web console.

Once the XML files are created, the agent plug in sends the files - either by using Tentacle or by copying them to local files, depending on the selected transfer method.

If you also intend to use the RHEV View and RHEV Manager extensions, you're required to use the Recon Script to do so.

The Recon Script updates several values of each Pandora FMS Agent present in the RHEV virtualization environment. These variables are required to visualize entities properly in the RHEV View extension and to manage virtual machines properly by the RHEV Manager extension.

1.3.3 Installation Requirements

The agent plug in requires the following software:

• curl
• perl-XML-Simple
• Pandora FMS Software Agent
• tentacle_client (if you want to use tentacle to send files. The 'tentacle_client' file is provided along with the Pandora FMS Software Agent)

1.3.3.1 Red Hat

Under Red Hat-based systems, you may install the dependencies by the following command:

yum install perl-XML-Simple curl

1.3.3.2 SLES

Under SUSE-based systems, you may install the dependencies by the following command:

zypper install perl-XML-Simple curl

1.3.3.3 Debian / Ubuntu

Under Debian or Ubuntu-based systems, you may install the dependencies by the following command:

sudo apt-get install libxml-simple-perl curl

1.3.3.4 Installing the Pandora FMS Software Agent

The Pandora FMS Software Agent Installation is explained in the section named Installing Pandora FMS, where you're able to find all relevant information regarding the installation of Pandora FMS Agents onto your platform.

1.3.4 Downloading the RHEV Certificate

Before you're able to execute the plug in, you're required to download the certificate to connect to the RHEV API using HTTPS. To download the certificate, please execute the following command:

curl -o rhevm.cer http://[RHEVM-HOST]:8080/ca.crt

[RHEVM-HOST] is the name of the RHEV API server, e.g.:

curl -o rhevm.cer http://rhevm.server:8080/ca.crt

Once the certificate is downloaded, you can make sure the API connection works fine with the following command:

curl -X GET -H "Accept: application/xml" -u [USER:PASS] --cacert [CERT] https://[RHEVM-HOST]:8443/api

Value explanation:

• USER: [[email protected]] to connect to the API.
• PASS: Password for the user to connect to API.
• CERT: The path of the downloaded certificate.
• RHEVM-HOST: The address of the host API.

A pretty good example with some real data could look like this:

curl -X GET -H "Accept: application/xml" -u [[email protected]:12345] --cacert /home/user/ca.crt https://rhevm.server:8443/api

If all goes fine, the command is going to return an output in the XML format, along with some general information about the RHEV API.

1.3.5 Considerations on RHEV Configuration

In the RHEV virtualization environment, it's possible for several entities to bear the same name. This feature creates quite a problem for Pandora FMS, because these entities are transformed to agents - and two agents bearing the same name are not allowed. In addition to this difficulty, it creates problems by parsing the output of an API in XML format, which could result in an error like this:

Warning: <data_center> element has non-unique value in 'name' key attribute: Default at ./plugin-rhev.pl line 199

To solve this problem, you're required to follow a name policy for entities of the RHEV virtualization environment which doesn't allow duplicate names.

1.3.6 Agent Plug-in Installation

To install the agent plug in, please copy the files 'rhev-plugin.pl' and 'rhev-plugin.conf' into a folder which is accessible by the Pandora FMS Agent and installed on the machine you want to execute the plug in on. The plugin could be executed by an agent which is installed on the same machine the Pandora FMS Server runs on or on another.

To execute the plug in, you're required to enter an additional line to the agent configuration file (which is located under '/etc/pandora/pandora_agent.conf' by default):

module_plugin /root/rhev-plugin.pl /root/rhev-plugin.conf

By adding this line to the configuration file, the agent plug in is going to perform its actions on every execution of the agent.

1.3.7 Monitoring RHEV Virtual Architecture

To see the result of the plug-in execution, please click on Monitoring > Views > Agent Detail.

As you can see, the plugin creates one Pandora FMS Agent for each detected entity if it discovers an RHEV architecture:

If you click on the agent name, you're able to see the monitoring modules created by the plug in. You're also able to see other agent-related data:

For each kind of entity, several modules are created automatically, which are monitoring important information from each of them. The next picture e.g. shows several modules which were created to monitor a virtual machine:

If an agent is associated to a Host instead of a Virtual Machine, the monitored modules are different. The following picture shows an example of modules for a host entity:

The RHEV plugin also monitors the occurred events in virtual architectures. The plugin creates a module for every event monitor in every affected agent:

The data of these event-based modules are:

• The hour in which the event occurred
• The event description.

You can see an example of this data on the picture below:

In addition to the agents and modules related to RHEV architecture, a module is generated on the agent which executes the plug in. This module is called RHEV Plugin by default. You're able to see a result example for this module on the image below:

The content of this plug in is going to be the result of the plug-in execution. It could be something simple like 'OK' if it's conduct was flawless. It could also show an error string, explaining the error if something unexpected occurs. This information is also available in a log file.

1.3.7.1 Monitoring the Status of Entities

The status modules of entities are going to return the predefined values of any RHEV architecture. This means the values are going to be strings which have a content similar to: 'up', 'down', 'error', 'maintenance', 'non_operational', etc. depending on the status and the monitored entity.

To assign 'warning' and 'critical' values, you're required to define a regular expression within the module configuration. To e.g. define the module to be in a 'critical' status if the values are error, down or non_operational, please add the following regular expression to the 'critical' value of the Str. field:

error|down|non_operational

It's not possible to use this option for older versions than Pandora FMS 4.0, but you're still able to define the alert by using the same condition. To create an alert template with the previous example, please follow the below mentioned steps:

1. Create an alert template with 'critical' priority and set the field named condition type to regular expression.
2. Insert the regular expression in the field value as follows: 'error|down|non_operational'. It means, the alert is going to be fired if the module values are error, down or non_operational.
3. Please complete the next step as usual.

Once the template is defined, you're able to select any action to execute in case the alert gets triggered, e.g. creating an event, sending an email or SMS, etc.

1.3.8 Agent Modules for the RHEV Architecture

The available modules for each element of the RHEV architecture are the following:

1.3.8.1 Data Centers

• Status: The Data Center's status.

1.3.8.2 Storage Domains

• Available Space: The available space of a storage domain.
• Committed Space: The dedicated space of a storage domain.
• Used Space: The currently used space of a storage domain.
• Percent Free Space: The percentage of free space on a storage domain.

1.3.8.3 Networks

• Status: The virtual network's status.
• STP Status: The Spanning Tree Protocol's status.

1.3.8.4 Clusters

• Overcommit Percent: The over-commit percentage of the cluster.
• Transparent HugePages: The transparent HugePage status.
• High threshold: The 'high' threshold for policy planning.
• Low threshold: The 'low' threshold for policy planning.
• Threshold duration: The threshold duration for policy planning.

1.3.8.5 Hosts

• Status: The host's status.
• Buffers size: The buffer size.
• Cache size: The cache size.
• Cached swap: The amount of memory for cached swap (in bytes).
• Free memory: The amount of free memory (in bytes).
• Percent free memory: The percentage of free memory.
• Swap cached percent: The percentage of cached swap memory.
• Swap free: The amount of free swapping space (in bytes).
• Swap free percent: The percentage of free swap memory.
• Total Memory: The amount of total memory for this Host (in bytes).
• Total Swap: The amount of swap memory (in bytes).
• Used memory: The amount of used memory (in bytes).
• Used Swap: The amount of used swap memory (in bytes).
• Nic [x] TX: The transmission rate for NIC x (in bytes per sec.). It's going to generate one module for each interface.
• Nic [x] RX: The reception rate for NIC x (in bytes per sec.). It's going to generate one module for each interface.
• Nic [x] erros TX: The number of transmission errors for NIC x. It's going to generate one module for each interface.
• Nic [x] erros RX: The number of reception errors for NIC x. It's going to generate one module for each interface.
• User CPU: The percentage of CPU used by user.
• System CPU: The used percentage of the CPU by the system.
• CPU Idle: The idle percentage of the CPU.
• CPU Load: The average CPU load for the last 5 minutes.
• KSM CPU: The percentage of the CPU which gets used by the KSM.
• Active VM: The number of active virtual machines on the host.
• Migrating VM: The number of virtual machines currently in the process of migrating on the host.
• Total VM: The total number of virtual machines for this host.
• Fence Status: The status of host fencing.

1.3.8.6 Virtual Machines

• Status: The virtual machine's status.
• Disk [x] read: The disk read rate for disk x (in bytes / sec.). It's going to generate one module for each disk.
• Disk [x] write: The disk write rate for disk x (in bytes / sec.). It's going to generate one module for each disk.
• Disk [x] size: The disk size for disk x. It's going to generate one module for each disk.
• Disk [x] status: The status of disk x. It's going to generate one module for each disk.
• Nic [x] TX: The transmission rate for NIC x (in bytes / sec.). It's going to generate one module for each NIC.
• Nic [x] RX: The reception rate for NIC x (in bytes / sec.). It's going to generate one module for each NIC.
• Nic [x] erros TX: The number of transmission errors for NIC x. It's going to generate one module for each NIC.
• Nic [x] erros RX: The number of reception errors for NIC x. It's going to generate one module for each NIC.
• Installed memory: The amount of installed memory (in bytes).
• Percent free memory: The percentage of free memory.
• Used memory: The amount of used memory (in bytes).
• Stateless: The status of the 'stateless' feature.
• HA Status: The status of the HA (High Accessibility) feature.
• Total CPU: The percentage of the total used CPU load by this virtual machine.
• Hypervisor CPU: The percentage of the hyper-visor CPU load used by virtual machine.
• Guest CPU: The percentage of host CPU load used by the virtual machine.

1.3.8.7 Events

• Event [x]: The description for event x which occurred on the system. For every detected event, one module is created within each affected agent.

1.3.9 Managing and Viewing of the RHEV Architecture

This section explains the installation and configuration of the RHEV Architecture and how the 'RHEV View' and 'RHEV Manager' extensions work.

The 'RHEV View' and 'RHEV Manager' extensions are only going to work in conjunction with Pandora FMS 4.0.2 or higher versions.

1.3.9.1 Recon Task Installation

The following is a detailed explanation of Recon Script Installation and Recon Task Creation which are going to update the variables used by the extensions.

1.3.9.1.1 Recon Script Installation

Prior to the creation of the Recon Task, you're required to register the Recon Script which updates the values which are required by the extensions. Please click on Manage Servers and on Manage recon script to do so.

Once the main screen of Manage recon script has popped up, please click on the Add button.

In this moment, a form to enter the details for the new Recon Script is going to appear. You're required to fill out the fields properly as shown on the image below. In the field called Script fullpath you're required to insert the interpreter or program which executes the script ('perl' in this case) and the full path to the script. Once the form is filled out properly, please click on Create.

The moment the recon script is registered, you're going to see a screen, showing the processing was executed properly and the script was registered, appearing on the list.

1.3.9.1.2 Recon Task Creation

To ensure the variables used by the extensions are updated periodically, you're required to create a Recon Task which is going to be executed on each defined time interval. To create a Recon Task, please click on Manage Servers and on Manage recontask.

As you can see on the image below, the main view of Recon Task is shown. Please click on Create to create a new one.

After clicking on Create, the form on the picture below is going to appear. It's very important to select the Custom Script option in the Mode field, because it's going to allow you to select a registered script (the RHEV Recon Script in this case).

The field called Script field is reserved for recon script parameters. For this recon script, you're required to use the following parameters:

• server: The address of the host which runs the API.
• user: The user to access the API (the syntax is '[email protected]').
• pass: The password to access the API.
• cert: The path to the API certificate.
• pandoraconf: The path to where the Pandora FMS configuration file is located.

The cert parameter is going to be used by the Recon Task and RHEV Manager extensions. It's very important to make sure the Pandora FMS Servers -and- Web Servers are allowed to access to this location.

Please click on the Add button to create a new Recon Task and to finish the process.

In this moment, the following screen is going to appear, showing the process was completed successfully. The new Recon Task is going to appear on the list.

In this moment, you possess one Recon Task which will be executed on each defined interval. It will update all variables related to the agents which are going to monitor the RHEV virtual architecture.

1.3.9.2 Installation of RHEV View and RHEV Manager Extensions

To install these extensions, please copy the content of the extensions folder to the extensions folder of the Enterprise part of the Pandora FMS Console which is going to appear after the decompression of the plug in. The command to perform these actions is shown below:

cp -R extensions/* <pandora_console_dir>/enterprise/extensions/

From now on, the RHEV monitoring extensions are available to you.

1.3.9.3 Using the RHEV View Extension

To use the RHEV View extension, please click on Monitoring and RHEV View.

The extension is going to open a map, showing all components of the RHEV architecture which gets discovered by the plug in.

The different elements of RHEV architecture (e.g. Data Centers, Storage Domains, Clusters, Networks, Hosts and Virtual Machines) will appear on the map. Each element is represented by a different icon for each kind of element. The relationship between icons show the relationship between the RHEV architecture elements. You're able to see the status of every element and their relationships to each other at a glance by this view.

The extension has a menu to configure the view: Hiding or showing the entities, enlarging the text size, zooming in and out to see a more detailed picture of the network.

On the picture below, the elements, networks, hosts and virtual machines are hidden, because we need to see a detailed view of the relationship between clusters and storage domains with a data center.

1.3.9.4 Using the RHEV Manager Extension

The RHEV Manager Extension is available in the agent operation view which represents RHEV virtual machines under Pandora FMS.

The RHEV Manager Extension utilizes the curl' command. The installation of this command is required and has to be accessible to the Web Server on which the Pandora FMS Console is installed on.

To access the extension, please click on the icon which is represented by the RedHat logo in the agent's tab bar.

The extensions allow you to manage the virtual machine ('switch on/off' and 'suspend') without being forced to use RHEV Management Console. The extension shows the current status of the virtual machine by a code of colors ('green' = powered on, 'orange' = suspended and 'grey' = powered off), and a combo containing the available states. You're able to change them by clicking on the Change Status button.

If you're clicking the Stop status to stop a virtual machine, the extension is going to contact the RHEV API and sends the command. The result is going to be a change in the virtual machine status and the combo options, as you can see on the picture below:

The change between some states consists of several steps, e.g. changing from Stop to Start. In this case, the extension is going to show the virtual machine status for each step. To change e.g. from 'Stop' to 'Start', the virtual machine crosses the states shown below:

1.3.10 Agent Plug-in Configuration

The agent's plug-in configuration is conducted by a configuration file called 'rhev-plugin.conf' by default.

The agent's plug in selects all entities and creates all modules along with default values for name and description by default. All these parameters can be customized by changing the configuration file.

1.3.10.1 Configuration File

The configuration file has two different areas: The global variables and monitoring configuration variables.

The global variables section begins on the token named Configuration and carries the information about the plug-in configuration. The parameters allowed in this section are the following:

• module_name: The name of the reported module on the agent which executes the plug in.
• server: The host name which runs the RHEV API.
• user: The user to connect to the API (the syntax is '[email protected]').
• pass: The password to connect to the API.
• cert: The path to the API's certificate.
• temporal: The path to the temporal folder.
• logfile: The name of the log file.
• transfer_mode: The transfer mode (can take the values 'local' or 'tentacle').
• tentacle_ip: The tentacle server's IP to send information. Typically it's installed on the same machine the Pandora FMS Server is installed on. This option is only available if you're using 'tentacle' under 'transfer_mode'.
• tentacle_port: The port of the Tentacle server. This option is only available if you're using 'tentacle' under 'transfer_mode'.
• tentacle_opts: These are extra options for the Tentacle server. This option is only available if you're using 'tentacle' under 'transfer_mode'.

The monitoring configuration section comes with several subsections. The first one contains the token named Reject and allows you to create a list which is going to contain the names of the entities of the virtualization environment which are going to get rejected. To reject an entity, you're required to put the name on the list as shown below:

#Dismissed entities
Reject
mv1
mv_WindowsXP
mv_WebServer1
...

It is possible to discard all entities of the same type, e. g. all hosts, all virtual machines, etc. The tokens for each entity are: all_dc (Data Center), all_host (Hosts), all_network (Networks), all_storage (Storage Domain), all_cluster (Cluster), all_vm (Virtual Machines). An example of using these tokens would be:

#Dismissed entities
Reject
all_dc
all_host
all_network
all_storage
all_cluster
all_vm

The second section is defined by the token named Rename and allows you to change entity names. This feature is very useful if you want to combine software agent and API information on the same agent. The configuration for this section is conducted by mentioning the old name followed by the new one and a space character between them as shown below.

#Rename entities
Rename
mv_WebServer1 WebServer1
mv_WindowsXP WindowsXP Test
...

The following subsections are related to the entity's monitoring configuration. Each entity has it's own token named DataCenter', StorageDomain, Network, Cluster, Host y VM. For each entity, it's e.g. possible to define whether the modules are disabled or enabled and to provide max. and min. values for the Warning and Critical states:

#VM Modules
VM
status disabled
errors_total_tx name = TX Error Net [%s]; desc = Total error TX net; limits = 60 70 71 100
memory_used name = Used Mem; desc = Memory used by the virtual machine; limits = 256 1024 1025 2048
...

Each line is associated to a monitoring module. There are two options:

• <module> disabled: The module is -not- going to be created.
• <module> name = <name>; desc = <description>; limits = <min_warning> <max_warning> <min_critical> <max_critical>: The module is going to be created with a specified name and description. It's also going to contain the thresholds for min. and max. values and for the 'Warning' and 'Critical' states.

It's very important to pay special attention to the configuration file's line structure and syntax, especially to the character ;. It's located in direct vicinity to the module's name and its description. The command line examples which are shown below are NOT THE SAME. Please take a good look at the blanks near the ;:

errors_total_tx name = TX Error Net [%s]; desc = Total error TX net; limits = 60 70 71 100 (RIGHT)
errors_total_tx name = TX Error Net [%s]    ; desc = Total error TX net    ; limits = 60 70 71 100 (WRONG!)

The modules are referenced by their short names, and a name is easier to write on the command line. A table that explains how to link full names and short names is located in the next section.

This is an example of the configuration of virtual machines:

To monitor virtual machines was defined as a list of enabled or disabled modules inside the configuration file in the VM section. The 'status' module is disabled and the modules named 'errors_total_tx' and 'memory_used' contain custom values. The rest of the modules which are not showing up on the list, are going to be created along with a set of default values for them. By this configuration, the module named 'memory_used' is going to get the following values:

• Name: The used memory.
• Description: The memory used by the virtual machine.
• Min Warning: 256
• Max Warning: 1024
• Min Critical: 1025
• Max Critical: 2048

The modules are generated dynamically, e.g. modules related to disks or network interfaces, which are going to create one module for each detected item, have a special syntax for the module's name:

errors_total_tx name = Errores TX Net [%s]; desc = Errores totales TX de red; limits = 60 70 71 100

In this case, the name has a dynamic part which allows you to use the macro % which will be replaced with the dynamic part of the module's name by the plugin.

The module named errors_total_tx e.g. has this default name:

Nic [nic1] errors TX 

By this configuration, the name is going to be:

TX Error Net [nic1] 

Where 'nic1' is the dynamic part of the module's name.

All errors related to the configuration file are shown in the log file. They are also going to be sent as an asynchronous module to Pandora FMS which is going to appear in the agent which is executing the plug in.

In addition to the section related to each element, the configuration file has a common section for the events. This section is defined by the token named 'EventCodes' and all event codes to monitor will be listed inside it:

EventCodes
30
920
980
509
956

If you don't define this section, the event monitoring is -not- going to be executed.

1.3.10.2 Sharing the Monitoring Load between several Software Agents

By configuration file, it's possible to share the monitoring load of RHEV Virtualization Environments between several Software Agents.

To do that, you're required to distribute the monitored entities between the agents. In this example, we have the following architecture:

DC1
 |
 |- Cluster 1.1
       |- c1.1mv1
       |- c1.1mv2
       |- c1.1mv3

 |- Cluster 1.2
       |- c1.2mv1
       |- c1.2mv2
       |- c1.2mv3

DC2
 |
 |- Cluster 2.1
       |- c2.1mv1
       |- c2.1mv2
       |- c2.1mv3

 |- Cluster 2.2
       |- c2.2mv1
       |- c2.2mv2
       |- c2.2mv3

One possibility to share the load could be assigning one Data Center to each agent. We're going to use the feature to reject entities named 'Reject' to do so.

The first agent only monitors the Data Center called 'DC1' and rejects the entities in Data Center 2 which is called 'DC2':

Reject
DC2
Cluster 2.1
Cluster 2.2
c2.1mv1
c2.1mv2
c2.1mv3
c2.2mv1
c2.2mv2
c2.2mv3

The second Software Agent monitors the data center 'DC2' and rejects the data center 'DC1':

Reject
DC1
Cluster 1.1
Cluster 1.2
c1.1mv1
c1.1mv2
c1.1mv3
c1.2mv1
c1.2mv2
c1.2mv3

It's also possible to split the load based on clusters. We have four software agents and each one is going to monitor a different cluster.

Software Agent 1 monitors cluster 1.1 and rejects the other entities:

Reject
DC1
Cluster 1.2
c1.2mv1
c1.2mv2
c1.2mv3
DC2
Cluster 2.1
Cluster 2.2
c2.1mv1
c2.1mv2
c2.1mv3
c2.2mv1
c2.2mv2
c2.2mv3

Software Agent 2 monitors cluster 1.2 and rejects the other entities:

Reject
DC1
Cluster 1.1
c1.1mv1
c1.1mv2
c1.1mv3
DC2
Cluster 2.1
Cluster 2.2
c2.1mv1
c2.1mv2
c2.1mv3
c2.2mv1
c2.2mv2
c2.2mv3

Software Agent 3 monitors cluster 2.1 and rejects the other entities:

Reject
DC1
Cluster 1.1
Cluster 1.2
c1.1mv1
c1.1mv2
c1.1mv3
c1.2mv1
c1.2mv2
c1.2mv3
DC2
Cluster 2.2
c2.2mv1
c2.2mv2
c2.2mv3

Software Agent 4 monitors cluster 2.2 and rejects the other entities:

Reject
DC1
Cluster 1.1
Cluster 1.2
c1.1mv1
c1.1mv2
c1.1mv3
c1.2mv1
c1.2mv2
c1.2mv3
DC2
Cluster 2.1
c2.1mv1
c2.1mv2
c2.1mv3

the rejected entities configuration is very flexible, because you're able to split the load, assigning several entities to each software agent.

1.3.10.3 Example Configuration Files

1.3.10.3.1 Configuration File with all Modules disabled

The lines marked by a '#' character are comments.

#Plug-in Configuration Parameters
Configuration
server rhevm.server
user [email protected]
pass 12345
cert /home/user/rhevm.cer
temporal /tmp
logfile /tmp/plugin-rhev.log
transfer_mode local
tentacle_ip 127.0.0.1
tentacle_port 41121
tentacle_opts

#Dismissed Entities
Reject

#Data Center Modules
DataCenter
status disabled

#Storage Domain Modules
StorageDomain
available disabled
used disabled
committed disabled
free_percent disabled 

#Network Modules
Network
status disabled
stp disabled

#Cluster Modules
Cluster
overcommit disabled
hugepages disabled
threshold_low disabled
threshold_high disabled
threshold_duration disabled

#Host Modules
Host
status disabled 
vm_active disabled
vm_migrating disabled
vm_total disabled
data_current_rx disabled
data_current_tx disabled
errors_total_rx disabled
errors_total_tx disabled
memory_cached disabled
memory_total disabled
swap_free_percent disabled
swap_cached_percent disabled
swap_free disabled
cpu_current_idle disabled
cpu_current_user disabled
memory_used disabled
ksm_cpu_current disabled
memory_free_percent disabled
swap_total disabled
memory_buffers disabled
cpu_current_system disabled
cpu_load_avg_5m disabled
swap_cached disabled
swap_used disabled
memory_free disabled
fence_status disabled

#VM Modules
VM
status disabled
stateless disabled
ha disabled
cpu_current_guest disabled
cpu_current_hypervisor disabled
memory_free_percent disabled
memory_installed disabled
memory_used disabled
cpu_current_total disabled
data_current_read disabled
data_current_write disabled
size disabled
disk_status disabled
data_current_rx disabled 
data_current_tx disabled
errors_total_rx disabled
errors_total_tx disabled

1.3.10.4 Table linking the Module Names

1.3.10.4.1 Data Centers

Long name	Short name
Status	status

1.3.10.4.2 Storage Domains

Long name	Short name
Available Space	available
Used Space	used
Committed Space	committed
Percent Free Space	free_percent

1.3.10.4.3 Networks

Long name	Short name
Status	status
STP Status	stp

1.3.10.4.4 Clusters

Long name	Short name
Overcommit Percent	overcommit
Transparent HugePages	hugepages
Low Threshold	threshold_low
High Threshold	threshold_high
Threshold duration	threshold_duration

1.3.10.4.5 Hosts

Long name	Short name
Status	status
Active VM	vm_active
Migrating VM	vm_migrating
Total VM	vm_total
Nic [x] RX	data_current_rx
Nic [x] TX	data_current_tx
Nic [x] errors RX	errors_total_rx
Nic [x] errors TX	errors_total_tx
Cache size	memory_cached
Total memory	memory_total
Swap free percent	swap_free_percent
Swap cached percent	swap_cached_percent
Swap free	swap_free
CPU Idle	cpu_current_idle
User CPU	cpu_current_user
Used memory	memory_used
KSM CPU	ksm_cpu_current
Percent free memory	memory_free_percent
Total swap	swap_total
Buffers size	memory_buffers
System CPU	cpu_current_system
CPU Load	cpu_load_avg_5m
Cached swap	swap_cached
Used swap	swap_used
Free memory	memory_free
Fence Status	fence_status

1.3.10.4.6 Virtual Machines

Long name	Short name
Status	status
Stateless	stateless
HA Status	ha
Guest CPU	cpu_current_guest
Hypervisor CPU	cpu_current_hypervisor
Percent free memory	memory_free_percent
Installed memory	memory_installed
Used memory	memory_used
Total CPU	cpu_current_total
Disk [x] read	data_current_read
Disk [x] write	data_current_write
Disk [x] size	size
Disk [x] status	disk_status
Nic [x] RX	data_current_rx
Nic [x] TX	data_current_tx
Nic [x] errors RX	errors_total_rx
Nic [x] errors TX	errors_total_tx

1.4 Nutanix

The Nutanix hyperconvergence solution offers us the possibility of managing all network, disk, processing and memory resources from a single point.

Pandora FMS monitoring plugin for Nutanix, allows us to control at all times the status of our Nutanix solution.

1.4.1 Plugin operation

The Nutanix plugin is a program written in Perl, which will connect to the REST API of Nutanix PRISM, retrieving the necessary metrics to monitor the following elements:

• Nutanix clusters
• Storing devices
• Containers
• Virtual machines
• Hosts
• Status of replication processes

1.4.2 Plugin requirements

In order to retrieve the REST API information, we will need:

• The IP address/ FQDN of the portal.
• A user with reading permissions on the API.
• The password of that user.

Regarding the communication of monitoring results to your Pandora FMS. You'll need:

• The information transfer mode, either locally or via Tentacle.
  • If it is local, the address of the directory where the XML files with the results are to be delivered as well as the write permissions in that directory.
  • In case of being communication via Tentacle, it will be necessary to be able to connect against the IP address or FQDN of the Pandora server, the port used by your Tentacle installation, the location of the Tentacle client as well as any extraordinary option you have defined.

1.4.3 Plugin Installation

Download the files required by the plugin from the module library

Transfer the files to the remote computer from where you want to monitor your Nutanix infrastructure and extract the files from the plugin:

tar xvzf pandora_nutanix.tar.gz

1.4.4 Plugin configuration

The following fields are declared:

Nutanix API configuration

nx_fqdn

Prism main server address

nx_port

Port where the REST API is published (default 9440)

nx_user

User with reading privileges on the REST API

nx_pass

that user's password

use_https

Use https (1) or not (0)

nx_rest_version

API Rest version (by default 'v1')

Nutanix agent configuration

agent_interval

Interval of the agents generated by the plugin (default 300)

agent_group

Group to which the agents generated will belong (if' autocreate_group' is commented in the configuration of your PandoraServer), by default Nutanix

module_interval

Interval of generated agent modules (multiplication factor, default 1)

module_tags

Tags associated with the new modules of the agents generated.

module_group

Group to which new modules will belong to.

Configuration of communication to the Pandora server

mode

Data transfer mode, "local" or "tentacle"

tentacle_ip

Pandora server IP address, only applies in tentacle mode

tentacle_port

Port where the Tentacle service is listening

tentacle_opts

Any extra options you have configured in your Tentacle service

tentacle_client

Complete path to your Tentacle customer

temp

Temporary Work Directory

local_folder

Delivery path for the "local" data transfer mode

Filters

cluster_monitoring

Enable (1) or not (0) cluster monitoring

storage_monitoring

Enable (1) or not (0) monitoring of storage devices

container_monitoring

Enable (1) or not (0) monitoring of storage containers

vm_monitoring

Enable (1) or not (0) monitoring of virtual machines

host_monitoring

Enable (1) or not (0) monitoring of virtual machine servers (Nutanix nodes)

pd_monitoring

Enable (1) or not (0) monitoring of protection domains

Customization

cluster_agent_header

Header for the agent name of cluster-type devices agent

storage_agent_header

Header for agent name of storage device type agent

host_agent_header

Header for the agent name of virtual machine server type devices (Nutanix nodes)

container_agent_header

Header for agent name of storage container type devices

vm_agent_header

Header for agent name of virtual machine type device agent

pd_agent_header

Header for the agent name of protection domain type devices agent

Module generation rules

vm_stat

Add-module aggregation rule for monitoring virtual machines, by default 'hypervisor_cpu_usage_ppm|hypervisor_memory_usage_ppm|.*avg.*', This indicates the extraordinary modules that will be generated, when the name of the metric matches the regular expressions indicated in this field. Add the value ".*" to monitor all available metrics.

host_stat

Rule for adding modules for monitoring virtual machine servers (Nutanix nodes), default' hypervisor_cpu_usage_ppm|hypervisor_memory_usage_ppm|. *avg*'.This indicates the extraordinary modules that will be generated when the name of the metric matches the regular expressions indicated in this field. Add the value ".*"to monitor all available metrics.

pd_stat

Rule for adding modules for the monitoring of protection domains, by default 'replication_transmitted_bandwidth_kBps|replication_total_transmitted_bytes', this indicates the extraordinary modules that will be generated, when the name of the metric matches the regular expressions indicated in this field".*" Add the value.to monitor all available metrics.

Renamed entities

RENAME aaa TO bbb

Rule for renaming entities, you can define as many directives as you need to rename elements

Entities exclusion

REJECT aaa

Rule for entity monitoring exclusion, you can define as many directives as you need to exclude

1.4.5 Plugin Execution

It is recommended to run the plugin remotely from a computer with access to both Pandora Server and its Nutanix infrastructure to monitor.

Manual execution:

./pandora_nutanix-linux-x64 pandora_nutanix.conf

You can automate the execution of the plugin in the system cron by adding the following line to /etc/crontab

*/5 * * * * root /path/to/plugin/pandora_nutanix-linux-x64 /path/to/plugin/pandora_nutanix.conf

1.4.6 Plugin results

Captures with the results obtained by the plugin:

Example of Nutanix architecture map:

Example of Nutanix architecture agent view:

Example of module view for Host Nutanix agents:

1.5 XenServer

Xen is an open source virtual machine monitor developed by the University of Cambridge.

The goal of the design is to be able to run instances of operating systems with all their features, fully functional in a simple computer.

Xen provides secure isolation, resource control, quality of service guarantees and hot migration of virtual machines. Operating systems can be explicitly modified to run Xen (although maintaining compatibility with user applications). This allows Xen to achieve high-performance virtualization without special hardware support.

Intel has made a number of contributions to Xen that have enabled it to add support for its VT-X Vanderpool architecture extensions. This technology allows unmodified operating systems to act as hosts within Xen virtual machines, as long as the physical server supports Intel or AMD's VT extensions.

1.5.1 Plugin operation

The Pandora FMS plugin for Xen environment monitoring is written in Python. Use XenAPI to retrieve all the necessary information. It allows the monitoring of the following types of elements:

• Virtualized systems in Xen
• Storage resources
• Xen's own server (host).

1.5.2 Plugin requirements

It is essential that the system running the plugin has the following requirements:

• Installed Python
• Python libraries installed:
  • XenAPI
  • xmltodict
• Access to your XenServer API (web, enable traffic from the computer running the plugin to port 443 or 80 of XenServer)
• It is recommended that virtual machines have Xen Server Tools installed, since the information available is quite excessive otherwise.

Plugin operation has been successfully tested on XenServer 6.5 and 7.2 systems

1.5.3 Plugin installation

Download your copy of the Pandora FMS plugin for XenServer from the module library.

You can deploy it on the computer of your choice (Windows or Linux), extracting the contents of the file in a non-volatile directory from where you can run it either using Pandora FMS agent or system cron.

1.5.4 Plugin configuration

Available configuration for Pandora FMS plugin for Xen:

Configuration block [CONF]

xen_server_ip

Xen Server IP/FQDN adress

user

User with query permissions about the Xen API

password

User password

temporal

Temporary Work Directory

Configuration block [PANDORA]

tentacle_client

Tentacle Location of the Tentacle client binary

tentacle_ip

IP address where the Tentacle service is listening.

tentacle_port

Port where the Tentacle service is listening.

logfile

Complete path to the log file

interval

Interval of generated agents

group

Group assigned to the generated agents.

Configuration block [TUNNING]

time_adjustment

Parameter that allows the adjustment of the possible time differences between the computer running the plugin and the Xen server. (default =10, measured in seconds).

scan_vm_ip

Parameter that allows to define if the plugin will try to obtain the IPs of the VMs of the Xen server. You can only take the IPs of those VMs with the XenTools installed. It can be enabled (scan_vm_ip=true) or disabled (scan_vm_ip=false). If not specified, it is considered enabled.

Configuration block [RENAME]

xen_element_name=pandora_agent_name

In this block, as many entries as desired can be defined with this format. It allows to change the names of the Xen Server elements to different ones to be used as agent names in Pandora. VMs, SRs and Xen Server itself can be renamed, for example:

[RENAME]
example-xen-server=Example Xen Server
Example Xen Server 2=example-xen-server-2
example-vm=Example VM
Example VM 2=example-vm-2
example-sr=Example SR
Example SR 2=example-sr-2

Even though names may have blank spaces, quotation marks should not be used.

1.5.5 Running the Plugin

You can program the execution of the plugin from any Pandora FMS agent adding the following to its configuration:

module_plugin python "<ruta>\xen-plugin.py" "<ruta>\xen-plugin.conf"

To program it through the system cron you can add the following line to /etc/crontab:

*/5 * * * * root python "<ruta>\xen-plugin.py" "<ruta>\xen-plugin.conf" > /dev/null 2>&1

If you run the plugin manually, the output must be similar to the one below:

python "<ruta>\xen-plugin.py" "<ruta>\xen-plugin.conf"
<module>
<name><![CDATA[XenServer Plugin]]></name>
<type><![CDATA[async_string]]></type>
<description><![CDATA[Result of XenServer Plugin execution]]></description>
<data><![CDATA[OK]]></data>
</module>

1.5.6 Plugin results

Captures with the results obtained by the plugin:

Example of Xen architecture map:

Example of Xen architecture agent view:

Example of module view for Host Xen agents:

1.6 OpenNebula

OpenNebula is a platform for cloud computing focused on distributed and heterogeneous data centers, providing the virtual infrastructure to build private, public, and hybrid implementations of infrastructure as a service (IaaS) clouds.

OpenNebula is open source software under the Apache 2 License.

1.6.1 Plugin operation

The Pandora FMS plugin for OpenNebula environment monitoring is written in Perl. It runs locally on the OpenNebula server and will retrieve all the necessary information using OpenNebula's own management commands. It allows the monitoring of the following types of elements:

• Clusters
• Hosts
• Virtual machines
• Storing resources

1.6.2 Plugin requirements

It is essential that the system running the plugin has the following requirements:

• Available Perl in the computer
• User with privileges to execute the following commands:
  • onehost
  • onecluster
  • onedatastore

Plugin operation has been successfully tested on OpenNebula systems 5.X.X

1.6.3 Plugin Installation

Download your copy of the Pandora FMS plugin for OpenNebula from the module library.

You must extract the contents of the file in a non-volatile directory from where you can run it either using Pandora FMS agent or system cron.

unzip pandora_OpenNebula.zip

1.6.4 Plugin configuration

Configuration available for Pandora FMS plugin for OpenNebula:

Configuration of communication to the Pandora server

mode

Data transfer mode, "local" or "tentacle"

tentacle_ip

Pandora server IP address, only applies in tentacle mode

tentacle_port

Port where the Tentacle service is listening

tentacle_opts

Any extra options you have configured in your Tentacle service

tentacle_client

Complete Path to your Tentacle client

temp

Temporary Work Directory

local_folder

Delivery path for the "local" data transfer mode

Agent configuration

agent_interval

Agent interval, 300 by default

agent_group

Agent group, OpenNebula by default

Module customization

MODULE_GROUP

Module group, OpenNebula by default

MODULE_INTERVAL

Module interval (multiplier), 1 by default

MODULE_TAGS

Tags for the modules

Name customization

cluster_agent_header

Header for the agent name of cluster-type devices agent

host_agent_header

Header for the agent name of virtual machine server type device agent

storage_agent_header

Header for agent name of storage device type agent

vm_agent_header

Header for agent name of virtual machine type device agent

Filters

cluster_monitoring

Enable(1) or not (0) cluster monitoring

host_monitoring

Enable(1) or not (0) virtual machine servers monitoring

storage_monitoring

Enable(1) or not (0) storage devices monitoring

vm_monitoring

Enable(1) or not (0) virtual machines monitoring

Entities renaming

RENAME aaa TO bbb

Rule for renaming entities, you can define as many directives as you need to rename elements

Entities exclusion

REJECT aaa

Rule for entity monitoring exclusion, you can define as many directives as you need to exclude

1.6.5 Running the plugin

To program it through the system cron you can add the following line to /etc/crontab:

*/5 * * * * root "<ruta>/pandora_opennebula" "<ruta>/pandora_opennebula.conf" > /dev/null 2>&1

If you run the plugin manually, the output must be similar to the one below:

[[email protected] ~]# ./pandora_opennebula pandora_opennebula.conf
[[email protected] ~]# echo $?
0

1.6.6 Plugin Results

List of modules generated by the OpenNebula plugin:

For Hosts:

1. Available CPU
2. Available Disk
3. Available memory
4. Error Error reported by OpenNebula
5. State
6. Total zombies
7. VMs running

For storage devices:

1. Disk Free %

For virtual machines:

1. Assigned CPU
2. Assigned Memory
3. Network RX
4. Network TX
5. State
6. Time running

Example of OpenNebula architecture map:

1.7 IBM HMC

This plugin allows you to monitor IBM AIX virtualization machines through the HMC hardware management console. This plugin will collect information from all logical partitions created in an AIX environment managed by an HMC system, creating one agent for each managed server, each logical partition, and each virtual IO server.

To collect information via SSH, the plugin can use three modes of operation:

1. Based on expect using the script ssh_launcher.sh
2. Based on the Net::SSH::Perl library
3. Based on the Net::SSH::Expect library

To complement the captured information, queries will also be made against the REST API. (in https://fqdn:12443/rest/api/{root_element} by default).

1.7.1 Requirements

The necessary parameters for the monitoring that has to be provided by the area that requires monitoring services are:

• User name for authentication in the HMC system (read-only)
  • The user must have permission to connect to the REST API and to login to the HMC shell and execute the following commands (at least):
    • lssyscfg
    • lshwres
• That user's password
• HMC location (FQDN/IP) (p.e. myhmc.mydomain)
• Base URL of the HMC rest API (e.g. https://myhmc.mydomain:12443)

1.7.2 Modules generated by the plugin

The parameters monitored by the plugin are (grouped by element type):

• Current logical partitions Current logical partitions deployed
• Max logical partitions Max available logical partitions
• Max memory available Available memory
• Max memory installed Max memory installed
• Proc pool DefaultPool current proc units
• Proc pool DefaultPool max proc units
• Proc pool DevelopmentPool current proc units
• Proc pool DevelopmentPool max proc units
• Proc pool ProductionPool current proc units
• Proc pool ProductionPool max proc units
• Proc pool TestPool current proc units
• Proc pool TestPool max proc units
• Proc pool VIOPool current proc units
• Proc pool VIOPool max proc units
• Processor pools configured Processor pools configured
• Processor units available Available processor units
• Processor units installed Installed processor units
• State Status of the managed system
• UUID Used to query the HMC API
• Virtual proc units max Max virtual processor units for logical partitions

LPAR:

• Auto start Logical partition autostart configuration
• LPAR type Logical partition type
• LPAR UUID Used to query the HMC API
• Max memory Max memory
• Max memory current Available memory
• Processor units available Available processor units
• Processor units current Installed processor units
• RMC IP address RMC IP address
• RMC state RMC status on the LPAR
• State Logical partition status
• Virtual proc units Virtual processor units assigned to this LPAR

Virtual IO:

• Auto start Logical partition autostart configuration
• LPAR type Logical partition type
• LPAR UUID Used to query the HMC API
• Max memory Max memory
• Max memory current Available memory
• Processor units available Available processor units
• Processor units current Installed processor units
• RMC IP address RMC IP address
• RMC state RMC status on the LPAR
• State Logical partition status
• Virtual proc units Virtual processor units assigned to this LPAR

1.7.3 Plugin configuration

The parameters monitored by the plugin are (grouped by element type):

Configuration of communication to Pandora's server

mode

Data transfer mode, "local" or "tentacle"

tentacle_ip

Pandora server IP address, only applies in tentacle mode

tentacle_port

Port where the Tentacle service is listening

tentacle_opts

Any extra options you have configured in your Tentacle service

tentacle_client

Complete path to your Tentacle client

temp

Temporary work directory

local_folder

Delivery path for the "local" data transfer mode

Access configuration to HMC

hmc_host

IP or FQDN of the HMC

hmc_user

User with reading permission

hmc_pass

Password

as_agent_plugin

The plugin output will be returned in XML format for programmed executions with Pandora FMS agent (as_agent_plugin = 1). Or standard output (as_agent_plugin = 0) for runs programmed with the cron system or performed as a server plugin.

Agent configuration

agent_name

Optional, set a name for the parent agent, 'hostname' by default

agent_interval

Agent interval, 300 by default

agent_group

Agent group, IBM by default

Module customization

module_group

Module group, IBM by default

module_interval

Module interval (multiplier), 1 by default

module_tags

Tags for the modules

Entities renaming

For renaming entities, a block rename is used:

rename
MyLPAR_NAME TO my new name
MyLPAR_NAME2 TO my second new name
rename_end

1.7.4 Running the plugin

The Pandora plugin for monitoring IBM AIX systems through HMC is deployed as follows:

Setting the as_agent_plugin parameter to 1 (execution as agent plugin):

module_plugin /usr/bin/perl pandora_hmc.pl pandora_hmc.conf

Setting the as_agent_plugin parameter to 0 (execution as server plugin):

# /etc/crontab
*/5 * * * * root /usr/bin/perl /root/hmc/pandora_hmc.pl /root/vmware/pandora_hmc .conf

1.8 HPVM

Machine virtualization, provided by Hewlett-Packard, allows multiple virtual machines to run concurrently on any Itanium server running HP-UX. It is a server-optimized product.

1.8.1 Plugin operation

This plugin allows you to monitor HPVM virtualization machines. It launches as an agent plugin, generating in parallel one more agent for each virtualized computer hosted in the monitored system.

Local commands are used to collect the information.

1.8.2 Plugin requirements

1. Deploy a Pandora FMS agent on the computer you want to monitor
2. To have a user with permissions to run the plugin
3. This user must have permission to execute the command hpvvmstatus to interpret the output:
   1. hpvmstatus
   2. hpvmstatus -X
   3. hpvmstatus -r -X

1.8.3 Plugin Installation

Download your copy of the Pandora FMS plugin for OpenNebula from the module library.

You can program the execution using collections and the Pandora FMS agent deployed or extract the content of the file in a non-volatile directory from where you can execute it through cron of your system.

unzip pandora_HPVM.zip

1.8.4 Plugin configuration

Configuration available for Pandora FMS plugin for HPVM:

Configuration of communication to the Pandora server

mode

Data transfer mode, "local" or "tentacle"

tentacle_ip

Pandora server IP address, only applies in tentacle mode

tentacle_port

Port where the Tentacle service is listening

tentacle_opts

Any extra options you have configured in your Tentacle service

tentacle_client

Complete Path to your Tentacle client

temp

Temporary Work Directory

local_folder

Delivery path for the "local" data transfer mode

Agent configuration

agent_name

Optional, set a name for the parent agent, 'hostname' by default

agent_interval

Agent interval, 300 by default

agent_group

Agent group, HPVM by default

Module customization

MODULE_GROUP

Module group, OpenNebula by default

MODULE_INTERVAL

Module interval (multiplier), 1 by default

MODULE_TAGS

Tags for the modules

1.8.5 Running the plugin

Running the plugin from Pandora FMS agent, it will appear in the agent configuration file:

module_plugin /usr/bin/perl pandora_hpvm.pl pandora_hpvm.conf

For a manual test, configure the plugin by following the steps described above, you can launch it as follows:

perl pandora_hpvm.pl pandora_hpvm.conf

1.8.6 Plugin results

List of modules generated by the plugin by element type.

Agentes servidor (quien lanza el plugin)

• Available VMs
• HPVM Plugin HPVM Plugin execution status
• HPVM Service Presence of hpvmctrld (X instances)
• HPVM Service CPU usage CPU usage of hpvmctrld (X instances)
• HPVM Service RAM usage RAM usage of hpvmctrld (X instances)
• hpvmapp Presence of hpvmapp (X instances)
• hpvmapp CPU usage CPU usage of hpvmapp (X instances)
• hpvmapp RAM usage RAM usage of hpvmapp (X instances)

Virtual Machines

• boot_type
• console_type
• CPU usage
• device_number
• direct_io_number
• distributed
• effective_serverid
• guest_type
• Host RAM available
• Host RAM free
• local_id
• memory total
• model_name
• networks_number
• run_pid
• run_serverid
• serial_number
• uuid
• vcpu_number
• vm_condition
• vm_config_label
• vm_config_version
• vm_state
• vm_version
• vm_version_label

Go back to Pandora FMS Documentation Index