AWS S3

• Introduction
• Compatibility matrix
• Prerrequisites
• Parameters and configuration
• Manual execution
• Discovery
• Agent and modules generated by the plugin

Introduction

This plugin aims to monitor AWS buckets, using metrics such as the number of objects and their size, which can help better track application costs and performance.

The plugin connects to the AWS API and monitors buckets using the aforementioned metrics, generating an agent for each region via XML that is sent to the Pandora server.

Compatibility matrix

Systems where it has been tested	Rocky linux, Fedora 34
Systems where it works	Any linux system

Prerrequisites

• AWS API Connectivity
  The plugin is a binary that contains the packaged dependencies so it is not necessary to install any type of dependency for its use.
• Permissions
  To make requests through the API, the account used to authenticate during execution requires certain permissions.

Permission Assignment 
Create a policy in JSON like the following:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "VisualEditor0",
           "Effect": "Allow",
           "Action": [
  			   "s3:ListAllMyBuckets",
               "s3:GetBucketLocation"
           ],
           "Resource": "*"
       }
   ]
}

The previous policy must be assigned to a new user.

The user who has these permissions will be able to access AWS S3 monitoring through this plugin.

Parameters and configuration

Parameters

--conf

Path to configuration file

Configuration file (--conf)

agents_group_name = < Name of the target group for the created agents >
interval = < Interval in seconds for agents and for metric analysis >
threads = < Number of execution threads, each zone/instance will be equally distributed in the number of threads >
transfer_mode = < Transfer mode, tentacle or local >
tentacle_ip = < IP of the target machine for the created agents >
tentacle_port = <tentacle port, default: 41121>
tentacle_opts = < Tentacle client additional options >
data_dir = < (Only activated if the transfer_mode is local) Destination path for the XML of each agent, by default "/var/spool/pandora/data_in/" >

s3_monitoring = < Enable with 1 to enable widespread bucket monitoring >
size_monitoring = < Enable with 1 to enable bucket size monitoring >
items_monitoring = < Activate with 1 to enable monitoring of the number of objects in the buckets >

stats_agent = < Activate with 1 to enable a global agent that will monitor based on the task created and the parameters used >
stats_agent_name = < Name for the agent that is activated with the "stats_agent" parameter. If you do not use and "stats_agent" is enabled, the agent will be called "Aws" by default >

aws_regions = < List of regions to monitor (when you mark a region to monitor, it automatically monitors all buckets found within that region) >
aws_buckets = < List of buckets to monitor >

creds_b64 = < Base64 credentials in the JSON file to authenticate >

Example

agents_group_name  = Aws
interval           = 300
threads            = 4
transfer_mode      = tentacle 
tentacle_ip        = 172.42.42.101
tentacle_port      = 41121
data_dir           = /var/spool/pandora/data_in/

s3_monitoring      = 1
size_monitoring    = 1
items_monitoring   = 1

stats_agent        = 1
stats_agent_name   = AwsCloud

aws_regions        = ["us-east-1","us-east-2","us-west-1","us-west-2","ca-central-1","eu-central-1","eu-west-1","eu-west-2","eu-west-3","ap-northeast-1","ap-northeast-2","ap-southeast-1","ap-southeast-2","ap-south-1","sa-east-1"]
aws_buckets        = ["Bucket-1","Bucket-2","Bucket-3","Bucket-4"]

creds_b64          =  ewdhBDJDdvb2tleV9pZGdhjDNDHDhbdjdKKDNDbdBiwKInNlY3JldFSHSHHDGJCJChfDHCNCNHCdjdghDMDBGBkxlSLiIKfQ==

 

 

Manual execution

The plugin execution format is as follows:

./pandora_aws_s3 --conf < path to configuration file >

For example:

./pandora_aws_s3 --conf /usr/share/pandora_server/util/plugin/aws_s3.conf

The execution will return an output in JSON format with information about the execution, and will generate an XML file for each agent in each monitored region that will be sent to the Pandora FMS server by the transfer method indicated in the configuration.

For example:

{"summary": {"Total buckets": 32, "Zones with buckets": 7}}

 

Discovery

This plugin can be integrated with Pandora FMS Discovery.
 
To do this, you must load the ".disco" package that you can download from the Pandora FMS library:

https://pandorafms.com/library/

Once uploaded, Amazon S3 environments can be monitored by creating Discovery tasks from the Management > Discovery > Cloud section.

For each task, the following minimum data will be requested:

• AWS Credentials: Connection credentials to the AWS API to be used by the task. They must be previously registered in the Pandora FMS credential store.
  
• Max threads: To optimize the execution time, multiple threads can be configured to monitor the agents of the task. Keep in mind that setting up multiple threads can increase the CPU usage of the task.
  
• Tentacle IP – Tentacle server IP to send agent data to. Normally this parameter will not be modified.
  
• Tentacle port: Tentacle server port to send data to. Normally this parameter will not be modified.
  
• Tentacle extra options: Additional options for sending data through the Tentacle client. Normally this parameter will not be modified.
  
• Add global stats agent: If activated, a global agent will be generated with task statistics.
  
• Stats agent name: It will allow you to indicate the name of the agent for global statistics of the task. If no name is provided, the default name of the plugin will be used.

If the credentials provided are correct and the Pandora FMS server is able to connect to the AWS API, you will be able to see a tree with AWS S3 zones and buckets, which can be marked for monitoring.

If a zone is selected, in addition to the zone itself, all the buckets it contains will be monitored (both at the time of configuring the task and later if new buckets are included).

If specific buckets are selected they will be monitored regardless of whether their zones have not been selected.

Finally, you can adjust the monitoring you want to obtain:

• S3 Monitoring: If activated, the task will be enabled to obtain additional monitoring for the agents, described in the following fields.
  
• Monitor bucket size: If activated, the bucket size modules will be included in the monitoring.
  
• Monitor bucket total items: If activated, the item modules of the buckets will be included in the monitoring.

Tasks that are successfully completed will have an execution summary with the following information:

• Total regions: Total number of regions monitored by the task.
  
• Total buckets: Total number of buckets monitored by the task.

The tasks that are not completed successfully will have an execution summary recording the errors produced.

Agent and modules generated by the plugin

Running the plugin will create the following agents and modules:

• Global agent

< Name used with the parameter "stats_agent_name" or failing that "Aws" >

Modules

AWS S3 Buckets count

Total buckets registered in AWS

• One agent for each monitored region.

Modules

summary.Aws.ec2.buckets

Total buckets registered in AWS

For each bucket monitored in the region, the following modules will be created:

bucket.items <Bucket name>	Number of objects in the bucket
bucket.size <Bucket name>	bucket size