AWS S3

This document describes the AWS S3 functionality of PandoraFMS discovery.

Introduction

This plugin aims to monitor AWS buckets, using metrics such as the number of objects and their size, which can help better track application costs and performance.

The plugin connects to the AWS API and monitors buckets using the aforementioned metrics, generating an agent for each region via XML that is sent to the Pandora server.

Compatibility matrix

Systems where it has been tested Rocky linux, Fedora 34
Systems where it works Any linux system

Prerrequisites

Permission Assignment 
Create a policy in JSON like the following:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "VisualEditor0",
           "Effect": "Allow",
           "Action": [
  			   "s3:ListAllMyBuckets",
               "s3:GetBucketLocation"
           ],
           "Resource": "*"
       }
   ]
}

The previous policy must be assigned to a new user.

image.png

The user who has these permissions will be able to access AWS S3 monitoring through this plugin.

Parameters and configuration

Parameters

--conf Path to configuration file

Configuration file (--conf)

agents_group_name = < Name of the target group for the created agents >
interval = < Interval in seconds for agents and for metric analysis >
threads = < Number of execution threads, each zone/instance will be equally distributed in the number of threads >
transfer_mode = < Transfer mode, tentacle or local >
tentacle_ip = < IP of the target machine for the created agents >
tentacle_port = <tentacle port, default: 41121>
tentacle_opts = < Tentacle client additional options >
data_dir = < (Only activated if the transfer_mode is local) Destination path for the XML of each agent, by default "/var/spool/pandora/data_in/" >

s3_monitoring = < Enable with 1 to enable widespread bucket monitoring >
size_monitoring = < Enable with 1 to enable bucket size monitoring >
items_monitoring = < Activate with 1 to enable monitoring of the number of objects in the buckets >

stats_agent = < Activate with 1 to enable a global agent that will monitor based on the task created and the parameters used >
stats_agent_name = < Name for the agent that is activated with the "stats_agent" parameter. If you do not use and "stats_agent" is enabled, the agent will be called "Aws" by default >

aws_regions = < List of regions to monitor (when you mark a region to monitor, it automatically monitors all buckets found within that region) >
aws_buckets = < List of buckets to monitor >

creds_b64 = < Base64 credentials in the JSON file to authenticate >

Example

agents_group_name  = Aws
interval           = 300
threads            = 4
transfer_mode      = tentacle 
tentacle_ip        = 172.42.42.101
tentacle_port      = 41121
data_dir           = /var/spool/pandora/data_in/

s3_monitoring      = 1
size_monitoring    = 1
items_monitoring   = 1

stats_agent        = 1
stats_agent_name   = AwsCloud

aws_regions        = ["us-east-1","us-east-2","us-west-1","us-west-2","ca-central-1","eu-central-1","eu-west-1","eu-west-2","eu-west-3","ap-northeast-1","ap-northeast-2","ap-southeast-1","ap-southeast-2","ap-south-1","sa-east-1"]
aws_buckets        = ["Bucket-1","Bucket-2","Bucket-3","Bucket-4"]

creds_b64          =  ewdhBDJDdvb2tleV9pZGdhjDNDHDhbdjdKKDNDbdBiwKInNlY3JldFSHSHHDGJCJChfDHCNCNHCdjdghDMDBGBkxlSLiIKfQ==

 

 

Manual execution

The plugin execution format is as follows:

./pandora_aws_s3 --conf < path to configuration file >

For example:

./pandora_aws_s3 --conf /usr/share/pandora_server/util/plugin/aws_s3.conf

The execution will return an output in JSON format with information about the execution, and will generate an XML file for each agent in each monitored region that will be sent to the Pandora FMS server by the transfer method indicated in the configuration.

For example:

{"summary": {"Total buckets": 32, "Zones with buckets": 7}}

 

Discovery

This plugin can be integrated with Pandora FMS Discovery.
 
To do this, you must load the ".disco" package that you can download from the Pandora FMS library:

https://pandorafms.com/library/

image-1687944718222.png

Once uploaded, Amazon S3 environments can be monitored by creating Discovery tasks from the Management > Discovery > Cloud section.

image-1687944820131.png

For each task, the following minimum data will be requested:

image-1687440444917.png

If the credentials provided are correct and the Pandora FMS server is able to connect to the AWS API, you will be able to see a tree with AWS S3 zones and buckets, which can be marked for monitoring.

If a zone is selected, in addition to the zone itself, all the buckets it contains will be monitored (both at the time of configuring the task and later if new buckets are included).

If specific buckets are selected they will be monitored regardless of whether their zones have not been selected.

image-1687250170226.png

Finally, you can adjust the monitoring you want to obtain:

image-1687250201341.png

Tasks that are successfully completed will have an execution summary with the following information:

image-1687250234469.png

The tasks that are not completed successfully will have an execution summary recording the errors produced.

Agent and modules generated by the plugin

Running the plugin will create the following agents and modules:

< Name used with the parameter "stats_agent_name" or failing that "Aws" >

Modules

AWS S3 Buckets count
Total buckets registered in AWS

Modules

summary.Aws.ec2.buckets
 Total buckets registered in AWS

For each bucket monitored in the region, the following modules will be created:

bucket.items <Bucket name> Number of objects in the bucket
bucket.size <Bucket name> bucket size