AWS S3
This document describes the AWS S3 functionality of PandoraFMS discovery.
- Introduction
- Compatibility matrix
- Prerrequisites
- Parameters and configuration
- Manual execution
- Discovery
- Agent and modules generated by the plugin
Introduction
This plugin aims to monitor AWS buckets, using metrics such as the number of objects and their size, which can help better track application costs and performance.
The plugin connects to the AWS API and monitors buckets using the aforementioned metrics, generating an agent for each region via XML that is sent to the Pandora server.
Compatibility matrix
Systems where it has been tested | Rocky linux, Fedora 34 |
Systems where it works | Any linux system |
Prerrequisites
- AWS API Connectivity
The plugin is a binary that contains the packaged dependencies so it is not necessary to install any type of dependency for its use. - Permissions
To make requests through the API, the account used to authenticate during execution requires certain permissions.
Permission Assignment
Create a policy in JSON like the following:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:GetBucketLocation"
],
"Resource": "*"
}
]
}
The previous policy must be assigned to a new user.
The user who has these permissions will be able to access AWS S3 monitoring through this plugin.
Parameters and configuration
Parameters
--conf | Path to configuration file |
Configuration file (--conf)
agents_group_name = < Name of the target group for the created agents >
interval = < Interval in seconds for agents and for metric analysis >
threads = < Number of execution threads, each zone/instance will be equally distributed in the number of threads >
transfer_mode = < Transfer mode, tentacle or local >
tentacle_ip = < IP of the target machine for the created agents >
tentacle_port = <tentacle port, default: 41121>
tentacle_opts = < Tentacle client additional options >
data_dir = < (Only activated if the transfer_mode is local) Destination path for the XML of each agent, by default "/var/spool/pandora/data_in/" >
s3_monitoring = < Enable with 1 to enable widespread bucket monitoring >
size_monitoring = < Enable with 1 to enable bucket size monitoring >
items_monitoring = < Activate with 1 to enable monitoring of the number of objects in the buckets >
stats_agent = < Activate with 1 to enable a global agent that will monitor based on the task created and the parameters used >
stats_agent_name = < Name for the agent that is activated with the "stats_agent" parameter. If you do not use and "stats_agent" is enabled, the agent will be called "Aws" by default >
aws_regions = < List of regions to monitor (when you mark a region to monitor, it automatically monitors all buckets found within that region) >
aws_buckets = < List of buckets to monitor >
creds_b64 = < Base64 credentials in the JSON file to authenticate >
Example
agents_group_name = Aws
interval = 300
threads = 4
transfer_mode = tentacle
tentacle_ip = 172.42.42.101
tentacle_port = 41121
data_dir = /var/spool/pandora/data_in/
s3_monitoring = 1
size_monitoring = 1
items_monitoring = 1
stats_agent = 1
stats_agent_name = AwsCloud
aws_regions = ["us-east-1","us-east-2","us-west-1","us-west-2","ca-central-1","eu-central-1","eu-west-1","eu-west-2","eu-west-3","ap-northeast-1","ap-northeast-2","ap-southeast-1","ap-southeast-2","ap-south-1","sa-east-1"]
aws_buckets = ["Bucket-1","Bucket-2","Bucket-3","Bucket-4"]
creds_b64 = ewdhBDJDdvb2tleV9pZGdhjDNDHDhbdjdKKDNDbdBiwKInNlY3JldFSHSHHDGJCJChfDHCNCNHCdjdghDMDBGBkxlSLiIKfQ==
Manual execution
The plugin execution format is as follows:
./pandora_aws_s3 --conf < path to configuration file >
For example:
./pandora_aws_s3 --conf /usr/share/pandora_server/util/plugin/aws_s3.conf
The execution will return an output in JSON format with information about the execution, and will generate an XML file for each agent in each monitored region that will be sent to the Pandora FMS server by the transfer method indicated in the configuration.
For example:
{"summary": {"Total buckets": 32, "Zones with buckets": 7}}
Discovery
This plugin can be integrated with Pandora FMS Discovery.
To do this, you must load the ".disco" package that you can download from the Pandora FMS library:
https://pandorafms.com/library/
Once uploaded, Amazon S3 environments can be monitored by creating Discovery tasks from the Management > Discovery > Cloud section.
For each task, the following minimum data will be requested:
- AWS Credentials: Connection credentials to the AWS API to be used by the task. They must be previously registered in the Pandora FMS credential store.
- Max threads: To optimize the execution time, multiple threads can be configured to monitor the agents of the task. Keep in mind that setting up multiple threads can increase the CPU usage of the task.
- Tentacle IP – Tentacle server IP to send agent data to. Normally this parameter will not be modified.
- Tentacle port: Tentacle server port to send data to. Normally this parameter will not be modified.
- Tentacle extra options: Additional options for sending data through the Tentacle client. Normally this parameter will not be modified.
- Add global stats agent: If activated, a global agent will be generated with task statistics.
- Stats agent name: It will allow you to indicate the name of the agent for global statistics of the task. If no name is provided, the default name of the plugin will be used.
If the credentials provided are correct and the Pandora FMS server is able to connect to the AWS API, you will be able to see a tree with AWS S3 zones and buckets, which can be marked for monitoring.
If a zone is selected, in addition to the zone itself, all the buckets it contains will be monitored (both at the time of configuring the task and later if new buckets are included).
If specific buckets are selected they will be monitored regardless of whether their zones have not been selected.
Finally, you can adjust the monitoring you want to obtain:
- S3 Monitoring: If activated, the task will be enabled to obtain additional monitoring for the agents, described in the following fields.
- Monitor bucket size: If activated, the bucket size modules will be included in the monitoring.
- Monitor bucket total items: If activated, the item modules of the buckets will be included in the monitoring.
Tasks that are successfully completed will have an execution summary with the following information:
- Total regions: Total number of regions monitored by the task.
- Total buckets: Total number of buckets monitored by the task.
The tasks that are not completed successfully will have an execution summary recording the errors produced.
Agent and modules generated by the plugin
Running the plugin will create the following agents and modules:
- Global agent
< Name used with the parameter "stats_agent_name" or failing that "Aws" > |
Modules
AWS S3 Buckets count
|
Total buckets registered in AWS |
- One agent for each monitored region.
Modules
summary.Aws.ec2.buckets
|
Total buckets registered in AWS |
For each bucket monitored in the region, the following modules will be created:
bucket.items <Bucket name> | Number of objects in the bucket |
bucket.size <Bucket name> | bucket size |