Kubernetes

This document describes the Kubernetes functionality of PandoraFMS discovery.

Introduction

The purpose of this plugin is to monitor Kubernetes nodes and pods, using key metrics regarding the CPU, memory, status of these and the containers of a pod, which are essential to control and track these resources.

The plugin connects to the Kubernetes REST API and monitors the environment using the aforementioned metrics, generating an agent for each zone and instance via XML that is sent to the Pandora server.

Prerrequisites

Permission Assignment

The steps to follow to create a user with sufficient permissions to obtain the monitoring data are described below. 

1.Creation of a reading "Cluster role". A .yaml file must be created with the following content :.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployment-reader
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metrics-reader
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["*"]
  verbs: ["get", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: namespace-reader
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: component-status-reader
rules:
- apiGroups: [""]
  resources: ["componentstatuses"]
  verbs: ["get", "list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-reader-binding
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: pod-reader-binding
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: ClusterRole
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: deployment-reader-binding
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: ClusterRole
  name: deployment-reader
  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-reader-binding
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: ClusterRole
  name: metrics-reader
  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: namespace-reader-binding
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: ClusterRole
  name: namespace-reader
  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: component-status-reader-binding
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
roleRef:
  kind: ClusterRole
  name: component-status-reader
  apiGroup: rbac.authorization.k8s.io
  
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list"]
- nonResourceURLs: ["/healthz", "/healthz/ping", "/healthz/log", "/healthz/etcd", "/healthz/poststarthook/crd-informer-synced", "/healthz/poststarthook/generic-apiserver-start-informers", "/healthz/poststarthook/start-apiextensions-controllers", "/healthz/poststarthook/start-apiextensions-informers"]
  verbs: ["get"]
  
---
    
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: service-reader
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/api/v1/services"]
  verbs: ["get"]
  
---
  
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: service-reader-binding
subjects:
- kind: ServiceAccount
  name: default
  namespace: default  
roleRef:
  kind: ClusterRole
  name: service-reader
  apiGroup: rbac.authorization.k8s.io

2. The cluster role must then be applied:

kubectl apply -f file.yaml

To install the “metrics-server” addon, it will be necessary to download it in the Kubernetes environment. You can get it from their github project:

https://github.com/kubernetes-incubator/metrics-server

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

You can verify that the deployment has been successful if when you launch the following command the status of your API appears as “​true​”:

kubectl get apiservices | grep metrics-server
v1beta1.metrics.k8s.io    kube-system/metrics-server True 1m

If the deployment is successful, the metrics-server API should be accessible:

​kubectl get --raw "/apis/metrics.k8s.io/v1beta1/"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","res
ources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics
","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"P
odMetrics","verbs":["get","list"]}]}

Additionally, after a few seconds, you should be able to obtain CPU and memory usage data for containers and nodes:

$ ​kubectl top node
NAMECPU(cores) CPU%MEMORY(bytes)MEMORY%
kube93m968Mi56%4%
$ ​kubectl top pod
NAMECPU(cores)MEMORY(bytes)
pod10m3Mi
pod20m3Mi
pod30m2Mi
pod40m3Mi
pod50m1Mi


Parameters and configuration

Parameters

--conf Path to configuration file

Configuration file (--conf)

ip = < IP or Kubernetes host >
port = <Kubernetes host port>
token = <Authentication Bearer Token>
connection = < Connection mode, normal or proxy >
proxy = <connection url>
interval = < Interval in seconds for agents and for metric analysis >
agent_group_name = < Name of the target group for the created agents >
prefix = < >
transfer_mode = < Transfer mode, tentacle or local >
data_dir = < (Only activated if the transfer_mode is local) Destination path for the XML of each agent, by default "/var/spool/pandora/data_in/" >
tentacle_ip = < IP of the target machine for the created agents >
tentacle_port = <tentacle port, default: 41121>

deployments = < Enable with 1 to enable deployment monitoring >
nodes = < Enable with 1 to enable nodes monitoring >
pods = < Enable with 1 to enable pod monitoring >

Example

ip               = 192.168.49.2
port             = 8443
token            = eyJhbGciOiJSUzI1NiIsImtpZCI6IkVpeVhfTFFzSWpmcGtjZzM0blZrZDY3YUtMbzNqTWZmcmdLZ2NKblVHbTgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50IiwisInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.aC1lPbS5kYR9Fg9hXJaPiQ8cHzjdWlaAfXdPlmPok6LRX0_OZESEhw8to4PZrz2vzt_BUbawUR0NOpHadujHIx7as4Jm8UCRHgmGseyB7mae9vRMFiD1B4EeH5L8lvucydlV2Avx1IV9SyssbypwGCH_jg7tfKz4EMz5aBfgipliRwFaRGdQMX5sjz4AknGxcb2UVOe9LFb_xC-awDEjaGvm_F0tSLxtyGPoJmsvknNmEC9hJLG_AU8Z9Ke5I5oTvEzRmUCqrKz6319p-x9aaF6yUfOFtlnQlLhIz2xkOs8DkkJ1Rk59i2MolzXGwmlGgWkgc7qCgvqXQNjDeX3ySA
connection       = normal
proxy            = http://localhost:8080
interval         = 300
agent_group_name = kubernetes
prefix           = kube.
transfer_mode    = tentacle
data_dir         = /var/spool/pandora/data_in
tentacle_ip      = 172.42.42.101
tentacle_port    = 41121

deployments      = 1
nodes            = 1
pods             = 1

 

 

Manual execution

The plugin execution format is as follows:

./pandora_kubernetes --conf < ruta al fichero de configuración >

For example :

./pandora_kubernetes --conf /usr/share/pandora_server/util/plugin/kubernetes.conf

The execution will return an output in JSON format with information about the execution, and will generate an XML file for each monitored agent that will be sent to the Pandora FMS server by the transfer method indicated in the configuration.

For example:

{"summary": {"Total agents": 18, "Nodes agents": 1, "Pods agents": 12, "Deployments agents": 4}}

 

Discovery

This plugin can be integrated with Pandora FMS Discovery.

To do so, you must load the ".disco" package that you can download from the Pandora FMS library:

https://pandorafms.com/library/

image.png

Once loaded, you will be able to monitor Kubernetes environments by creating Discovery tasks from the Management > Discovery > App section.

The following minimum data will be requested for each task:

image.png

You can also adjust the task settings to customize the desired monitoring:

image.png

Successfully completed tasks will have an execution summary with the following information:

image.png

Agents and modules generated by the plugin

Running the plugin will create the following agents and modules:

< prefix.Kubernetes >

Modules

Api status
< Endpoint healthz>
Services
Namespaces
< Component status healthz >
Deployments
< prefix. Deployment name>

Modules

Replicas
Updated replicas
Ready replicas
Avalaible replicas
Unavalaible replicas
Available
Progressing
< prefix. Name node >

Modules

Pods
Pods (%)
CPU (cores)
CPU (%)
Memory (bytes)
Memory (%)
Condition < condition > status

< prefix. Pod name >

Modules

Pod status
<Condition status>
Containers
Container <container name> CPU (cores)
Container <container name> CPU (%)
Container <container name> Memory (bytes)
Container <container name> Memory (%)