Difference between revisions of "Pandora: Documentation en: Netflow"

From Pandora FMS Wiki
Jump to: navigation, search
(Introduction to real time network analysis)
(Introduction)
Line 34: Line 34:
 
= NetFlow network monitoring =
 
= NetFlow network monitoring =
  
== Introduction ==
+
== Introduction to Netflow ==
  
The Pandora FMS versions 5 and above are designed to monitor the IP traffic by using the NetFlow protocol. This protocol allows to you review the traffic's most useful patterns and general data.<br>
+
Pandora FMS version 5 and above are designed to monitor IP traffic by using the NetFlow protocol. This protocol allows to review the traffic's most useful patterns and general data.<br>
  
'NetFlow' is a network protocol, developed by Cisco Systems to collect IP traffic information. It has become an industrial standard for network traffic monitoring and is currently supported by several platforms besides Cisco's IOS and NXOS like Juniper devices, Enterasys Switches and operating systems like Linux, FreeBSD, NetBSD and OpenBSD.
+
'NetFlow' is a network protocol, developed by Cisco Systems to collect IP traffic information. It has become an industrial standard for network traffic monitoring and is currently supported by several platforms besides Cisco IOS and NXOS like Juniper devices, Enterasys Switches and operating systems like Linux, FreeBSD, NetBSD and OpenBSD.
  
 
<center>
 
<center>
Line 44: Line 44:
 
</center>
 
</center>
  
Take a look on more information about what is Netflow in our blog at https://blog.pandorafms.org/what-is-netflow/
+
There is an article about Netflow on our blog, go check it out https://blog.pandorafms.org/what-is-netflow/
  
=== NetFlow ===
+
=== NetFlow protocol ===
  
NetFlow-capable devices (NetFlow probes) are generating NetFlow records, which consist of small chunks of information which are sent to a central device or NetFlow Server (or NetFlow collector), which stores and processes that information.
+
NetFlow-enabled devices generate "NetFlow records", which consist of small pieces of information which are sent to a central device (NetFlow server or collector), which receives device information (Netflow probes), stores and processes it.
  
Data is transmitted using the NetFlow protocol via UDP or SCTP protocols. A NetFlow record is a small packet which only contains statistical information about a connection, not the whole raw data or the payload.
+
Data is transmitted using the NetFlow protocol based on UDP or SCTP protocols. A NetFlow record is a small packet that contains only statistical information about a connection, not the whole raw data. That means it does not send the traffic payload that goes through the collector, only statistical data.
  
There are several NetFlow implementations that may differ from the original specification and include additional information, but most of them provide at least the following:
+
There are several NetFlow implementations that may differ from the original specification and include additional information, but most of them provide at least the following information.
 +
Although Netflow has been described in many ways, Cisco's traditional definition is using a 7-element key, where the flow is defined as one-way sequence of packets that share the following 7 values:
  
* The source's IP address.
+
* The source IP address.
* The target's IP address.
+
* The target IP address.
* The source's UDP or TCP port.
+
* The source UDP or TCP port.
* The target's UDP or TCP port.
+
* The target UDP or TCP port.
 
* The IP protocol.
 
* The IP protocol.
 
* An interface (SNMP ifIndex)
 
* An interface (SNMP ifIndex)
 
* The type of service.
 
* The type of service.
  
With time, some manufacturers have designed similar protocols with different names but for the same purpose:
+
In time, some manufacturers have designed similar protocols with different names but for the same purpose:
  
* 'Jflow' or 'cflowd' from Juniper Networks
+
* Juniper Networks Jflow or cflowd
* 'NetStream' from 3Com/H3C/HP
+
* 3Com/H3C/HP NetStream
* 'NetStream' from Huawei
+
* Huawei NetStream
* 'Cflowd' from Alcatel Lucent
+
* Alcatel Lucent Cflowd
* 'Rflow' from Ericsson
+
* Ericsson Rflow
* 'AppFlow'
+
* AppFlow
 +
* sFlow
  
Pandora FMS also supports sFlow( Industry standard for packet export ), which allows to Pandora FMS to analyse sniffered packets at Layer 2 of the OSI model. Moreover, because sFlow is an standard, many vendors use it on their devices.
 
  
=== The NetFlow Collector ===
+
=== NetFlow Collector ===
  
A NetFlow collector is a device (a PC or a Server), placed in a network to gather all the NetFlow information which is sent by routers and switches.
+
A NetFlow collector is a device (a PC or a Server), embedded in a network to gather all NetFlow information which is sent by routers and switches.
  
A NetFlow Server is required to receive and store that information. Pandora FMS uses 'nfcapd' for this purpose, and it's required to be installed '''before''' Pandora FMS is able to process any NetFlow-related data. Pandora FMS starts and stops this server automatically in the moment the need arises.
+
NetFlow generates and collects that information, but if needs a software that allows to store and analyze said traffic. Pandora FMS uses an specific server for this purpose, that will be started and shut down when Pandora FMS starts. That server's name is '''nfcapd''' and it is necessary to install it to be able to use Netflow monitoring.
  
=== The NetFlow Probe ===
+
=== NetFlow Probe ===
  
Probes are usually NetFlow-capable routers, configured to send NetFlow data to its collector - in our case, a Pandora FMS server with 'nfcapd' running.
+
Probes are usually NetFlow-enabled routers, configured to send information to NetFlow collector (in this case Pandora FMS server with 'nfcapd' daemon running).
  
 
<center>
 
<center>
Line 87: Line 88:
 
</center>
 
</center>
  
In our blog we wrote an step-by-step technical article about how to create a Netflow probe using a 60€ RaspBerry Pi hardware, take a look at https://blog.pandorafms.org/netflow-probe-using-raspberry/
+
There is an step-by-step technical article in our blog about how to create a Netflow probe using a 60€ RaspBerry Pi hardware, take a look at https://blog.pandorafms.org/netflow-probe-using-raspberry/
  
 
== Installation and Requirements ==
 
== Installation and Requirements ==

Revision as of 01:58, 25 September 2019

Go back to Pandora FMS documentation index

1 Introduction to real time network analysis

Pandora FMS uses two alternative and complementary systems to analyze the network in real time: Pandora NTA and Netflow. Both systems use the same principle: "listening" to the ethernet cable constantly and analyzing the traffic to generate statistics. In both cases, it is necessary to "intercept" network traffic in some way to send it to a probe that analyzes it and sends those results to Pandora FMS.

In order to intercept network traffic and be able to analyze it, it is necessary to have physical access to that network or at least understand its topology, since the network capture point must be the most appropriate one. It is not the same, for example, capturing network traffic on a router or local AP, than all server network traffic just before reaching the outgoing router.


There are two possible ways to capture traffic:

  • Reroute traffic from one switch port to another one by means of a port-mirror. Not all network devices allow this (only high/medium range). You may also port-mirror some commercial firewalls. It is the easiest way to intercept traffic and requires no additional hardware. By sending all traffic to a port, that port connects directly to the network analyzer (netflow probe or pandora nta/ntop).
  • Capture traffic using a network TAP. A tap is a very simple network device that copies traffic from one port to another in one direction only (it is impossible to interfere with the network). It is a PASSIVE device that cannot be "down" or cause trouble of any kind as it is a hardware driven physical copy of network traffic. It is undetectable. There are TAPs from €12 to €900, but the principle is the same. The tap generates an output for each direction of communication, so you will need a probe that listens in two ports, or just listen to a single address.

Basic-network-tap.jpg Real example of a €12 TAP

Basic-network-tap-2.jpg Real example of a €35 TAP

If you are going to use Netflow to analyze your network only through Pandora FMS and you have a high end switch or firewall, it will be possible to monitor in a simple way. This is due to the fact that these devices allow to send network flow statistical information directly to Pandora FMS Netflow collector without using an independent probe. Check out the hardware characteristics to find out whether you can enable Netflow and send the flows to an independent Netflow collector (in this case Pandora FMS Netflow collector).

In short, this could be a working scenario to be able to analyze network traffic in real time. It would only be necessary a pair of TAPS of 12€ (or a pair of port-mirrors) and the Pandora FMS OpenSource version:

Diagram-how-to-use-a-network-tap.png

2 NetFlow network monitoring

2.1 Introduction to Netflow

Pandora FMS version 5 and above are designed to monitor IP traffic by using the NetFlow protocol. This protocol allows to review the traffic's most useful patterns and general data.

'NetFlow' is a network protocol, developed by Cisco Systems to collect IP traffic information. It has become an industrial standard for network traffic monitoring and is currently supported by several platforms besides Cisco IOS and NXOS like Juniper devices, Enterasys Switches and operating systems like Linux, FreeBSD, NetBSD and OpenBSD.

Netflow architecture.png

There is an article about Netflow on our blog, go check it out https://blog.pandorafms.org/what-is-netflow/

2.1.1 NetFlow protocol

NetFlow-enabled devices generate "NetFlow records", which consist of small pieces of information which are sent to a central device (NetFlow server or collector), which receives device information (Netflow probes), stores and processes it.

Data is transmitted using the NetFlow protocol based on UDP or SCTP protocols. A NetFlow record is a small packet that contains only statistical information about a connection, not the whole raw data. That means it does not send the traffic payload that goes through the collector, only statistical data.

There are several NetFlow implementations that may differ from the original specification and include additional information, but most of them provide at least the following information. Although Netflow has been described in many ways, Cisco's traditional definition is using a 7-element key, where the flow is defined as one-way sequence of packets that share the following 7 values:

  • The source IP address.
  • The target IP address.
  • The source UDP or TCP port.
  • The target UDP or TCP port.
  • The IP protocol.
  • An interface (SNMP ifIndex)
  • The type of service.

In time, some manufacturers have designed similar protocols with different names but for the same purpose:

  • Juniper Networks Jflow or cflowd
  • 3Com/H3C/HP NetStream
  • Huawei NetStream
  • Alcatel Lucent Cflowd
  • Ericsson Rflow
  • AppFlow
  • sFlow


2.1.2 NetFlow Collector

A NetFlow collector is a device (a PC or a Server), embedded in a network to gather all NetFlow information which is sent by routers and switches.

NetFlow generates and collects that information, but if needs a software that allows to store and analyze said traffic. Pandora FMS uses an specific server for this purpose, that will be started and shut down when Pandora FMS starts. That server's name is nfcapd and it is necessary to install it to be able to use Netflow monitoring.

2.1.3 NetFlow Probe

Probes are usually NetFlow-enabled routers, configured to send information to NetFlow collector (in this case Pandora FMS server with 'nfcapd' daemon running).

NewNetFlowApproach.png

There is an step-by-step technical article in our blog about how to create a Netflow probe using a 60€ RaspBerry Pi hardware, take a look at https://blog.pandorafms.org/netflow-probe-using-raspberry/

2.2 Installation and Requirements

Pandora FMS uses an open-source tool called 'nfcapd' to process all NetFlow traffic. This daemon is automatically started by the Pandora FMS Server. This system stores the data in binary files at a specific location. You're required to install 'nfcapd' on your system before working with NetFlow. 'nfcapd' listens on port 9995 UDP by default. Please keep in mind to open port 9995 UDP in case you have firewalls in place.

2.2.1 Installation of 'nfcapd'

You're required to install 'nfcapd' manually, because Pandora FMS is not going to install it by default. For more information on how to install it, please visit the
Official NFCAPD Project Page.

Pandora FMS uses the directory '/var/spool/pandora/data_in/netflow' by default to store all NetFlow data. The 'nfcapd' daemon is going to point to this directory when it's getting started by the Pandora FMS Server. Do not change it unless you know exactly what you're doing.

Pandora FMS requires the nfdump version 1.6.8p1 in order to process any NetFlow data properly.

In order to test your 'nfcapd' installation manually, please execute the command below.

nfcapd -l /var/spool/pandora/data_in/netflow -D

Please keep in mind that the Pandora FMS Console (and more specifically the web server which hosts it) requires access to the directory of '/var/spool/pandora/data_in/netflow' in order to read any NetFlow-related data files.

2.2.2 The NetFlow Probe Installation

If a NetFlow capable router is not available, but you use a Linux server to route your traffic, you may install a NetFlow software probe which sends all NetFlow-related information to its server.

In Linux there is a program called 'fprobe' which obtains the traffic and sends it to a NetFlow Server.

To download the rpm package you can use the following command and then install it:

wget http://repo.iotti.biz/CentOS/7/x86_64/fprobe-1.1-2.el7.lux.x86_64.rpm
yum install fprobe-1.1-2.el7.lux.x86_64.rpm

By this program you're able to generate NetFlow traffic which goes through its interfaces, e.g.:

/usr/sbin/fprobe -ieth0 -fip 192.168.70.185:9995

Once the traffic has been generated, you're able to review the traffic's statistics by entering the following command:

nfdump -R /home/netflow_data/

The above mentioned command displays information similar to the one shown below.


Aggregated flows 1286
Top 10 flows ordered by packets:
Date flow start          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2011-12-22 20:41:35.697   901.035 TCP     192.168.60.181:50935 ->     192.168.50.2:22        2105   167388     4
2011-12-22 20:41:35.702   900.874 TCP       192.168.50.2:22    ->   192.168.60.181:50935     1275   202984     4
2011-12-22 20:48:15.057     1.347 TCP       157.88.36.34:80    ->    192.168.50.15:40044      496   737160     1
2011-12-22 20:48:14.742     1.790 TCP     91.121.124.139:80    ->    192.168.50.15:60101      409   607356     1
2011-12-22 20:46:02.791    76.616 TCP      192.168.50.15:80    ->   192.168.60.181:40500      370   477945     1
2011-12-22 20:48:15.015     1.389 TCP      192.168.50.15:40044 ->     157.88.36.34:80         363    22496     1
2011-12-22 20:46:02.791    76.616 TCP     192.168.60.181:40500 ->    192.168.50.15:80         303    24309     1
2011-12-22 20:48:14.689     1.843 TCP      192.168.50.15:60101 ->   91.121.124.139:80         255    13083     1
2011-12-22 20:48:14.665     1.249 TCP     178.32.239.141:80    ->    192.168.50.15:38476      227   335812     1
2011-12-22 20:48:21.350     0.713 TCP     137.205.124.72:80    ->    192.168.50.15:47551      224   330191     1  

Top 10 flows ordered by bytes:
Date flow start          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2011-12-22 20:48:15.057     1.347 TCP       157.88.36.34:80    ->    192.168.50.15:40044      496   737160     1
2011-12-22 20:48:14.742     1.790 TCP     91.121.124.139:80    ->    192.168.50.15:60101      409   607356     1
2011-12-22 20:46:02.791    76.616 TCP      192.168.50.15:80    ->   192.168.60.181:40500      370   477945     1
2011-12-22 20:48:14.665     1.249 TCP     178.32.239.141:80    ->    192.168.50.15:38476      227   335812     1
2011-12-22 20:48:21.350     0.713 TCP     137.205.124.72:80    ->    192.168.50.15:47551      224   330191     1
2011-12-22 20:48:15.313     1.603 TCP       89.102.0.150:80    ->    192.168.50.15:52019      212   313432     1
2011-12-22 20:48:14.996     1.433 TCP     212.219.56.138:80    ->    192.168.50.15:36940      191   281104     1
2011-12-22 20:51:12.325    46.928 TCP      192.168.50.15:80    ->   192.168.60.181:40512      201   245118     1
2011-12-22 20:52:05.935    34.781 TCP      192.168.50.15:80    ->   192.168.60.181:40524      167   211608     1
2011-12-22 20:41:35.702   900.874 TCP       192.168.50.2:22    ->   192.168.60.181:50935     1275   202984     4 

Summary: total flows: 1458, total bytes: 5.9 M, total packets: 15421, avg bps: 49574, avg pps: 15, avg bpp: 399
Time window: 2011-12-22 20:40:46 - 2011-12-22 20:57:21
Total flows processed: 1458, Records skipped: 0, Bytes read: 75864
Sys: 0.006s flows/second: 208345.2   Wall: 0.006s flows/second: 221177.2  

If your system works properly, the following chapter is intended to configure Pandora FMS in order to use this particular configuration appropriately.

2.3 Working with NetFlow under Pandora FMS

Pandora FMS doesn't store NetFlow data in its database. The information is processed on demand in order to render reports.

Pandora FMS works with NetFlow data by using filters, which are sets of rules that match certain traffic patterns. A rule can be as simple as 'all the traffic from the 192.168.70.0/24 subnet' or a complex 'pcap' filter expression.

Once the filters are created, we're required to define reports that determine how the information matched by those filters is going to be displayed (e.g. charts and tables) and the time frame. The NetFlow reports can be accessed on demand like any other Pandora FMS reports.

There is also a live NetFlow Viewer to analyze the traffic, modify and create rules on the spot. It can be very useful to investigate problems or temporarily display a chart that we don't intend to save for a later usage.

2.3.1 Configuration

First of all, you're required to authorize NetFlow in order to become accessible from the 'Operation' and 'Administration' menus.

Netflow manager0.png

You can find the NetFlow option in the 'Configuration' chapter of the 'Administration' menu in which we specify the path in which the files of the Netflow traffic are captured, e.g. '/tmp/netflow'. It's also very important to determine whether the path to the 'nfcapd' daemon is appropriately specified or not.

Netflow manager.png

The configurable fields pertaining to this particular feature are the following:


Data Storage Path:
The directory in which the NetFlow data files are stored. IMPORTANT: The disk's access speed on which the NetFlow data is stored is usually the limiting performance factor.

Daemon Interval:
The time interval in seconds for the data rotation. The recommended value is '3600'. A bigger interval means potentially bigger files, which means less I/O overhead, but it also renders accessing the data for a specific time interval slower.

Daemon Binary Path:
The path to the 'nfcapd' binary.

Nfdump Binary Path:
The path to the 'nfdump' binary.

Nfexpire Binary Path:
The path to the 'nfexpire' binary. This program was designed to delete old NetFlow data.

Maximum Chart Resolution:
The maximum number of points which a NetFlow Area Chart is going to display. The higher the resolution the lower the performance. Values between '50' and '100' are recommended here.

Disable Live View Custom Filters:
If enabled, only Netflow filters previously created by an administrator can be used in the Netflow live view.

Netflow max. Lifetime:
The NetFlow data which are older than the specified number of days are going to be deleted.

Once the NetFlow configuration is enabled, the Pandora FMS Server is required to be restarted in order to be able to start the 'nfcapd' server. This server must be properly installed and accessible from the system path. Please check the server logs if you're unsure on that. This server is not going to appear in the Pandora FMS server view mode, because it isn't considered a Pandora FMS Server.

2.4 Filters

You may access the creation and edition of filters by clicking on 'Administration' and 'NetFlow Filters'. This section contains a list of already created filters which can be of course altered or deleted.


Netflow3.png



You can also create a filter directly from the "Netflow live view", saving the active filter as a new one. Netflow filters can be "basic" or "advanced". The difference is that the former have fixed filtering fields (source IP, destination IP, source port, destination port) and the advanced ones are defined by an expression pcap (standard in filtering expressions for network traffic) and use all kinds of tools.

2.4.1 Filter creation

This would be a basic editing view of a Netflow filter:

Netflow4.png




The configurable NetFlow filters pertaining to this particular feature are the following:

  • Name: It's recommended for the filter's name to be as descriptive and clear as necessary.
  • Group: A user is only able to create a filter or edit the group's filters it has access to.
  • Filter: There are two types of filters: Basic and advanced. Advanced filters allow the usage of advanced expressions in the same format as 'nfdump'. Basic filters can filter traffic by source and destination IP and source or destination port. Lists of comma-separated IPs or ports are also accepted here.
  • Aggregate by: All traffic data can be grouped by one of the following fields:

IP Origin: It displays the traffic of different origin for each IP.
IP Destination: It displays the traffic of different destinations for each IP.
Origin Port: It displays the traffic for each port of different origins.
Destiny Port: It displays the traffic for different destinations for each port.

2.4.1.1 Examples

Basic web traffic filter example:



Netflow5.png



Advanced intranet traffic filter example:




Netflow6.png



Here are other examples of advanced filters:

  • Capture traffic to or from 192.168.0.1:
host 192.168.0.1
  • Capture traffic to 192.168.0.1:
dst host 192.168.0.1
  • Capture traffic from 192.168.0.0/24:
src net 192.168.0.0/24
  • Capture HTTP and HTTPS traffic:
(port 80) or (port 443)
  • Capture all traffic except DNS:
port not 53
  • Capture SSH traffic to 192.168.0.1:
(port 22) and (dst host 192.168.0.1)

2.5 Reports

Netflow reports are integrated with Pandora FMS reports (see Reports for more information).

To create a report item, choose one of the available netflow report items.

Netflow report item types.png

And configure it. The following options are available:

Netflow report item configuration.png

  • Type: Item types will be explained below.
  • Filter: Netflow filter to use.
  • Description: Item description.
  • Period: Length of the interval of data to display.
  • Resolution: Some reports require samples to be collected every certain period. This parameter is used to define the number of samples. The resolution may be low (6 samples), medium (12 samples), high (24 samples) or ultra-high (30 samples). There are two special values (hourly and daily) so that a fixed value of samples is not collected but one every certain period.
  • Max. values: Maximum number of elements for aggregates. For example, if a chart of HTTP traffic is drawn aggregated by source IP address and Max. values is set to 5, only 5 IP addresses will be shown.

There are three types of netflow report items:

  • Area chart: An area chart, either aggregated or unaggregated.

Netflow chart area aggregated.png

  • Data table: A text representation of the area chart.

Netflow table data.png

  • Netflow summary chart: Summary of traffic for the given period. There are three elements: a table with global information, a pie chart with the most relevant IPs or ports and a table with the same information as the broken down pie chart.

Netflow9.png

2.6 Netflow live view

This view is used to consult the history of data captured based on different search filters. You can use filters and different ways of displaying information. It is necessary to define the way to group the displayed information, as well as the way to obtain this information in order to start visualizing data.

Netflow view1.png

The way to get the information can be by: Source IP, Destination IP, Source Port or Destination Port. If you choose, for example, to show the destination IP information, the information ordered by the IP's with the most traffic to the destination from highest to lowest will be shown. The same would be true for knowing the consumption of your network by protocol, choosing by destination port.

The possible ways of visualization are the following:

  • Area Graphs (stacked): show over time (from the date of origin to the date of destination), the evolution of the data. The level of precision of the graph in the "Resolution" token must be chosen.

Netflow grafico area.png

  • Summary: Displays a summary table, a pie chart and a table with data for the entire period.

Netflow grafico sumario.png

  • Detailed: Shows a map of portions that represent the IP traffic.

Netflow grafico detailed.png.png

  • Data table: Displays a data table with each IP and a number of rows depending on the chosen resolution.

Netflow datatable.png

  • Circle graph: Displays an interactive pie chart representing the pairs of connections between IP and traffic volume.

Netflow bola.png

The filters can be viewed in real time from "Operation > Netflow Live View". This tool allows you to visualize the changes that are made to a filter and save it once the desired result is obtained. It is also possible to load and modify existing filters.

See Reports and Filters to learn how to configure live view options.

2.7 Network traffic maps

This is a new features introduced in OUM 733 and will be improved in the future. It creates dynamic network maps, based on the traffic between nodes. It show you the relationship (connections) between different address, showing the top N connections (by size of data transferred between them).

Network Usage map.jpg

2.8 Distributed configuration

It is possible to locate the pandora node that collects Netflow data on a host independent from the console. In environments with a lot of Netflow data it is more than recommended to place it on a server with fast disks and a fast CPU of at least two cores. In order for Pandora console to extract Netflow data it will be necessary to modify the default configuration of the system, following the steps described below:

  • Configure automatic SSH authentication between the user who owns the web daemon and the user with the ability to run nfdump on the collector node.

For its configuration the steps below must be followed:

Enable the apache user. In order to do this, the line of the apache user in the file /etc/passwd must be modified with this configuration :

apache:x:48:48:Apache:/var/www:/bin/bash

Create the .ssh directory inside the /var/www directory and give it the correct permissions:

#mkdir /var/www/.ssh
#chown apache:apache /var/www/.ssh

Create ssh keys from the user and copy them to the server where the Netflow traffic is hosted.

#su apache
bash-4.2$ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/var/www/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /var/www/.ssh/id_rsa.
Your public key has been saved in /var/www/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:vYvl5V00E4faa14zN08ARzGUQ9IfAQJnMzkaqLAGRHI [email protected]<server>
The key's randomart image is:
+---[RSA 2048]----+
|+oE     ...*o=B+.|
|.o .   . .oo+o++ |
|  . o .   o o o+o|
|   o .   o   =  +|
|  .     S . . oo.|
|           .   +o|
|          o . o+=|
|         + + + +*|
|        . o . o .|
+----[SHA256]-----+
bash-4.2$ ssh-copy-id [email protected]<netflow_server>

Once shared, it must be verified that it is possible to access the server through the apache user without entering a password:

bash-4.2$ ssh [email protected]<netflow_server>
  • Create a script in Pandora FMS console that replaces /usr/bin/nfdump with one similar to the following
#!/bin/bash
NFDUMP_PARAMS=$(sed 's/(\(.*\))/\"\(\1\)\"/' <<< "[email protected]");

ssh [email protected]<netflow_server> "/usr/bin/nfdump $NFDUMP_PARAMS"

Give the script execution permissions:

chmod 755 /usr/bin/nfdump

Try executing the script like this:

/usr/bin/nfdump -V

It should return something similar to:

nfdump: Version: 1.6.13

3 Network monitoring with Pandora NTA

Pandora Network Traffic Analyzer (Pandora NTA) is a network traffic analysis tool designed for environments where you don't want, or can't, use Netflow to do a network analysis. It is important to highlight that they offer slightly different functionalities,'Netflow shows information (related to ports) that Pandora NTA does not show and Netflow allows a much more advanced real time interface, as well as unique features such as being able to create data modules as a result of advanced filters of PCAP expressions.

As a base for Pandora NTA/NTOP we use a fork of the NTOP project (of the version licensed as GPL2), besides the own code of the Pandora FMS project. It is in charge of collecting the data and sending them in XML format to Pandora FMS server. All that code is available in our public repository for anyone who wants it, because it is a 100% opensource functionality.

Pandora NTA uses one or several network sensors to inspect traffic and generate consumption statistics by source IP and destination IP. It does not generate specific traffic information by port or application, for that more advanced function NetFlow should be used.

Pandora NTA is a simple way to monitor your network at a low level, without investing in specialized hardware or third party tools, and incorporate this information in your existing monitoring platform.

Pandora NTA offers:

  • Detailed consumption by incoming and outgoing traffic of each local IP of the local network.
  • Detection of network problems (by generating events).
  • Specific reports on network consumption, by source IP.
  • A list of the destination IPs with the most traffic per source IP on the network.
  • Reports of local network consumption by origin, dynamic maps and search and filtering options with the accumulated data.

With the individual traffic data of each network equipment, Pandora NTA will be able to generate alerts, TopN reports and use any other Pandora FMS function, since they are saved as modules of an agent.

3.1 Architecture and functioning

Se necesitará instalar la sonda Pandora-NTOP en una máquina que tenga acceso al tráfico de la red LAN, generalmente en un servidor Linux que actúa como router o firewall, o bien redireccionando el tráfico a un puerto por medio de un port-mirror desde un switch, firewall o router. También se puede duplicar el tráfico en un sólo sentido por medio de un TAP, pero requerirá un equipamiento hardware específico.

Info.png

It's important to understand that if Pandora NTA is installed in an ordinary computer, without previously making a portmirror or connecting a TAP that redirects traffic to that machine, it will not receive all the network traffic, it will only observe the traffic generated by that machine, not the rest of the network.

 


Pandora-NTOP will listen to the traffic and will generate data without storing them in any place (it keeps them in the RAM). Pandora NTA will send the data collected by Pandora-NTOP to Pandora FMS Data Server. You can install as many Pandora NTA as you need, if there are several local networks and several listening points, you can perform a distributed deployment.

Pandora NTA also offers information related to problems in the local network in real time, since it can generate events of three types:

  • Invalid mask (Wrong netmask).
  • Sending data via port p (Host sent data to zero port).
  • Duplicate MAC Address (Duplicated mac).

Alerts of events can be created to know in real time, for example, when a duplicate MAC appears in your system. The English text strings described above should be used to create an event alert.

3.2 Installation

A tarball compatible with CentOS 7 is currently available. It is distributed compressed in a tgz, available in Pandora FMS modules library in the Pandora FMS library. To install it you have to unzip it:

tar xvzf pandora_nta.tgz

Then run the installation script inside the uncompressed directory.

cd pandora_nta_tarball
./install_pandora_nta.sh --install

To uninstall Pandora NTA you have to launch the same script in the following way:

./install_pandora_nta.sh --uninstall

If the version of Pandora FMS distributed in ISO is installed, it will already be installed in the system, only having to activate it (From OUM 733).

To start it, in a CentOS7 it will be enough to execute

systemctl start pandora_nta

3.2.1 Requirements

The Perl part of Pandora NTA needs some dependencies, besides having a Perl interpreter in the machine where it runs.

Pandora NTA uses external Perl modules. Some of them are part of the Core and others are distributed in the normal installations of the interpreter. This is the list of external modules used:

  • Getopt::Std
  • Config::Simple
  • LWP::Simple
  • Sys::Hostname
  • JSON
  • POSIX
  • MIME::Base64
  • XML::Simple
  • Digest::SHA

To find out if a dependency is missing just run the main script with the -h option, and if an error appears instead of a help window it means that dependencies are missing. With this error you can see what they are and you can install them with CPAN or directly by downloading the Perl packages from the official repositories of each distribution.

From the binary part of Pandora NTA based on NTOP (Pandora-NTOP), you can obtain the code for its compilation from the public repository of github (https://github.com/pandorafms), or use one of the precompiled binaries that are distributed in the Pandora FMS modules library in (https://pandorafms.com/library). It is distributed by default in the ISO images of Pandora FMS installation from NG 733.

3.3 Pandora NTA Configuration

Parameters accepted by command line

-h: Shows help.
-f: Path of the configuration file. It is not necessary because the default configuration can be used. The user must have read permissions on the file.

Parameters of the Configuration file pandora_nta.conf

daemon

If set to 1, the program is executed in the background (0 by default).

encoding

Encoding of XML sent through Tentacle. It will go in the XML header (UTF-8 by default).

interval

Interval in seconds between two Pandora NTA work cycles (300 seconds by default).

log_file

File where to dump the application logs. It has to have writing permissions on the file and the folder that contains it. If it doesn't exist, Pandora NTA creates it automatically (by default it overturns it to STDOUT).

retries

The number of consecutive failures that the application can give before it is considered to be a serious error and stops. If it is at 0 it will never stop, no matter how many bugs there are (2 by default).

transfer_timeout

Maximum time in seconds to send the files by Tentacle. If this time is exceeded, Pandora NTA will restart all its state and memory (15 seconds by default).

verbose

Level of information dumped by the log. The higher it is, the more information it yields. With 0 nothing is shown, with 3 serious errors are shown, with 5 warnings are shown and with 9 everything is shown. Very high values are not recommended so that you do not consume too much disk (3 by default).

quiet

Doesn't show error messages.

cache_file

Folder where the cache file of the application is. It is necessary to write in this file. If it is not created, Pandora NTA creates it. You have to be careful when changing it, because you can create a new one and duplicate the agents that send information to Pandora, since it doesn't find their names and would generate new ones (* /tmp/pandorata_cache.json*).

ntop_host

Host to which to make Pandora-NTOP web server requests (by default localhost).

ntop_port

Port through which Pandora-NTOP process requests are made (default 3000).

ntop_local_subnets

Subnets that are considered local by NTOP. If you want to specify more than one subnet, you can concatenate them with commas (for example ntop_local_subnets 192.168.50.0/24,114.15.0.0/16.

Template warning.png

If there is no subnet configured it will dump information to Pandora of all the hosts that NTOP discovers (default option). Please make sure to configure at least one local network.

 


ntop_logs_to_syslog

If it's set to 1 NTOP it dumps its logs into syslog. Otherwise it doesn't dump them anywhere (0 by default).

self_name

Alias of the self-monitoring agent. Once the cache file has been created, it cannot be changed since the Data Server does not support changing the alias through XML (pandoraNTA by default).

tentacle_host

Host where is the Tentacle server that will receive the XML (by default localhost).

tentacle_port

Port where the Tentacle server is listening to receive the XML (default 41121).

temp_dir

Directory where XML is written for the Tentacle client to send them. You have to have write permissions. (default /tmp).

3.3.1 Deployment and boot

The pandora_db script deletes the NTA connection data from the history. The number of days this information remains in the system can be configured in the Performance section of the console configuration.

3.4 Visualization

By default (although this behavior can be modified in the configuration of pandora_nta.conf) an agent called Pandora NTA will be created that will contain the following metrics:

  • The state of the NTA system
  • The number of IP's discovered by the system.
  • Sum of the traffic flow of the whole network (input and output).
  • Current transfer rate of the network (input and output).
  • Packages in the network (input and output).

Nta main agent.png
Vista del nodo central NTA que resume todos los datos

In addition, the system will create an agent for each one of the IP's it finds in the local network (as defined in pandora_nta.conf).

For each managed IP, the same parameters will be monitored:

  • Sum of the traffic flow of the whole network (input and output).
  • Current network transfer rate (input and output).
  • Packages in the network (input and output).
  • MAC address associated with the IP.

Nta agent view.png
"View of an NTA agent with its data modules."

3.5 Reports

An NTA-specific report is available when creating reports:

Pandora NTA report create.png

This report shows a top-N of network consumption in the last X days of all IP analyzed by Pandora NTA:

Pandora NTA sample report 1.png

3.6 NTA explorer

Pandora has a view in which you can visualize network data provided by Pandora NTA in real time. It is a much more flexible view than reports and useful to detect network problems with a few clicks.

Nta explorer.png

This view shows the IPs with the most outgoing or incoming traffic. The top of IPs can be set by number of packets or by number of bytes. In addition, there is the possibility of filtering by a specific IP, to see the IP's to which there is traffic from that source.

For example, if there is an IP that sends a lot of data and you want to see where it sends it, just click on the filter icon next to the IP and you will see a list and a graph with the addresses that receive data from that IP. In this way, detecting the most overloaded pairs on a given date is quite simple.

3.7 NTA usage map

This view allows you to display the traffic in a certain time interval in the form of a topological map. Simply select a start date and an end date, showing the IPs with the most outgoing or incoming traffic.

Usage map.png

Go back to Pandora FMS documentation index