Network Monitoring with NetFlow and sFlow

Pandora FMS uses a tool to analyze the network in real time: NetFlow® and sFlow®. It uses the principle of “listening” through Ethernet continuously and analyzing traffic to generate statistics.

To intercept network traffic and be able to analyze it, physical access to that network is needed, since the network capture point must be the most appropriate. To capture such data, traffic must be redirected from one switch port to another port using a “port-mirror”. Not all network devices allow this (only mid/high-range ones). A port-mirror can also be done on some commercial firewall devices. It is the easiest way to intercept traffic and requires no additional hardware. By sending all traffic to one port, that port connects directly to the network analyzer (probe).

These high-range switches and/or firewalls allow monitoring to be performed more easily. This is because these devices send network flow statistical information directly to the Pandora FMS collector without the need for an independent probe. Hardware specifications should be consulted to see if NetFlow and/or sFlow can be enabled and flows sent to an independent collector (in this case, the Pandora FMS collector).

Pandora FMS is capable of monitoring IP traffic using the NetFlow protocol.

NetFlow® is a network protocol developed by Cisco Systems® and is currently supported for several platforms in addition to Cisco IOS® and NXOS®, such as devices from manufacturers like Juniper®, Enterasys Switches®, and in operating systems like Linux®, FreeBSD®, NetBSD®, and OpenBSD®.

NetFlow-enabled devices, when this feature is activated, generate “NetFlow records” consisting of small fragments of information that they send to a central device (a NetFlow server or collector), which receives information from the devices (NetFlow probes) to store and process it.

This information is transmitted using the NetFlow protocol, based on UDP or SCTP. Each NetFlow record is a small packet containing a minimum amount of information, but in no case does it contain raw traffic data. That is, it does not send the payload of the traffic flowing through the collector, only the statistical data.

The traditional Cisco definition is to use a 7-element key:

• Source IP address.
• Destination IP address.
• Source UDP or TCP port.
• Destination UDP or TCP port.
• IP protocol.
• Interface (SNMP ifIndex).
• IP Type of Service.

Over time, other manufacturers have designed equivalent systems for their network devices, with different names but similar purposes:

• Jflow or cflowd by Juniper Networks®.
• NetStream by 3Com/H3C|HP®.
• NetStream by Huawei®.
• Cflowd by Alcatel Lucent®.
• Rflow by Ericsson®.
• AppFlow®.
• sFlow®.

This is a device (PC or server) located on the network to collect all the NetFlow information sent from routers and switches.

NetFlow generates and collects this information, but software is needed to store and analyze said traffic. With Pandora FMS, a special server is used for this purpose, which Pandora FMS will start and stop when the PFMS Server is started. This server is called nfcapd and it must be installed to use NetFlow monitoring.

Probes (such as Raspberry Pi) are generally routers with NetFlow enabled, configured, and sending information to the NetFlow collector (which in this case will be the Pandora FMS server with the nfcapd daemon enabled).

Pandora FMS uses an OpenSource tool called nfcapd (belonging to the nfdump package) to process all NetFlow traffic. This daemon is automatically started by the Pandora FMS Server. This system stores data in binary files in a specific location. nfcapd must be installed on the operating system before working with NetFlow in Pandora FMS.

The nfcapd daemon defaults to listening on port 9995/UDP, so it must be taken into account if firewalls are enabled to open this port number and when configuring NetFlow probes.

The installation of nfcapd must be done manually, as Pandora FMS will not install it. For more information, go to the official nfcapd project page.

Pandora FMS by default uses the /var/spool/pandora/data_in/netflow directory to process information, so when nfcapd starts, it will use this directory. Avoid modifying this location path, unless strictly necessary and with full knowledge of it.

If you want to check that nfcapd is correctly installed, run the following command to start the process in the foreground:

nfcapd -l /var/spool/pandora/data_in/netflow

If everything goes well, you should get an output similar to this:

 Add extension: 2 byte input/output interface index
 Add extension: 4 byte input/output interface index
 Add extension: 2 byte src/dst AS number
 Add extension: 4 byte src/dst AS number
 Add extension: 4 byte output bytes
 Add extension: 8 byte output bytes
 Add extension: NSEL Common block
 Add extension: NSEL xlate ports
 Add extension: NSEL xlate IPv4 addr
 Add extension: NSEL xlate IPv6 addr
 Add extension: NSEL ACL ingress/egress acl ID
 Add extension: NSEL username
 Add extension: NSEL max username
 Add extension: NEL Common block
 Bound to IPv4 host/IP: any, Port: 9995
 Startup.
 Init IPFIX: Max number of IPFIX tags: 62

/var/spool/pandora/data_in/netflow

If a router with NetFlow is not available, but its traffic passes through a Linux system, software can be installed that acts as a probe and sends NetFlow traffic information to the collector.

The fprobe software captures traffic and forwards it to a NetFlow® server. With it, you can generate NetFlow traffic from all the network traffic passing through its interfaces.

In Ubuntu 22 it is installed and configured with:

sudo apt install fprobe

The installation offers eth0 as the default interface; it must be configured with a valid network interface; in recent distros, the Ethernet interface is named enp1s0 and the wireless network interface wlp2s0, these names may vary depending on the hardware.

Next, it will ask for the IP address and the listening port number of the device that will collect the data (in this case the PFMS server), in a format similar to 192.168.70.185:9995.

Once the installation is finished, operation can be checked:

sudo systemctl status fprobe

And it will return a response similar to this:

To enable it on every system startup:

sudo systemctl enable fprobe

To read the complete fprobe help:

https://manpages.ubuntu.com/manpages/resolute/man8/fprobe.8.html

Once traffic has been generated, its statistics can be seen in the PFMS Server NetFlow collector with the command:

nfdump -R /var/spool/pandora/data_in/netflow

Among many features of the pmacct probe are the ability to work with NetFlow v1/v5/v7/v8/v9 and sFlow v2/v4/v5 over IPv4 and IPv6.

The source code is hosted at:

Install dependencies with administrator rights:

Rocky Linux 8

dnf config-manager --set-enabled powertools
dnf groupinstall 'Development Tools'
dnf install libpcap libpcap-devel

Rocky Linux 9

dnf config-manager --set-enabled crb
dnf groupinstall 'Development Tools'
dnf install libpcap libpcap-devel

Download the pmacct source code (you can use curl instead of wget) and compile:

cd /tmp
wget -O pmacct-1.7.7.tar.gz \ 
  "https://github.com/pmacct/pmacct/releases/download/v1.7.7/pmacct-1.7.7.tar.gz"
tar xvzf pmacct-1.7.7.tar.gz
cd pmacct-1.7.7
./autogen.sh
./configure
make && make install

Start pmacct as a NetFlow probe in daemon mode:

• Create a configuration for pmacct.

In this particular case, all traffic from the eth0 interface will be sent to the NetFlow collector listening on port number 9995 of the IP address 192.168.70.185:

cat> ~/pmacctd_probe.conf <<EOF
daemonize: true
pcap_interface: eth0
aggregate: src_host, dst_host, src_port, dst_port, proto, tos
plugins: nfprobe
nfprobe_receiver: 192.168.70.185:9995
nfprobe_version: 9
EOF

• Start pmacctd:

/usr/local/sbin/pmacctd -f ~/pmacctd_probe.conf

Pandora FMS works with NetFlow as an auxiliary system, that is, it does not store NetFlow information in the database. Pandora FMS displays this information as reports requested on demand.

Pandora FMS operates with NetFlow using sets of rules to visualize specific traffic. These rules can be something as simple as “All traffic from the network 192.168.70.0/24” or something more complex using pcap expressions.

Once filters are defined, reports are defined that determine how to visualize the data (charts, lists, …) and the time interval. By defining filters and reports, that information remains defined, similar to how it works with Pandora FMS reports, to be used on demand when needed. NetFlow reports will also appear as a “report type” in the Pandora FMS Custom Reports section, so they can be incorporated into normal Pandora FMS reports.

On the other hand, there is a “real-time” view console to analyze traffic, composing the rules directly. It is useful for investigating problems, seeing specific charts that do not correspond to a certain filter, etc.

First, NetFlow must be enabled so that it is accessible from the Operation and Administration menus. In the Configuration section, there is an option to enable or disable NetFlow globally:

Once activated, a new NetFlow configuration option will appear in the configuration section:

This section must be configured correctly so that the nfcapd daemon can start without problems along with the Pandora FMS Server:

• Data storage path: Directory where NetFlow data files will be stored. Only the directory name should be placed, by default netflow (see General Setup).
• Daemon binary path: Path to the nfcapd binary.
• Nfdump binary path: Path to the nfdump binary.
• Nfexpire binary path: Path to the nfexpire binary.
• Maximum chart resolution: Maximum number of points that a NetFlow area chart will display. The higher the resolution, the worse the performance. Values between 50 and 100 are recommended.
• Disable custom live view filters: Disables the definition of custom filters from the NetFlow view (already created filters can still be used).
• Max. NetFlow lifespan: Indicates the maximum time in days of NetFlow data that will be stored.
• Enable IP address name resolution: Allows IP address resolution to try to obtain the hostnames of NetFlow devices.
• NetFlow interval: Allows adjusting the NetFlow daemon time interval to 10, 30, or 60 minutes. After making a change and applying it in the time selector, it is necessary to restart the server for this change to take effect.

Once NetFlow is configured in the console, the PFMS Server will need to be restarted so that it starts the nfcapd server. This must be correctly installed before attempting to start it. Check the server logs in case of any doubt.

If you decide to store NetFlow data on a device other than the PFMS Server (see nfcapd installation procedure and the distributed configuration), the binary file /usr/bin/nfexpire must be copied to that device and the following entry added to the /etc/crontab file:

0 * * * * root yes 2>/dev/null | /usr/bin/nfexpire -e "/var/spool/pandora/data_in/netflow" -t X_days d

Where x_days is the maximum number of days of antiquity, of the NetFlow data, to be retained on said device (in this particular case the PFMS Web Console configuration, for that field, will not take effect).

The menu for creating and editing filters is found in Management → Resources → Netflow filters. In that view, there is a list of already created filters that can be modified and/or deleted.

A filter can also be created directly from the Netflow live view, saving the active filter as a new one. NetFlow filters can be basic or advanced: The difference is that the former have fixed filtering fields (source IP address, destination IP address, source port number, destination port number) and the advanced ones are defined using a pcap expression (standard in filtering expressions for network traffic) and use tools of all kinds.

When creating the filter, monitoring can be activated by enabling the Enable NetFlow monitoring token.

• This allows creating an agent that monitors the traffic volume of this filter.
• It creates a module that measures if the traffic of any IP address in this filter exceeds a certain threshold.
• A text-type module will be created with the traffic rate of each IP address within this filter every five minutes (the top 10 IP addresses with the most traffic).

The parameters are as follows:

• Maximum traffic value of the filter: Specifies the maximum rate (in bytes per second) of the filter traffic. It is subsequently used to calculate the percentage of maximum traffic per IP address.
• WARNING threshold for the maximum % of traffic for an IP: If any IP address within the filter exceeds the established percentage, the WARNING state will be generated.
• CRITICAL threshold for the maximum % of traffic for an IP: If any IP address within the filter exceeds the established percentage, the CRITICAL state will be generated.

NetFlow reports are integrated with Pandora FMS reports.

To create a report element, choose one of the available NetFlow report elements.

The following configuration options are available:

• Type: The element types will be explained below.
• Filter: NetFlow filter to use.
• Period: Length of the data interval to show.
• Resolution: Some reports require samples to be collected every certain period. This parameter serves to define the number of samples. Resolution can be low (6 samples), medium (12 samples), high (24 samples), or ultra-high (30 samples). There are two special values (hourly and daily) so that a fixed number of samples is not collected, but one every certain period of hours or days.
• Max. values: Maximum number of elements for aggregates. If an HTTP traffic chart is aggregated by source IP address and this token is configured to 5, only five IP addresses will be shown.

There are three types of NetFlow report elements:

• NetFlow area chart: An area chart, aggregated or non-aggregated.
• NetFlow data chart: A text representation of the area chart.
• NetFlow summary chart: Traffic summary for the given period. There are three elements: a table with global information, a pie chart with the most relevant IP addresses or port numbers, and a table with the same information from the pie chart broken down.

This view is used to consult the history of captured data based on different search filters. Filters and different ways of displaying information can be used. The way to group the information shown must be defined, as well as the way to obtain said information to start visualizing data.

Filters can be viewed in real time from Operation → Monitoring → Network → NetFlow Live View. This tool allows visualizing the changes made to a filter and saving it once the desired result is obtained. It is also possible to load and modify existing filters.

Information can be obtained by: source IP address, destination IP address, source port number, or destination port number. If destination IP address information is chosen to be shown, information will be displayed sorted by IP addresses with the most traffic to the destination from highest to lowest. The same would apply to knowing your network consumption by protocol, choosing by destination port number.

The possible visualization ways are as follows:

• Area graph: Displays area charts, of the stacked type, over time (from the source date to the destination date), the evolution of the data. The level of precision of the chart must be chosen in the “Resolution” token.
• Circular mesh: Displays an interactive circular mesh graph representing connection pairs between IP and traffic volume.
• Data table: Displays a data table with each IP address and a number of rows depending on the chosen resolution.
• Detailed host trafic: Shows a portion map with detailed host traffic representing traffic by IP.
• Summary: Shows a summary table, a pie chart, and a table with the data for the entire period.
• Top-N connections: A table that shows the “TOP-N” connections between Source IP address - Destination IP address pairs, based on traffic between those IP addresses (the sum of the percentages of the N elements in the table will not necessarily be one hundred because there may be other connection pairs src/dst).

Allows creating dynamic network maps based on traffic between nodes. Shows the relationship (connections) between different addresses, showing the N most important connections (by size of data transferred between them).

It is possible to locate the Pandora FMS node that collects NetFlow data on a host independent of the Web Console. In environments with a lot of NetFlow data, it is more than recommended to locate it on a server with fast disks and a fast CPU with two or more cores. For the Web Console to extract NetFlow data, it will be necessary to modify the default system configuration:

• Configure automatic SSH authentication between the user owner of the web daemon and the user capable of executing nfdump on the collector node.

To perform this configuration, follow these steps:

Only for Pandora FMS environments on EL 8

• First create the folder where the Apache2 SSH keys will be stored:

mkdir /usr/share/httpd/.ssh/

• Grant permissions to Apache2 over the created folder:

chown -R apache. /usr/share/httpd/.ssh/

• Log in to BASH with Apache2 (the terminal user will change):

su apache -s /bin/bash

• Now generate SSH keys with Apache2 user:

ssh-keygen

• Copy the keys to the target machine, this step will request the credentials of the remote machine user:

ssh-copy-id <User>@<IP_Address>

• Finally check that an SSH connection is successful to the target machine with the same user and without entering a password:

ssh <User>@<IP_Address>

From the Pandora FMS configuration, in the NetFlow section, you can fill in the Remote Settings section with the above data to access the remote NetFlow data:

As of version 770 of Pandora FMS, support for sFlow is included, which is a network protocol that is an industrial standard in hardware manufacturing for data network traffic.

The operation of sFlow in PFMS is similar to the one established with NetFlow. In case both protocols are active, data will be grouped; in any case, they will always be viewed by accessing the Operation menu in the left sidebar, and then clicking on Views → Network.

You must enable sFlow so that it is accessible from the Operation and Management menus. In the NetFlow configuration section, there is an option to enable or disable sFlow globally.

A new tab will be enabled specifically for sFlow:

• Data storage path: Directory where sFlow data files will be stored (see General Setup).
• Daemon binary path: Path to the nfcapd binary.
• Nfdump binary path: Path to the nfdump binary.
• Nfexpire binary path: Path to the nfexpire binary.
• Maximum chart resolution: Maximum number of points that an sFlow area chart will display. The higher the resolution, the worse the performance. Values between 50 and 100 are recommended.
• Disable custom live view filters: Disables the definition of custom filters from the sFlow view (already created filters can still be used).
• sFlow max lifetime: Indicates the maximum time in days of sFlow data that will be stored.
• Enable IP address name resolution: Allows IP address resolution to try to obtain the hostnames of sFlow devices.

←Back to Pandora FMS documentation index