Pandora: Documentation en: Netflow
- 1 Netflow
- 1.1 Introduction
- 1.2 Installation and requisites
- 1.3 Working with Netflow in Pandora FMS
- 1.4 Filters
- 1.5 Reports
- 1.6 Netflow live view
From versión 5.0, Pandora FMS is able to monitor the IP traffic using the NetFlow protocol. It allows to show patterns and general data of the traffic that are very useful.
Netflow is a network protocol developed by Cisco Systems to collect IP traffic information. Netflow has become an industry standard for network traffic monitoring, and is currently supported by several platforms besides Ciso IOS and NXOS, like Juniper devices, Enterasys Switches and operating systems like Linux, FreeBSD, NetBSD and OpenBSD.
1.1.1 Net flow
Netflow capable devices (netflow probes) generate netflow records consisting of small chunks of information that are sent to a central device or netflow server (netflow collector), which stores and processes that information.
Data is transmitted using the Netflow protocol via UDP o SCTP. A netflow record is a small packet that only contains statistical information about a connection, not the whole raw data or the payload.
There are several Netflow implementations that may differ from the original specification and include additional information, but most of them provide at least the following:
- Source IP address.
- Target IP address.
- Source UDP or TCP port.
- Target UDP or TCP port.
- IP protocol.
- Interface (SNMP ifIndex)
- Type of service.
With time, some manufacturers have designed similar protocols with different names but the same purpose:
- Jflow o cflowd de Juniper Networks
- NetStream de 3Com/H3C|HP
- NetStream de Huawei
- Cflowd de Alcatel Lucent
- Rflow de Ericsson
1.1.2 Netflow collector
A Netflow collector is a device (PC or server) placed in a network to gather all the Netflow information sent by routers and switches.
A Netflow server is needed to receive and store that information. Pandora FMS uses nfcapd for this purpose, and it must be installed before Pandora FMS can process Netflow data. Pandora FMS starts and stop this server automatically as needed.
1.1.3 Netflow probe
Probes are usually Netflow capable routers configured to send Netflow data to the Netflow collector (in our case, a Pandora FMS server with nfcapd running).
1.2 Installation and requisites
Pandora FMS uses an OpenSource tool callednfcapd to process all the netflow traffic. This daemon is automatically started by Pandora FMS Server. This system stores the data in binary files, in an specific location. You should install nfcapd in your system before working with Netflow, nfcapd by default listen in the port 9995/UDP. Please, consider this if you have firewalls to open this port.
1.2.1 Installation of nfcapd
nfcapd must be manually installed. Pandora FMS will not install nfcapd. For more information on how to install it go to the nfcapd project official page.
Pandora FMS by default uses the directory /var/spool/pandora/data_in/netflow to store Netflow data. nfcapd will point to this directory when started by the Pandora FMS Server. Do not change it unless you know what you are doing.
Pandora FMS needs nfdump version 1.6 to process Netflow data.
To manually test your nfcapd installation run:
nfcapd -l /var/spool/pandora/data_in/netflow -D
Bear in mind that the Pandora FMS Console, and more specifically the Web server that hosts it, needs access to /var/spool/pandora/data_in/netflow to read Netflow data files.
1.2.2 Netflow probe installation
If a Netflow capable router is not available, but you use a Linux server to route your traffic, you can install a software Netflow probe that sends netflow information to the Netflow server.
In Linux there is available a program called fprobe that gets the traffic and send it to a NetFlow server.With this program you can generate Netflow traffic that goes through its interfaces,i.e:
/usr/sbin/fprobe -ieth0 -fip 192.168.70.185:9995
Once the traffic has been generated, you could see stadistics of this traffic in the command:
nfdump -S -R /home/netflow_data/
That should show you information similar to the following one:
Aggregated flows 1286 Top 10 flows ordered by packets: Date flow start Duration Proto Src IP Addr:Port Dst IP Addr:Port Packets Bytes Flows 2011-12-22 20:41:35.697 901.035 TCP 192.168.60.181:50935 -> 192.168.50.2:22 2105 167388 4 2011-12-22 20:41:35.702 900.874 TCP 192.168.50.2:22 -> 192.168.60.181:50935 1275 202984 4 2011-12-22 20:48:15.057 1.347 TCP 184.108.40.206:80 -> 192.168.50.15:40044 496 737160 1 2011-12-22 20:48:14.742 1.790 TCP 220.127.116.11:80 -> 192.168.50.15:60101 409 607356 1 2011-12-22 20:46:02.791 76.616 TCP 192.168.50.15:80 -> 192.168.60.181:40500 370 477945 1 2011-12-22 20:48:15.015 1.389 TCP 192.168.50.15:40044 -> 18.104.22.168:80 363 22496 1 2011-12-22 20:46:02.791 76.616 TCP 192.168.60.181:40500 -> 192.168.50.15:80 303 24309 1 2011-12-22 20:48:14.689 1.843 TCP 192.168.50.15:60101 -> 22.214.171.124:80 255 13083 1 2011-12-22 20:48:14.665 1.249 TCP 126.96.36.199:80 -> 192.168.50.15:38476 227 335812 1 2011-12-22 20:48:21.350 0.713 TCP 188.8.131.52:80 -> 192.168.50.15:47551 224 330191 1 Top 10 flows ordered by bytes: Date flow start Duration Proto Src IP Addr:Port Dst IP Addr:Port Packets Bytes Flows 2011-12-22 20:48:15.057 1.347 TCP 184.108.40.206:80 -> 192.168.50.15:40044 496 737160 1 2011-12-22 20:48:14.742 1.790 TCP 220.127.116.11:80 -> 192.168.50.15:60101 409 607356 1 2011-12-22 20:46:02.791 76.616 TCP 192.168.50.15:80 -> 192.168.60.181:40500 370 477945 1 2011-12-22 20:48:14.665 1.249 TCP 18.104.22.168:80 -> 192.168.50.15:38476 227 335812 1 2011-12-22 20:48:21.350 0.713 TCP 22.214.171.124:80 -> 192.168.50.15:47551 224 330191 1 2011-12-22 20:48:15.313 1.603 TCP 126.96.36.199:80 -> 192.168.50.15:52019 212 313432 1 2011-12-22 20:48:14.996 1.433 TCP 188.8.131.52:80 -> 192.168.50.15:36940 191 281104 1 2011-12-22 20:51:12.325 46.928 TCP 192.168.50.15:80 -> 192.168.60.181:40512 201 245118 1 2011-12-22 20:52:05.935 34.781 TCP 192.168.50.15:80 -> 192.168.60.181:40524 167 211608 1 2011-12-22 20:41:35.702 900.874 TCP 192.168.50.2:22 -> 192.168.60.181:50935 1275 202984 4 Summary: total flows: 1458, total bytes: 5.9 M, total packets: 15421, avg bps: 49574, avg pps: 15, avg bpp: 399 Time window: 2011-12-22 20:40:46 - 2011-12-22 20:57:21 Total flows processed: 1458, Records skipped: 0, Bytes read: 75864 Sys: 0.006s flows/second: 208345.2 Wall: 0.006s flows/second: 221177.2
If you have this system working, the following thing will be to configure Pandora FMS to it could use this configuration.
1.3 Working with Netflow in Pandora FMS
Pandora FMS does not store Netflow data in its database, information is processed on demand to render reports.
Pandora FMS works with Netflow data using filters, which are sets of rules that match certain traffic patterns. A rule can be as simple as 'all the traffic from the 192.168.70.0/24 subnet', or it can be a complex pcap expression.
Once filters are created, we have to define reports that determine how the information matched by those filters is going to be displayed (charts, tables...) and the time frame. Netflow reports can be accessed on demand like any other Pandora FMS reports.
Una vez definidos los filtros, definiremos los informes, que son, como queremos ver los datos (graficos, listas...) y en que intervalo de tiempo. Al definir filtros e informes, dejamos definida esa información, de forma similar a como se opera con los informes de Pandora, para utilizarla -bajo demanda- cuando queramos.
Los informes Netflow aparecerán también como "tipo de informe" en la seccion de Informes generales de Pandora, para poderlos "incorporar" también a los informes "normales" de Pandora.
There is also a live Netflow viewer to analyze traffic, modify and create rules on the spot. It can be very useful to investigate problems or temporarily display a chart that we don't want to save for later usage.
1.3.1 Enterprise version: Analysis in Pandora
The Enterprise version allows you to store average traffic values for any filters as Pandora FMS modules. This allows you to configure alerts, generate combined charts or work with it as with any other Pandora FMS module. For more information see section Netflow monitoring with Prediction server
First of all, we have to authorize Netflow to it would be accessible from the Operation and Administration menus.
In the Administration menu, in the Configuration chapter, we find the Netflow option in which we specify the path in which are the files captured of the Netflow traffic. For example:/tmp/netflow. It is also important to verify that the path to the nfcapd daemon is correct.
The following configuration options are available:
- Data storage path: Directory where netflow data files will be stored.
- Daemon interval: Time interval in seconds after which data files are rotated.
- Daemon binary path: Path to the nfcapd binary.
- Nfdump binary path: Path to the nfdump binary.
- Maximum chart resolution: Maximum number of points that a netflow area chart will display. The higher the resolution the lower the performance. Values between 50 and 100 are recommended.
Once netflow configuration is enabled, Pandora FMS server must be restarted to start nfcapd server. This server must be accesible on system path and properly installed. Check server logs on any doubt. This server will not appear in Pandora FMS server view because it is not a Pandora server.
The creation and edition of filters is at "Administración / Netflow filters".In this snapshot there is a list of the filters already created and that can be changed and deleted.
The Netflow filters allows to define some features that we are going to explain now.
- Name: It's advisable that the name of the filter would be descriptive.
- Group:An user could only create one filter or edit one filter of one group to it has access to.
- Filter: There are two types of filters, basic and advanced. Advanced filters allow using advanced expressions in the same format as nfdump. Basic filters can filter traffic by source Ip, destination Ip, source port or destination port. Lists of comma separated Ips or ports are accepted.
- Aggregate by: Traffic data can be grouped by one of this fields:
IP Origin: shows the traffic for each IP of different origin
IP Destination: shows the traffic for each IP of different destination
Origin Port: shows the traffic for each port of different origin Destiny Port: shows the traffic for each port of different destination
Protocol: shows the traffic for each different protocol.
Any: (the data will be total).
- Output format: Data will be displayed in the chosen unit:
Bytes per second.
Kilobytes per second.
Megabytes per second.
Basic web traffic filter example:
Advanced intranet traffic filter example:
Here are other examples of advanced filters:
- Capture traffic to or from 192.168.0.1:
- Capture traffic to 192.168.0.1:
dst host 192.168.0.1
- Capture traffic from 192.168.0.0/24:
src net 192.168.0.0/24
- Capture HTTP and HTTPS traffic:
(port 80) or (port 443)
- Capture all traffic except DNS:
port not 53
- Capture SSH traffic to 192.168.0.1:
(port 22) and (dst host 192.168.0.1)
Netflow reports are integrated with Pandora FMS reports (see Reports for more information).
To create a report item, choose one of the available netflow report items.
And configure it. The following options are available:
- Type: Item types will be explained below.
- Filter: Netflow filter to use.
- Description: Item description.
- Period: Length of the interval of data to display.
- Resolution: Data will be retrieved in blocks of size equal to the resolution. If Period / Resolution is bigger than the configure maximum chart resolution the resolution will be dynamically readjusted. For example, for a period of 1 day and a resolution of 1 hour 24 points will be drawn in the chart.
- Max. values: Maximum number of elements for aggregates. For example, if a chart of HTTP traffic is drawn aggregated by source IP address and Max. values is set to 5, only 5 IP addresses will be shown.
There are five types of netflow report items:
- Area chart: An area chart, either aggregated or unaggregated.
- Pie chart: An aggregated pie chart.
- Data table: A text representation of the area chart.
- Statistics table: A text representation of the pie chart.
- Summary table: Traffic summary for the given period.
1.6 Netflow live view
Filters can be visualized live from "Operation / Netflow Live View". This tool allows you to preview changes made to a filter and save it when the desired result is achieved. It is also possible to load and modify already existing filters.
To modify an existing filter load if from the Load filter selector, make the desired changes and click on Update current filter.
To create a new filter, configure it, click on Save as new filter, enter a name and optionally select a group and click on Save as new filter again.