Pandora: Documentation en: Netflow

From Pandora FMS Wiki
Revision as of 18:57, 22 January 2013 by Rnovoa (talk | contribs) (Netflow live view)
Jump to: navigation, search

Go back to Pandora FMS documentation index

1 Netflow

1.1 Introduction

From versión 5.0, Pandora FMS is able to monitor the IP traffic using the NetFlow protocol. It allows to show patterns and general data of the traffic that are very useful.

Netflow is a network protocol developed by Cisco Systems to collect IP traffic information. Netflow has become an industry standard for network traffic monitoring, and is currently supported by several platforms besides Ciso IOS and NXOS, like Juniper devices, Enterasys Switches and operating systems like Linux, FreeBSD, NetBSD and OpenBSD.



Netflow architecture.png



1.1.1 Net flow

Netflow capable devices (netflow probes) generate netflow records consisting of small chunks of information that are sent to a central device or netflow server (netflow collector), which stores and processes that information.

Data is transmitted using the Netflow protocol via UDP o SCTP. A netflow record is a small packet that only contains statistical information about a connection, not the whole raw data or the payload.

There are several Netflow implementations that may differ from the original specification and include additional information, but most of them provide at least the following:

  • Source IP address.
  • Target IP address.
  • Source UDP or TCP port.
  • Target UDP or TCP port.
  • IP protocol.
  • Interface (SNMP ifIndex)
  • Type of service.

With time, some manufacturers have designed similar protocols with different names but the same purpose:

  • Jflow o cflowd de Juniper Networks
  • NetStream de 3Com/H3C|HP
  • NetStream de Huawei
  • Cflowd de Alcatel Lucent
  • Rflow de Ericsson
  • AppFlow
  • sFlow

1.1.2 Netflow collector

A Netflow collector is a device (PC or server) placed in a network to gather all the Netflow information sent by routers and switches.

A Netflow server is needed to receive and store that information. Pandora FMS uses nfcapd for this purpose, and it must be installed before Pandora FMS can process Netflow data. Pandora FMS starts and stop this server automatically as needed.

1.1.3 Netflow probe

Probes are usually Netflow capable routers configured to send Netflow data to the Netflow collector (in our case, a Pandora FMS server with nfcapd running).



NewNetFlowApproach.png



1.2 Installation and requisites

Pandora FMS uses an OpenSource tool callednfcapd to process all the netflow traffic. This daemon is automatically started by Pandora FMS Server. This system stores the data in binary files, in an specific location. You should install nfcapd in your system before working with Netflow, nfcapd by default listen in the port 9995/UDP. Please, consider this if you have firewalls to open this port.

1.2.1 Installation of nfcapd

nfcapd must be manually installed. Pandora FMS will not install nfcapd. For more information on how to install it go to the nfcapd project official page.

Pandora FMS by default uses the directory /var/spool/pandora/data_in/netflow to store Netflow data. nfcapd will point to this directory when started by the Pandora FMS Server. Do not change it unless you know what you are doing.

Pandora FMS needs nfdump version 1.6 to process Netflow data.

To manually test your nfcapd installation run:

nfcapd -l /var/spool/pandora/data_in/netflow -D

Bear in mind that the Pandora FMS Console, and more specifically the Web server that hosts it, needs access to /var/spool/pandora/data_in/netflow to read Netflow data files.

1.2.2 Netflow probe installation

If a Netflow capable router is not available, but you use a Linux server to route your traffic, you can install a software Netflow probe that sends netflow information to the Netflow server.

In Linux there is available a program called fprobe that gets the traffic and send it to a NetFlow server.With this program you can generate Netflow traffic that goes through its interfaces,i.e:

/usr/sbin/fprobe -ieth0 -fip 192.168.70.185:9995

Once the traffic has been generated, you could see stadistics of this traffic in the command:

nfdump -S -R /home/netflow_data/

That should show you information similar to the following one:


Aggregated flows 1286
Top 10 flows ordered by packets:
Date flow start          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2011-12-22 20:41:35.697   901.035 TCP     192.168.60.181:50935 ->     192.168.50.2:22        2105   167388     4
2011-12-22 20:41:35.702   900.874 TCP       192.168.50.2:22    ->   192.168.60.181:50935     1275   202984     4
2011-12-22 20:48:15.057     1.347 TCP       157.88.36.34:80    ->    192.168.50.15:40044      496   737160     1
2011-12-22 20:48:14.742     1.790 TCP     91.121.124.139:80    ->    192.168.50.15:60101      409   607356     1
2011-12-22 20:46:02.791    76.616 TCP      192.168.50.15:80    ->   192.168.60.181:40500      370   477945     1
2011-12-22 20:48:15.015     1.389 TCP      192.168.50.15:40044 ->     157.88.36.34:80         363    22496     1
2011-12-22 20:46:02.791    76.616 TCP     192.168.60.181:40500 ->    192.168.50.15:80         303    24309     1
2011-12-22 20:48:14.689     1.843 TCP      192.168.50.15:60101 ->   91.121.124.139:80         255    13083     1
2011-12-22 20:48:14.665     1.249 TCP     178.32.239.141:80    ->    192.168.50.15:38476      227   335812     1
2011-12-22 20:48:21.350     0.713 TCP     137.205.124.72:80    ->    192.168.50.15:47551      224   330191     1  

Top 10 flows ordered by bytes:
Date flow start          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Packets    Bytes Flows
2011-12-22 20:48:15.057     1.347 TCP       157.88.36.34:80    ->    192.168.50.15:40044      496   737160     1
2011-12-22 20:48:14.742     1.790 TCP     91.121.124.139:80    ->    192.168.50.15:60101      409   607356     1
2011-12-22 20:46:02.791    76.616 TCP      192.168.50.15:80    ->   192.168.60.181:40500      370   477945     1
2011-12-22 20:48:14.665     1.249 TCP     178.32.239.141:80    ->    192.168.50.15:38476      227   335812     1
2011-12-22 20:48:21.350     0.713 TCP     137.205.124.72:80    ->    192.168.50.15:47551      224   330191     1
2011-12-22 20:48:15.313     1.603 TCP       89.102.0.150:80    ->    192.168.50.15:52019      212   313432     1
2011-12-22 20:48:14.996     1.433 TCP     212.219.56.138:80    ->    192.168.50.15:36940      191   281104     1
2011-12-22 20:51:12.325    46.928 TCP      192.168.50.15:80    ->   192.168.60.181:40512      201   245118     1
2011-12-22 20:52:05.935    34.781 TCP      192.168.50.15:80    ->   192.168.60.181:40524      167   211608     1
2011-12-22 20:41:35.702   900.874 TCP       192.168.50.2:22    ->   192.168.60.181:50935     1275   202984     4 

Summary: total flows: 1458, total bytes: 5.9 M, total packets: 15421, avg bps: 49574, avg pps: 15, avg bpp: 399
Time window: 2011-12-22 20:40:46 - 2011-12-22 20:57:21
Total flows processed: 1458, Records skipped: 0, Bytes read: 75864
Sys: 0.006s flows/second: 208345.2   Wall: 0.006s flows/second: 221177.2  


If you have this system working, the following thing will be to configure Pandora FMS to it could use this configuration.

1.3 Working with Netflow in Pandora FMS

Pandora FMS does not store Netflow data in its database, information is processed on demand to render reports.

Pandora FMS works with Netflow data using filters, which are sets of rules that match certain traffic patterns. A rule can be as simple as 'all the traffic from the 192.168.70.0/24 subnet', or it can be a complex pcap expression.

Once filters are created, we have to define reports that determine how the information matched by those filters is going to be displayed (charts, tables...) and the time frame. Netflow reports can be accessed on demand like any other Pandora FMS reports.

Una vez definidos los filtros, definiremos los informes, que son, como queremos ver los datos (graficos, listas...) y en que intervalo de tiempo. Al definir filtros e informes, dejamos definida esa información, de forma similar a como se opera con los informes de Pandora, para utilizarla -bajo demanda- cuando queramos.

Los informes Netflow aparecerán también como "tipo de informe" en la seccion de Informes generales de Pandora, para poderlos "incorporar" también a los informes "normales" de Pandora.

There is also a live Netflow viewer to analyze traffic, modify and create rules on the spot. It can be very useful to investigate problems or temporarily display a chart that we don't want to save for later usage.

1.3.1 Enterprise version: Analysis in Pandora

The Enterprise version allows you to store average traffic values for any filters as Pandora FMS modules. This allows you to configure alerts, generate combined charts or work with it as with any other Pandora FMS module. For more information see section Netflow monitoring with Prediction server

1.3.2 Configuration

First of all, we have to authorize Netflow to it would be accessible from the Operation and Administration menus.



Netflow manager0.png



In the Administration menu, in the Configuration chapter, we find the Netflow option in which we specify the path in which are the files captured of the Netflow traffic. For example:/tmp/netflow. It is also important to verify that the path to the nfcapd daemon is correct.



Netflow manager.png



The following configuration options are available:

  • Data storage path: Directory where netflow data files will be stored.
  • Daemon interval: Time interval in seconds after which data files are rotated.
  • Daemon binary path: Path to the nfcapd binary.
  • Nfdump binary path: Path to the nfdump binary.
  • Maximum chart resolution: Maximum number of points that a netflow area chart will display. The higher the resolution the lower the performance. Values between 50 and 100 are recommended.

Once netflow configuration is enabled, Pandora FMS server must be restarted to start nfcapd server. This server must be accesible on system path and properly installed. Check server logs on any doubt. This server will not appear in Pandora FMS server view because it is not a Pandora server.

1.4 Filters

The creation and edition of filters is at "Administración / Netflow filters".In this snapshot there is a list of the filters already created and that can be changed and deleted.

The Netflow filters allows to define some features that we are going to explain now.

  • Name: It's advisable that the name of the filter would be descriptive.
  • Group:An user could only create one filter or edit one filter of one group to it has access to.
  • Filter: There are two types of filters, basic and advanced. Advanced filters allow using advanced expressions in the same format as nfdump. Basic filters can filter traffic by source Ip, destination Ip, source port or destination port. Lists of comma separated Ips or ports are accepted.
  • Aggregate by: Traffic data can be grouped by one of this fields:

IP Origin: shows the traffic for each IP of different origin
IP Destination: shows the traffic for each IP of different destination
Origin Port: shows the traffic for each port of different origin Destiny Port: shows the traffic for each port of different destination
Protocol: shows the traffic for each different protocol.
Any: (the data will be total).

  • Output format: Data will be displayed in the chosen unit:

Bytes.
Bytes per second.
Kilobytes.
Kilobytes per second.
Megabytes.
Megabytes per second.

Basic web traffic filter example:



Netflow filter normal.png



Advanced intranet traffic filter example:



Netflow filter advanced.png



Here are other examples of advanced filters:

  • Capture traffic to or from 192.168.0.1:
host 192.168.0.1
  • Capture traffic to 192.168.0.1:
dst host 192.168.0.1
  • Capture traffic from 192.168.0.0/24:
src net 192.168.0.0/24
  • Capture HTTP and HTTPS traffic:
(port 80) or (port 443)
  • Capture all traffic except DNS:
port not 53
  • Capture SSH traffic to 192.168.0.1:
(port 22) and (dst host 192.168.0.1)

1.5 Reports

Netflow reports are integrated with Pandora FMS reports (see Reports for more information).

To create a report item, choose one of the available netflow report items.

Netflow report item types.png



And configure it. The following options are available:



Netflow report item configuration.png



  • Type: Item types will be explained below.
  • Filter: Netflow filter to use.
  • Description: Item description.
  • Period: Length of the interval of data to display.
  • Resolution: Data will be retrieved in blocks of size equal to the resolution. If Period / Resolution is bigger than the configure maximum chart resolution the resolution will be dynamically readjusted. For example, for a period of 1 day and a resolution of 1 hour 24 points will be drawn in the chart.
  • Max. values: Maximum number of elements for aggregates. For example, if a chart of HTTP traffic is drawn aggregated by source IP address and Max. values is set to 5, only 5 IP addresses will be shown.

There are five types of netflow report items:

  • Area chart: An area chart, either aggregated or unaggregated.



Netflow chart area aggregated.png



  • Pie chart: An aggregated pie chart.



Netflow chart pie.png



  • Data table: A text representation of the area chart.



Netflow table data.png



  • Statistics table: A text representation of the pie chart.



Netflow table statistics.png



  • Summary table: Traffic summary for the given period.



Netflow summary.png



1.6 Netflow live view

Filters can be visualized live from "Operation / Netflow Live View". This tool allows you to preview changes made to a filter and save it when the desired result is achieved. It is also possible to load and modify already existing filters.

See Reports and Filters to learn how to configure live view options.



Netflow live view.png



To modify an existing filter load if from the Load filter selector, make the desired changes and click on Update current filter.



Netflow update filter.png



To create a new filter, configure it, click on Save as new filter, enter a name and optionally select a group and click on Save as new filter again.



Netflow save filter.png



Go back to Pandora FMS documentation index