View source for Pandora: Documentation en: Web Monitoring

[[Pandora:Documentation_en|Go back to Pandora FMS documentation index]]

= Monitoring User Browsing Behavior =

== Introduction ==

In the Enterprise version, it is possible to monitor a Web using the WEB Server component, also called Goliat Server.

[[File: GoliatLogo 2.jpg|right]]

This functionality comes from an old project of the founder of Pandora FMS. Goliat F.I.S.T. was an open source project to perform dynamic load audits on web services. You can still find the source code (from 2002) whose updating stopped  more or less in 2010  [https://sourceforge.net/projects/goliat/].

In Pandora FMS, it works as an independent server, similar to the network server, WMI server or remote plugin server. 
This system operates under the principle of ''web transaction'', where each transaction is defined by one or more consecutive steps, which must be successfully concluded in order to consider that the transaction has been properly completed. The execution of a ''web transaction'' faithfully reproduces the complete browsing process that can include aspects such as authenticating yourself in a form, clicking on a menu option or filling in a form, verifying that each step returns a specific text string. 

Any failure at any point in the process would result in a checking failure. The complete transaction includes the downloading of all resources (graphics, animations, etc.) that the actual navigation includes. In addition to performing response time and performance checks, it is possible to extract values from the web pages and then process them.

Goliat is able to monitor both HTTP and HTTPS in a transparent way for the user, supports session management through cookies, parameter passage, and of course, downloading the resources associated with each page. It also has '''important limitations such as the dynamic management of javascript at runtime'''. For more complex web transactions, Pandora FMS has another much more powerful  (and complex) component called WUX Monitoring ('''Web User Experience'''), for more information click on the following link  [https://wiki.pandorafms.com/index.php?title=Pandora:Documentation_en:Monitoring_WUX]

==Installation and Configuration==

To be able to use Goliat, activate it in the Pandora FMS Enterprise server:

 webserver 1

Depending on the number of requests you want to make, you may have to increase the number of threads and the default timeout:

 web_threads 1 
 web_timeout 60

There is an advanced configuration token that will allow you to change the type of library you use under Goliat, LWP or CURL. LWP is used by default, but if you encounter any issues (SSL connections, long timeouts or concurrency problems in case of many threads), switch to CURL:

 web_engine curl

== Creating Web Modules ==

To remotely monitor a web page, we have to click on the modules tab. Then, we'll select 'Create a new webserver module' and click on ''Create'':

<center>
[[image:Gagita2.png]]
</center>

Click ''Create'', to see a form appear in which you fill out the necessary fields to monitor a web page. The name and, above all, the type of WEB check-up, are the most basic:

<center>
[[image:goliat_types.jpg]]
</center>

There are various check types to choose from:

* '''Remote HTTP module to check latency''': obtains the total time that elapses from the first request until the last one is checked (in a WEB test there are one or more intermediate requests that complete the transaction). If you have defined in the check definition that the transaction is carried out more than once, the average time of each request is used.

* '''Remote HTTP module to check server response''':  you get a 1 (OK) or 0 (CRITICAL) as a result of checking the entire transaction. If there are several attempts, but at least one of them has failed, the test as a whole is also considered to have failed. Precisely the number of attempts is sometimes used to avoid false positives. In case you want to perform the test several times in case of a failure, use the'' retries'' field (see advanced fields below).

* '''Remote HTTP module to retrieve numeric data''': Retrieves a numeric value from an HTTP response using a regular expression.

* '''Remote HTTP module to retrieve string data''': Retrieves a string from an HTTP response using a regular expression.

'''Web Checks'''

This essential field defines the WEB check to be performed. This is defined in one or more steps, or simple requests. These simple requests must be written in a special format in the Web checks field. Checks start with the tab page ''task_begin'' and end with the tab page ''task_end''.

A complete example of a simple transaction would be the following:

 task_begin 
 get http://apache.org/
 cookie 0
 resource 0
 check_string Apache Software Foundation
 task_end

In this basic example, we are checking if there is a string in a web page, for that there is the variable '''check_string'''. This variable does not allow you to check HTML itself, it only searches for sub-strings of text. We are looking for "Apache Software Foundation" on the web http://apache.org. If that text string exists, the check will return OK (if it is ''Remote HTTP module to check server response'')

To ensure that a string does not exist on a web page, you can use the variable' check_not_string':

<pre>
check_not_string Section 3
</pre>


There are several extra variables to check forms:

*'''resource (1 or 0)''': Download all the web resources (images, videos, etc).
*'''cookie (1 or 0)''': Keeps a cookie or an open session for later checks.
*'''variable_name''': The name of a variable in a form.
*'''variable_value''': The value of the previous variable on the form.

By using these variables, it's possible to send data to forms and check whether they work appropriately or not.

{{Warning|In some specific cases, the domain redirection may '''not''' work. To solve this problem, you're required to create a module which uses the final domain address after all redirections are completed.}}

{{Warning|The arguments taken by the  "check_string" syntax are not normal text strings, they are "regular expressions". That is to say, if you search for the string "Pandora FMS (4.0)" you will have to search for it with a regular expression, e. g. Pandora FMS \(4.0\). This allows you to do much more powerful searches, but you should be aware that any character other than a letter or number will have to be escaped with \.}}

== Checking the Latency of a Website ==

If you want to check the latency of a website, you're just required to select the module type named ''Remote HTTP module to check latency''. If you want to learn the latency of the website ''http://pandorafms.com'' , the code should look like the one below.

 task_begin
 get http://pandorafms.com
 task_end

We can add the configuration token '''resource 1''' so that the download time you calculate is downloading all resources (javascript, CSS, images, etc.), thus calculating a more real data. 
<br><br>

{{Tip|The download time of the web site is NOT the time it takes to view a web site in a browser, as this usually depends on the loading time of Javascript, and Goliat downloads the javascript, but does not run it.}}

== Website Checks through a Proxy==

You're also able to conduct website checks by using a proxy. To configure the proxy, you're required to add the proxy URL in the field 'Proxy URL' which is located under 'Advanced options':

<center>
<br><br>
[[Image:Goliat_proxy_conf.png|center]]
<br><br>
</center>

An example of the URL could be:

 http://proxy.domain.com:8080

If the proxy requires an authentication, you may use an URL like this:

 http://my_user:my_pwd@proxy.domain.com:8080

== Retrieving data from a website ==

We may not want to know if a specific Web site is working or how long it takes, but to get a real time value, such as Google's stock market value. To do this, we will use a ''Remote HTTP module to retrieve numeric data'' with the appropriate regular expression:

<pre>
task_begin
get http://finance.google.com/finance/info?client=ig&q=NASDAQ%3aGOOG
get_content \d+\.\d+
task_end
</pre>

The output is going to be something like this:

<br><br><center>
[[image:Google_stock_quote.png|center|600px]]
</center><br><br>

It is also possible to specify a more complex regular expression for collecting data from more complex HTTP responses with the configuration token'' get_content_advanced'':


<pre>
task_begin
get http://finance.yahoo.com/q?s=GOOG
get_content_advanced <span id="yfs_l84_goog">([\d\.]+)</span>
task_end
</pre>

{{Warning|The part of the regular expression to be returned (which is defined in ''get_content_advanced'') must be enclosed in brackets.}}

To configure the thresholds that trigger the warning or critical status, we will use the module configuration to verify that the received string matches what is expected.

== Form Checking on a Website ==

A more interesting check is that of a web form. However, which is much more complex than simply checking a text on a web page. This example check will use Pandora's own console, log in, and will verify that it has been able to do it, verifying a text in the workspace section where it shows the data of the user who has logged in. If it is a default console, the admin user contains the description "Admin Pandora".

To be able to perform this type of checks you must have the necessary credentials to be able to log in (since we use these values to "send" them to the HTML form. In addition, you must go to the page and get the HTML code to be able to see the names of the variables, then it is necessary to have minimal knowledge of HTML to understand how Goliat works. In this example we are using "admin" with the password "pandora" which is the default credentials. I should have changed them, if you haven't already, this is a good time to do it!

{{Tip|The ideal thing to do when designing a WEB transactional test with several steps is to test it step by step, in case we missed something in one of the steps.}}

Suppose that the login URL of our Pandora Console is:
 http://192.168.70.116/pandora_console/
Analyzing the HTML code, it is observed that the variables of the login form are:

*''nick'': user name
*''pass'': user password

The variables variable_name and variable_value should be used tpgether to validate the form. 

The first step is to access the form, send the user and password and authenticate (determining the success of that authentication we will see in the second step).

<pre>
task_begin
post http://192.168.70.116/pandora_console/index.php?login=1
variable_name nick
variable_value admin
variable_name pass
variable_value pandora
cookie 1
resource 1
task_end
</pre>

With the previous task, it would have been possible to access and validate the website, now it will be verified that you are actually registered in the website by searching for something on it that you can only see when you are registered. We use the cookie 1 token to maintain the persistence of cookies obtained in the previous step. Without them, we will not be able to "simulate" a session.

In the second step we will access the user details page and look for the phone, which by default for the user "admin" is 555-555-555. If we can see it is that we have logged in correctly on the console:

<pre>
task_begin
get http://192.168.70.116/pandora_console/index.php?sec=workspace&sec2=operation/users/user_edit
cookie 1
resource 1
check_string 555-555-5555
task_end
</pre>

And finally, we will disconnect from the console and look for the logout message:

<pre>
task_begin
get http://192.168.70.116/pandora_console/index.php?bye=bye
cookie 1
resource 1
check_string Logged out
task_end
</pre>

With which the total check would remain in Pandora FMS as follows:

<center>
[[Image:goliat_full_sample.jpg]]
</center>

== Comportamiento de las peticiones WEB ==

The fields of the advanced properties are similar to those of other types of modules, although there are some different and specific fields of the WEB checks:

'''Timeout'''

This is the expiry time during the request, if this time is exceeded the request for verification will be discarded.

'''Agent browser id'''

This is the web browser identifier to use, as certain pages only accept some web browsers (see zytrax.com for more information).

'''Requests'''

Pandora FMS will repeat the check the number of times indicated in this parameter. If one of the checks fails, the check will be considered erroneous. Depending on the number of checks in the module, a certain number of pages will be obtained; that is, if the module consists of three checks, three pages will be downloaded, and if some value has been set in the Requests field, then the number of downloads will be multiplied by it. It is important to keep this in mind to know the total time it will take the module to complete the operations.

'''Retries'''

It has nothing to do with requests and implies the opposite, i. e. if its value is > 1, and fails the first time, it will retry a certain number of times until it succeeds. For example with retries = 2 and Requests = 1, if the first test failed, it would retry it once more, and if the second one works, the check would be considered as valid. If it was Requests = 2, and Retries = 1, it would perform two checks, but if either failed, it would consider the test as failed.

== Simple HTTP Authentication ==

Some websites might require HTTP authentication. This authentication method doesn't consist of the 'normal' user / password form. HTTP authentication is the one in which a pop-up appears, informing that the site "xxxx" asks us for credentials.

<center>
[[Image:Conexion http.png]]
</center>

It can be configured in the advanced options (as seen in the previous screenshot) or directly in the WEB task definition with the following configuration tokens:

* Check type - HTTP server check type.
* http auth (login) - The user.
* http auth (password) - The password.
* Proxy auth realm - The auth realm's name.
* Proxy auth (server) - The domain and HTTP port on which to listen on.
* Proxy URL - Proxy server url
* Proxy auth (login) - Proxy connection user
* Proxy auth (pass) - Proxy connection password

This is a full example:

 task_begin
 get http://artica.es/pandoraupdate4/ui/
 cookie 1
 resource 1
 check_string Pandora FMS Update Manager \(4.0\)
 http_auth_serverport artica.es:80
 http_auth_realm Private area
 http_auth_user admin
 http_auth_pass xxxx
 task_end

== Webservice Monitoring ==

With Pandora FMS and Goliat's Web Surveillance features, you're also able to monitor web services and APIs which are based on the REST specification, but you're unable to conduct such a surveillance on SOAP or XML-RPC based web services.

Let's assume for a moment that you want to check a web API by a specific call which returns a number (from '0' to 'n') if it works properly. If not, it's not going to return anything, which will be considered a failure by Pandora FMS:

 task_begin
 get http://artica.es/integria/include/api.php?user=slerena&pass=xxxx&op=get_stats&params=opened,,1
 check_string \n[0-9]+
 task_end

This is going to return a reply like this:

<pre>
HTTP/1.1 200 OK
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection: close
Date: Mon, 13 May 2013 15:39:27 GMT
Pragma: no-cache
Server: Apache
Vary: Accept-Encoding
Content-Type: text/html
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Client-Date: Mon, 13 May 2013 15:39:27 GMT
Client-Peer: 64.90.57.215:80
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
Set-Cookie: a81d4c5e530ad73e256b7729246d3d2c=pcasWqI6pZzT2x2AuWo602; path=/

0
</pre>

This is going to produce an 'OK' (Green) module status, because the regular expression matching ('regexp') found the '0' just before a carriage return. It's important to check the entire response data - not just the 'data' section, so you can match the HTTP headers, too. For other responses, a different regular expression is required.

== HTTPS Monitoring ==

Goliat is able to check both HTTP and HTTPS. To conduct checks on secured websites which are utilizing HTTPS, you only have to incorporate the protocol into its URL, e.g.:

<pre>
task_begin
get https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F%3Fui%3Dhtml%26zy%3Dl&bsv=zpwhtygjntrz&ss=1&scc=1&ltmpl=default&ltmplcache=2
cookie 1
resource 0
check_string Google
task_end
</pre>

== IPv6 Support ==

Goliat supports IPv6, although it needs to use FQDN addresses, this means that URLs must have full names (e.g. ipv6.google.com). The numerical representation of IPv6 addresses (pe:[:: 1],[2404:6800:4004:4004:803::1014] etc.) is not valid for IPv6 checks with Goliat.

== Advanced Options ==

=== Modifying HTTP Headers (available from versions 4.0.2 and above) ===

With the ''header'' option, you're able to modify HTTP headers or create your own. The example below e.g. changes the ''Host'' HTTP header:

 task_begin
 get http://192.168.1.5/index.php
 header Host 192.168.1.1
 task_end

=== Debugging Web Surveillance (available from versions 4.0.2 and above) ===

If you want to debug a website check, please add the ''debug <log_file>'' option. The files log_file.req and log_file.res are going to be created along with the contents of the HTTP request and response:

 task_begin
 get http://192.168.1.5/index.php
 debug /tmp/request.log
 task_end

The website check above is going to generate the files ''/tmp/request.log.req'' and ''/tmp/request.log.res''.

=== Using CURL instead of LWP ===

LWP sometimes crashes when multiple threads issue HTTPS requests simultaneously. To solve this problem, you just have to edit the file named '/etc/pandora/pandora_server.conf' and to add the below mentioned line to it:

 web_engine curl

After you've restarted the Pandora FMS Server, the CURL binary is going to be used to perform website checks instead of LWP.

= Distributed Transactional WEB Monitoring by Selenium =

[[file:selenium_logo.png|right]]

In addition to the features of Goliat, which is integral part of all Pandora FMS Enterprise versions, there is another way to perform Transactional Monitoring by Pandora FMS. This method is using an agent-like approach instead of a centralized system. It allows you to distribute the load and to use servers in remote networks to monitor different websites or applications.

In such a case, [http://docs.seleniumhq.org/ 'Selenium'] is used as the back end instead of Goliat. It supports a navigation 'by clicks' which is much more precise than the ones offered by Goliat. The Selenium plug-in agent interacts with the Selenium Server - and this in turn with the system's browser. This method offers the possibility to browse the web using Chrome, Firefox or the Internet Explorer. It runs on Windows and Linux, enables latency and session checks and '''supports navigation with heavy implementations of JavaScript and Java applets, Flash or any other technology, supported by browsers today'''.

The Selenium plug-in Documentation is very extensive and specific and can be found in the [http://library.pandorafms.com/Library/repository/en Pandora FMS Module Library], along with the Enterprise Selenium Plug in.

'''This functionality had been improved, now you have Pandora UX to automatize and monitor both desktop and web User eXperience sessions'''

[[Pandora:Documentation_en|Go back to the Pandora FMS Documentation Index]]

[[Category:Pandora FMS]]
[[Category:Documentation]]