WUX monitoring with WUX server and Selenium 3

The user experience monitoring consists of recording automated tasks for both web navigation (PWR/WUX) and interaction with the Windows system desktop and applications (PDR).

These recordings can range from a mouse click on a web browser, typing text, or performing a web search, to opening an application on the desktop. This allows the configuration of functions and automatic recording for later execution to seek results.

Both monitoring systems are used for executing automated web navigation tasks through the Pandora Web Robot Daemon (PWRD).

The UX system performs these monitorings via the EndPoint installed on a machine, while the WUX (Web User Experience) monitoring relies on a server integrated within Pandora FMS.

PWRD is a service that provides the necessary tools to automate web browsing sessions. It uses a file that contains a list of required actions to navigate the web portal being monitored.

Additionally, the UX monitoring allows the execution of automated tasks that interact with the desktop and MS Windows® system applications. These types of tasks cannot be performed with WUX.

If a task automation system is available, Pandora FMS UX also allows the execution of pre-written scripts and monitors their execution.

• It is important to clarify the use of this type of monitoring. Both methods are based on executing the monitoring via plugins on the EndPoints installed on the respective machines.

Considering the differences between UX and WUX monitoring, in general, UX monitoring must follow a series of prerequisites related to the environment being monitored, with the following prior steps:

• Configure a profile in Mozilla Firefox®.
• Install a Selenium® service.
• Distribute PWR in the system.
• Install the Selenium® IDE for Mozilla Firefox®.
• Record a PWR session.

For the standard execution of pre-recorded sessions, the Pandora UX Plugin is used in its latest version 28-04-2022.

For deploying Selenium on WUX servers, a container stack will be used for rapid deployment and easy scaling.

Docker® and Docker Compose® must be installed; Rocky Linux® 9 is recommended as the base operating system.

To install Docker, follow the documentation according to the operating system being used:

For Selenium®, use the official images for the installation and deployment of the stack:

In the Selenium® repository, different images with browsers are available. For Pandora FMS, Mozilla Firefox® and Google Chrome® containers are recommended.

To deploy the Selenium stack, a dedicated directory must be created for the task. Inside it, create a compose.yaml file with the necessary configuration:

# To execute this docker-compose yml file use `docker-compose -f  up`
# Add the `-d` flag at the end for detached execution
# version: "3"
services:
  selenium-hub:
    image: pandorafms/pandorafms-selenium-hub
    mem_limit: 2G
    container_name: selenium-hub-v3
    logging:
        driver: "json-file"
        options:
            max-file: "5"
            max-size: "4m"
            mode: "non-blocking"
    environment:
      - TZ=Europe/Amsterdam
    ports:
      - "4444:4444"

  chrome:
    image: pandorafms/pandorafms-selenium-node-chrome
    mem_limit: 2G
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    logging:
        driver: "json-file"
        options:
            max-file: "5"
            max-size: "4m"
            mode: "non-blocking"
    environment:
      - TZ=Europe/Amsterdam
      - HUB_HOST=selenium-hub
      - HUB_PORT=4444

  firefox:
    image: pandorafms/pandorafms-selenium-node-firefox
    mem_limit: 2G
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    logging:
        driver: "json-file"
        options:
            max-file: "5"
            max-size: "4m"
            mode: "non-blocking"
    environment:
      - TZ=Europe/Amsterdam
      - HUB_HOST=selenium-hub
      - HUB_PORT=4444

To start and run the container with the defined configuration, inside the directory created for the task:

docker compose up -d

To check the services running in the container:

docker compose ps

To view the status and logs of the Selenium services:

docker compose logs

Once the necessary checks are done, to verify that the grid is working correctly and the workers have been registered as defined in the configuration file, it will be necessary to access the following URL:

http://<ip_selenium_server>:4444/grid/console

If you need to increase the number of workers (X, Y):

docker compose scale chrome=X firefox=Y

Selenium operates as a HUB where a container is launched acting as a grid to deploy the necessary worker containers.

Regardless of the chosen mode, once started, you can begin assigning navigation session executions by adding the WUX Server configuration parameters to the Pandora FMS server's configuration file.

Add the following configuration at the end of the /etc/pandora/pandora_server.conf file
(replace <ip_wux_host> with the server's IP address; if it's the same server running the Pandora FMS Server, use 127.0.0.1):

wuxserver 1
wux_host <ip_wux_host>
wux_port 4444

The server's configuration file includes a new token to clean any sessions that may be queued every time the Pandora FMS server is started.

clean_wux_sessions 1 #(default)

1. The management of the wuxserver threads is done automatically when the pandora_server service starts.
2. This is done based on the number of the lower browser nodes in the Selenium hub:

• If there are 2 Firefox nodes and 2 Chrome nodes configured in the hub, the number of threads for the wuxserver will be 2.
• If there is 1 Firefox node and 4 Chrome nodes configured in the hub, the number of threads will be 1.
• If there are 6 Firefox nodes configured in the hub, the number of threads will be 6.

If it is necessary to launch web transactions against Microsoft® browsers, you need to configure a machine (physical or virtual) with the desired version of MS Windows® and set up the driver following the official documentation.

Documentation for installing the MS Internet Explorer® driver:

https://www.selenium.dev/documentation/ie_driver_server/

Documentation for installing the Microsoft Edge® driver:

https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/

For running Selenium, Java® must be installed on the MS Windows® device.

To check if Java® is installed, run:

java -version

The output should be similar to the following:

The JAR file for Selenium is also required to run the server locally and register it on the grid. It can be obtained at:

https://www.selenium.dev/downloads/

To start the Microsoft Edge® server, execute, in a terminal in the directory where the downloaded JAR file is located (all of this in one line, broken down for simplification):

java
  -jar selenium-server-standalone-<VER>.jar
  -port 5555 -role node
  -hub http://<ip_selenium_server>:4444/grid/register
  -browser "browserName=MicrosoftEdge, platform=WINDOWS, maxInstances=1"

To start the Internet Explorer® server, the command is similar, specifying the path <PATH> to the downloaded driver (all of this in one line, broken down for simplification):

java
  -Dwebdriver.ie.driver=<PATH>IEDriverServer.exe
  -jar selenium-server-standalone<VER>.jar
  -port 5555
  -role node
  -hub http://ip_selenium_server:4444/grid/register
  -browser "browserName=internet explorer, platform=WINDOWS, maxInstances=1"

To record a new session, the Selenium IDE extension must be installed in the browser of choice.

Firefox

https://addons.mozilla.org/es/firefox/addon/selenium-ide/

Chrome

https://chrome.google.com/webstore/detail/selenium-ide/mooikfkahbdckldjjndioackbalphokd

The icon will be used to start the recording environment once the extension is installed.

Clicking it will open a menu that allows starting new recording projects.

Once the recording is complete, you will get a result similar to this:

The list of compatible commands for use in Pandora FMS can be found in the following link.

Here are a few recommendations and tips for recording transactions with Selenium IDE and integrating the more complex commands with Pandora FMS:

• Break the transaction into phases whenever possible. This way, the modules that will be created for status, times, and screenshots will be segmented, making it easier to identify where the transaction failed.
  
  
• Use the Selenium command set speed and wait for to avoid false negatives. When executing a transaction, Selenium commands have no default delay between them, meaning some may not have a completion time or timeout. This causes the transaction to execute as quickly as possible, but if the webpage is a bit slow or takes a second longer to load, the check may fail. A common scenario: after executing a click command and changing pages, if the next element to be interacted with is delayed by a second, it won’t be found, and the check will fail.
  To avoid this, use the set speed command, which adds a delay in milliseconds between commands. It is recommended to set this at the start of the transaction. Also, for cases where page loading or element appearance is known to be delayed, use the commands wait for element present, wait for visible, and wait for text, where you can set the waiting time in milliseconds before marking the transaction as failed. While these commands significantly improve check reliability, they also increase the time it takes to execute the transaction.
  
  
• Check the elements. Use commands like assert and verify in their various forms. For example, finishing a transaction with a click does not guarantee that the page opened by the clicked element will actually load, only that clicking the element is possible. If a verify text command is added to text that only loads after the click, it will confirm that the page linked by the click is available.
  
  
• Use store window handle for transactions where you switch windows. A window change (with select window) may fail if the identifier for the initial window is not stored beforehand.
  
  
• Use xpath when the Target fails due to CSS identifier or when you want to search for content on the page. By default, the Selenium IDE recorder uses the CSS locator in the Target, but it also records the locator by xpath, so you can see all the locators it saves by clicking on the Target box in the recorder:

Additionally, using xpath paths, you can search for text inside the tags of the page, making recordings more dynamic. In the screenshot above, you could use an xpath to search for the text “Innovadores de la monitorización” within all span tags on the page, not just a specific locator.

• Correct usage of the execute script command. This command runs a piece of JavaScript code in the window where it is located in the transaction. It is recommended to read the guide to understand its usage and the different options available: https://ui.vision/rpa/docs/selenium-ide/executescript.
  However, the use of previously stored variables (via the store text command, for example) should be placed in double quotes for Pandora FMS’s webdriver to interpret them correctly.

This is an example of a variable stored with store text and then used in execute script, so that Pandora FMS’s server correctly interprets it. This use of the variable in quotes will fail when running the script in the Selenium IDE recorder:

Once you have a recorded browsing session, it’s time to deploy it as a Module in the Pandora FMS Web Console.

To do this, go to any agent associated with a server that has WUX functionality enabled and click on the Create Module button:

By clicking the Create button, you will need to fill out the fields presented in the form (only those related to this topic are shown):

• Name: WUX Module (Parent); remember that all submodules with monitoring results will be dependent on this new module.
  
  
• Execute tests from: Indicates which WUX Server will execute the check. By default, it is set to None, so select a server from the list.
  
  
• Run performance tests: Specifies whether to run performance tests, i.e., monitor the browsing experience and also the performance statistics of the target website access (field Target web site).
  
  
• Browser: Web browser to execute the check. Compatibility with IE and Edge browsers is experimental, and tests may not work as expected.

  The indicated browser must be listed in the grid or Selenium server used by the WUX server.

• User data dir: Optional, only if Google Chrome is used, it allows specifying a directory for data.
  
  
• Profile: Optional, only if Google Chrome is used, it allows specifying a user profile. If the profile name does not exist, the default profile will be used.
  
  
• Accept insecure certificates: If enabled, it will accept any insecure certificate (self-signed, expired, etc.) as part of the browsing process.

  This option is only available for Google Chrome® and Mozilla Firefox® browsers with a Selenium 3 server.

• Upload your selenium test in html or side (only Selenium 3) format: Use the Select file button to upload the content of the previously recorded browsing session files.
  
  

The functioning of custom macros involves replacing certain text strings in the browsing session file with customizable values.

For web analysis modules, this functionality has been improved to allow dynamic macros, which enable these values to be translated into variable dates and times.

Why this functionality?

Suppose you need to monitor, via a browsing session, the correct functioning of a web-based room reservation application.

If you set a fixed date and time when filling out the form, the system may cancel the reservation because at some point it will be in the past.

It is also possible that there is a maximum time limit for making that reservation, and that the system forces you to reserve the room within a specific time frame.

To avoid having to edit the macro every few days, and to not worry about the configuration part, you can use dynamic macros, telling the system to always reserve the meeting room for the day after the web test.

To use this functionality, the values must follow a specific format, allowing the following possible replacements:

• @DATE_FORMAT: current date/time with user-defined format.
• @DATE_FORMAT_nh: hours.
• @DATE_FORMAT_nm: minutes.
• @DATE_FORMAT_nd: days.
• @DATE_FORMAT_ns: seconds.
• @DATE_FORMAT_nM: month.
• @DATE_FORMAT_nY: years.

Where “n” can be an unsigned (positive) or negative number, and FORMAT follows the standard of Perl's strftime:

@DATE_%Y-%m-%d %H:%M:%S
@DATE_%H:%M:%S_300s
@DATE_%H:%M:%S_-1h

The information generated by WUX will be displayed in the form of modules, and enabling the view in hierarchical mode in the list of modules will help to display the information much more clearly:

Within this hierarchy are:

• module_Global_Status: It will indicate the overall status of the entire navigation.

1. If there is a recording, it gives the status of the WUX recording.
2. If Run performance Test is enabled but recording is not included, the status of the Global Status module is that of the check performed when these modules are obtained.

• module_Global_Time: It will indicate the total time spent on the entire navigation.
• module_Phase X: Phase name_Status: Indicates the navigation status during phase X.
• module_Phase X: Phase name_Time: Indicates the time spent in phase X.
• module_Global_Screenshot: Contains an image with the result of the navigation error, only generated in case of error:

By accessing the agent's WUX Console, you can view additional details about the transaction and its phases:

Website statistics are summarized in the following concepts:

• Stats_TT: Total time to obtain the website.
• Stats_TDNS: Total time taken to resolve the target IP address.
• Stats_TTCP: Time spent connecting via TCP.
• Stats_TSSL: Time taken to establish SSL communication.
• Stats_TST: Time elapsed until data transfer began.
• Stats_TTC: Time transferring data; will group all resource transfer times.
• Stats_TTR: Time spent transferring the page content.
• Stats_TTR_main: Time spent transferring the HTML code.
• Stats_TTR_image: Time spent transferring image resources (png|jpg|jpeg|bmp|tiff|gif|webp|svg).
• Stats_TTR_css: Time spent transferring style sheets.
• Stats_TTR_js: Time spent transferring JavaScript files.

Alerts associated with web analysis modules follow the same operating dynamics as the entire alert system in Pandora FMS.

For compatibility, it is recommended to assign alert templates to subelements autogenerated by the web analytics module:

• The state of global navigation.
• Time threshold alerts.
• Alerts with notification template set to “always” for result image capture modules.

In order to display user navigation information, two types of widgets have been added to the Dashboards: Agent WUX transaction and WUX transaction stats:

• run: Run a test.
• build_transaction: Pre-build a transaction based on the last test (internal).
• get_transaction: Returns the current transaction, always used to retrieve results, phase order, and variables.
• sessions: Retrieves the list of sessions present in the hub.
• kill_sessions: Deletes all existing sessions in the hub.

• extract: Combines source and proprietary code for extraction.
• storeExtraction: Store the result of extract in a variable.
• dispatchEvent: Triggers a JavaScript event on a given element.
• takeElementScreenshot: Take a screenshot of a specific element.
• phase_start: Start a phase and transform the test into a WUX transaction.
• phase_end: Complete a phase.

• getValue: Extract a value.

• getVariable: Extracts a specific value from a variable.

• getScreenshot: Take a screenshot.

←Back to Pandora FMS documentation index