Citrix monitoring plugin

• Introduction
• Compatibility Matrix
• Prerequisites
• Configuration
• Most Important Citrix Performance Counters
• Additional citrix monitoring
• Modules generated by the plugin

Introduction

We have chosen a number of modules "base" based on our experience in monitoring systems and the optimal parameters to diagnose potential bottlenecks on these servers. To extract the information is used:

• The software already installed on the system (WMI, Performance Counters, Powershell) for monitoring performed by the plugin without having to install third-party libraries or utilities.
  
• The system, which integrates with the Windows agent and has the ability to distribute file collections, so that the plugin can be distributed on one side and the file collections individually—by agent—and/or by policy.

Compatibility Matrix

System where it has been tested	Windows Server 2008, Citrix XenApp 6
Systems on which it is expected to run	Same system or higher

Depending on the type of operating system and language, the format of the performance counters may change and need to be adapted accordingly.

Prerequisites

The requirements to run the plugin are:

• System with Powershell and WMI enabled.
• Agent PandoraFMS with permissions to run Powershell scripts. 

Configuration

Instalation

Copy the plugin in the plugin directory of agent or distributing by file collections. Same with the config files. The execution from the agent will be similar to this, but using the paths where the plugin and confs are installed. 

module_plugin "<powershell-path>\powershell.exe" -file C:\'<plugin-path>\pandora_citrix.ps1' -perf C:\'<conf-path>\counters_data.txt' -serv C:\'<conf-path>\services.txt’ -inc C:\'<conf-path>\counters_inc.txt' 2> error.log

If we want no monitor any perf counter or only a specific type, would have to carry out these executions:

• Only with generic_data counters: 

module_plugin "<powershell-path>\powershell.exe" -file C:\'<plugin-path>\pandora_citrix.ps1' -perf C:\'<conf-path>\counters_data.txt' -serv C:\'<conf-path>\services.txt' 2> error.log

• Only with generic_inc counters: 

module_plugin "<powershell-path>\powershell.exe" -file C:\'<plugin-path>\pandora_ citrix.ps1' -perf none -serv C:\'<conf-path>\services.txt'   -inc C:\'<conf-path>\counters_inc.txt'' 2> error.log

• Without counters: 

module_plugin "<powershell-path>\powershell.exe" -file C:\'<plugin-path>\pandora_ citrix.ps1' -perf none -serv C:\'<conf-path>\services.txt' 2> error.log

• With counters and without services: 

module_plugin "<powershell-path>\powershell.exe" -file C:\'<plugin-path>\pandora_ citrix.ps1' -perf C:\'<conf-path5>\counters_data.txt' -inc C:\'<conf-path>\counters_inc.txt' 2> error.log

Configuration

First of all we have enabled the plugin execution from powershell For this, we must run this command in a powershell terminal: 

set-executionPolicy -Scope Process -ExecutionPolicy Bypass

or

set-executionPolicy RemoteSigned

If we run the plugin from the terminal and returns the information correctly, but if you execute it in from Pandora agent and returns no results, and in debug mode tells us that remote_signed not enabled, we should add the next module to be activated within the execution of the agent. This, you have to add only the first time, then stay enabled and we won´t have problems. 

module_begin 
module_name Remote_signed 
module_type generic_data_string 
module_exec powershell.exe –command set-executionPolicy -Scope Process -ExecutionPolicy bypass 
module_end

or

module_begin 
module_name Remote_signed 
module_type generic_data_string
module_exec powershell.exe -command set-executionPolicy RemoteSigned 
module_end

As we noted in the plugin implementation, it is accompanied by two or three configuration files, one with the name of the services and another two as the name of the performance counters to be monitored. At one we indicate the counters that we can see as generic_data type and in the second we will use the incremental type (generic_data_inc). In the event that the incremental type is not used should not be added at runtime. Each service, each performance counter should be on one line and if If you do not want monitor any counter or service, delete it from the list.

Configuration files examples 

Services.txt 

Citrix Independent Management Architecture
Citrix XML
Citrix XTE Server
Citrix CPU Utilization Management/Resource Management
Citrix CPU Utilization Management/User Session sync
Citrix CPU Utilization Management/CPU Rebalancer Services states
Citrix Print Manager
Citrix Virtual Memory Optimization
Citrix MF COM Remote Desktop Services/Terminal Services
Citrix WMI
Citrix Client Network
Citrix Streaming/Citrix Streaming Helper
Server
Netlogon
Remote Procedure Call (RPC)
Workstation
Print Spooler

Counters_data.txt 

\Processor(*)\% Processor Time
\System\Processor Queue Length
\Memory\Available Bytes
\Memory\Pages /sec
\Paging File\% Usage
\PhysicalDisk(_Total)\% Free Space
\LogicalDisk(_Total)\% Free Space
\LogicalDisk(_Total)\% Disk Time
\PhysicalDisk(_Total)\Current Disk Queue Length
\LogicalDisk(_Total)\Current Disk Queue Length
\LogicalDisk(_Total)\ Avg. Disk Sec/Read
\LogicalDisk(_Total)\ Avg. Disk Sec/Write
\LogicalDisk(_Total)\ Avg. Disk Sec/Transfer
\PhysicalDisk(_Total)\ Avg. Disk Sec/Read
\PhysicalDisk(_Total)\ Avg. Disk Sec/Write
\PhysicalDisk(_Total)\ Avg. Disk Sec/Transfer
\Citrix MetaFrame Presentation Server\Application Resolution Time (ms)
\Citrix MetaFrame Presentation Server\DataStore Connection Failure
\Citrix MetaFrame Presentation Server\Number of busy XML threads
\Citrix MetaFrame Presentation Server\Resolution WorkItem Queue Ready Count
\Citrix MetaFrame Presentation Server\WorkItem Queue Ready Count
\Citrix Licensing\Citrix Licensing - License Server Connection Failure
\Citrix Licensing\Citrix Licensing - Last Recorded License Check - Out Response Time
\Process(ImaSrv)\Private Bytes

Most Important Citrix Performance Counters

1.-  System Performance Counters (Citrix ) 

\Processor(*)\% Processor Time -> This counter is the main indicator of processor activity and shows the average percentage of time observed during the sampling interval. It is calculated by monitoring the time the service is idle and subtracting that value from 100%. The manufacturer indicates that between 80% and 90% over a 15-minute period should be set to Warning status and over 90% to Critical status.

\System\Processor Queue Length -> This is the current size in bytes of the virtual address space the process is using. The use of virtual address space does not necessarily imply corresponding use of either disk or main memory pages. Virtual space is finite, and the process can limit its ability to load libraries.

\Memory\Available Bytes -> Available memory indicates the amount of memory remaining after non-paged block allocation, paged block allocations, working sets, and all file system cache memory have taken their share. <30% RAM in warning state, <15% RAM in critical state.

\Memory\Pages /sec -> This is the speed at which pages are read or written to disk to resolve serious page faults. Warning status between 10 and 20, Critical > 20.

\Paging File\% Usage -> This is the percentage of the page file instance in use. Warning 40-70%, Critical > 70%.

\PhysicalDisk(_Total)\% Free Space -> This is the percentage of total usable space on the selected physical disk that is free. Critical <5%, Warning 5-10%.

\LogicalDisk(_Total)\% Free Space -> This is the percentage of total usable space on the selected logical disk that is free. Critical <5%, Warning 5-10%.

\LogicalDisk(_Total)\% Disk Time -> This indicates how busy the disk is. Warning > 70% consistently and Critical > 90%.

\PhysicalDisk(_Total)\Current Disk Queue Length -> This indicates the number of I/O operations waiting for the hard disk to become available. If the value here is greater than twice the number of spindles, it means that the disk itself may be the bottleneck. >=1 Warning, >=2 Critical.

\LogicalDisk(_Total)\Current Disk Queue Length -> Provides a primary measure of disk congestion. It is an indication of the number of transactions waiting to be processed. >=1 Warning, >=2 Consistently Critical, possibility of creating it with FF Threshold 2.

\LogicalDisk(_Total)\ Avg. Disk Sec/Read -> Shows the average time in seconds for a read from or to a disk. >= 15 ms Warning, >=20 ms Critical.

\LogicalDisk(_Total)\ Avg. Disk Sec/Write -> Shows the average time in seconds for a write from or to a disk. >= 15 ms Warning, >=20 ms Critical.

\LogicalDisk(_Total)\ Avg. Disk Sec/Transfer -> Shows the average time in seconds for a transfer from or to a disk. >= 15 ms Warning, >=20 ms Critical.

\PhysicalDisk(_Total)\ Avg. Disk Sec/Read -> Shows the average time in seconds for a read from or to a disk. >= 15 ms Warning, >=20 ms Critical.

\PhysicalDisk(_Total)\ Avg. Disk Sec/Write -> Shows the average time in seconds for a write from or to a disk. >= 15 ms Warning, >=20 ms Critical.

\PhysicalDisk(_Total)\ Avg. Disk Sec/Transfer -> Shows the average time in seconds for a transfer from or to a disk. >= 15 ms Warning, >=20 ms Critical.

\Network Interface (*)\ Bytes Total/sec -> Shows the speed at which the network adapter processes data bytes. This counter includes all data files and applications, as well as protocol information such as packet headers. 

Note: The names of all these counters may vary depending on the version and language of the OS used.

2.- XenApp Performance Counters

\Citrix MetaFrame Presentation Server\Application Resolution Time (ms) -> The time in milliseconds that resolution was completed. This is also the time required to determine the “least loaded” server during application launch. A baseline would need to be established in the environment in order to accurately set threshold values.

\Citrix MetaFrame Presentation Server\DataStore Connection Failure -> The number of minutes that the XenApp server has been disconnected from the data store. Thresholds should take into account events such as reboots and scheduled maintenance.

\Citrix MetaFrame Presentation Server\Number of busy XML threads -> The number of XML threads currently being processed. There are 16 worker threads in the Citrix XML Service. A count of 16 or more shows that XML requests are not being processed in a timely manner.

\Citrix MetaFrame Presentation Server\Resolution WorkItem Queue Ready Count -> The number of resolution work items (related to application launches) that are ready to be executed. A value greater than 0 indicates that requests are queued while IMA handles other requests.

\Citrix MetaFrame Presentation Server\WorkItem Queue Ready Count -> The number of work items that are ready to be executed. A value greater than 0 indicates that requests are queued while IMA handles other requests. This counter should not be greater than 1 for an extended period of time.

\Citrix Licensing\Citrix Licensing - License Server Connection Failure -> Shows the number of minutes that XenApp has been disconnected from the license server. Warning > 1 minute. > 1440 minutes Critical.

\Citrix Licensing\Citrix Licensing – Last Recorded License Check – Out Response Time  -> Shows the last recorded license check - out response time in milliseconds. A value greater than 5,000 milliseconds indicates a performance issue on the license server.

\Process(ImaSrv)\Private Bytes -> The current size in bytes of memory that this process (IMA) has allocated and cannot be shared with other processes.

If we want to monitor the % usage of the different processes that are running:
\Process(Process_Name)\% Processor Time -> % CPU in use by the process in question.

3.- Web Interface Performance Counters 

\ASP.NET\Request Queued -> The number of requests waiting to be processed by ASP. A baseline must be established in the environment in order to accurately set threshold values.

\ASP.NET\Request rejected -> The number of requests rejected because the request queue was full.

\ASP.NET\Request Current -> The current number of requests, both running and queued.

\ASP.NET\Request Execution Time -> The amount of time (in milliseconds) it took to execute the most recent request.

\Web Service\Concurrent Connections -> Current connections is the current number of connections established with the web service.

4.- Xen Desktop Performance Counters 

\Citrix Broker Service\Database Avg. Transaction Time -> Average time, in seconds, to execute a database transaction. A baseline must be established in the environment in order to accurately set threshold values.

\Citrix Broker Service\Database Connected -> Indicates whether this service is in contact with its database. (1 is connected, 0 is not connected).

\Citrix Broker Service\Database Transaction Errors/sec -> The rate at which database transactions are failing.

\Citrix Broker Service\Brokered Sessions -> The number of virtual desktop sessions brokered by the Citrix Service Broker.

\Citrix XML Service\Avg. Transaction Time -> The number of virtual desktop sessions brokered by the Citrix Service Broker.

5.- Other Citrix Performance Counters 

Get-Counter –listSet * | Select-Object -ExpandProperty Paths > counters.txt 

Additional citrix monitoring

Network modules

We can remotely monitor the different ports that must be operative in our Citrix environment, here's the essential, but in the following catalog has details of all ports used by different components of Citrix 

http://support.citrix.com/content/dam/supportWS/kA460000000Cc9tCAC/Communication_Ports_ Used_By_Citrix.pdf

1. Controller,  Console Director and StoreFront

Ports 80 and/or 443 TCP 

2.- License Server 

Ports 7279, 27000, 8083, 8082 TCP 

It may do this through port monitoring server or network PandoraFMS through local modules from the same server or another we wanted to use for this purpose. 

Windows: 

module_begin
module_name Check port 80
module_type generic_proc
module_tcpcheck IP_Citrix_server
module_port 80
module_end

Linux:

module_begin
module_name Check port 80
module_type generic_proc
module_exec nmap IP_Citrix_server  | grep 80/tcp | awk '{print $2}' | grep open | wc -l
module_end

Monitoring Citrix Events

We can monitor any of the Citrix events writtent to the Event Log.

For them we can use module logevent type. 

module_begin 
module_name Event Citrix 
module_type async_string 
module_source Application  #Source (Application, System, or Security)
module_eventtype Information #Severity: Error, Information,
module_eventcode # Event ID 
module_description 
module_end

In the links below you can find details of all the events that are generated on the Citrix server: 

Event Codes generated by PVS

XenDesktop Broker LogEvents

XenDesktop VDA LogEvents

XenDesktop MCS LogEvents

Web Interface LogEvents

As an example some of the most important events are listed below. Recall that all these configurations should be added to the agent configuration file (pandora_agent.conf) 

module_begin
module_name EventControllerWrongUserAccount
module_description
module_type async_string
module_logevent module_source Application
module_eventcode 1000
module_end

module_begin
module_name EventBrokerThrottleStartupException
module_descriptio
nmodule_type async_string
module_logevent
module_source Application
module_eventcode 1004
module_end 

module_begin
module_name NoDatabaseConnectivity
module_description
module_type async_string
module_logevent
module_source Application
module_eventcode 1202
module_end 

module_begin
module_name Event_1388
module_description Citrix internal monitoring
module_type async_string
module_logevent
module_source Application
module_eventcode 1388
module_application HCAService
module_end 

module_begin
module_name Event_3621
module_description Citrix cannot connect SQL server
module_type async_string
module_logevent
module_source Application
module_eventtype Error
module_eventcode 3621
module_end 

module_begin
module_name Event_3961
module_description The Data Collector is out of memory, and the Dynamic Store data might be out of sync. Please elect a new Data Collector and make sure you have enough memory on the new Data Collector
module_type async_string
module_logevent
module_source System
module_eventcode 3961
module_end 

module_begin
module_name Event_5774
module_description Server goes to grace period (Not enough licences)
module_type async_string
module_logevent
module_source system
module_eventtype Error
module_eventcode 5774
module_application MetaFrame
module_end 

module_begin
module_name Event_9014
module_description Invalid licence
module_type async_string
module_logevent
module_source system
module_eventtype Error
module_eventcode 9014
module_application MetaFrame
module_end

Modules generated by the plugin

The plugin generates the following modules:

• Users Connect.
• Citrix Sessions 

Service statuses

1.- XenApp Servers

• Citrix Independent Management Architecture.
• Citrix XML.
• Citrix XTE Server.
• Citrix CPU Utilization Management/Resource Management.
• Citrix CPU Utilization Management/User Session sync.
• Citrix CPU Utilization Management/CPU Rebalancer Estados Servicios.
• Citrix Print Manager.
• Citrix Virtual Memory Optimization.
• Citrix MF COM.
• Remote Desktop Services/Terminal Services.
• Citrix WMI.
• Citrix Client Network.
• Citrix Streaming/Citrix Streaming Helper.
•  Server.
• Netlogon.
• Remote Procedure Call (RPC).
• Workstation.
• Print Spooler.

2.- XenDesktop Services 

• Citrix AD Identity.
• Citrix Broker.
• Citrix Configuration.
• Citrix Diagnostic Facility COM Server.
• Citrix Host.
• Citrix Machine Creation.
•  Citrix Machine Identity .

3.- Provisioning Services  

• Citrix PVS PXE.
• Citrix PVS Stream.
• Citrix PVS SOAP.
• Citrix PVS TFTP .
• Citrix PVS Two Stage Boot .

4.- Licensing Services 

• Citrix License.
• Citrix Licensing WMI 

5.- Web Interface Services 

• World Wide Web Publishing Service 

Most important performance counters

Another thing we can monitor with this plugin is any of the performance counters we deem necessary. Below are the most important ones for detecting bottlenecks in Citrix and a list of all those available.