====== Command Center (Metaconsole) ======

{{indexmenu_n>4}}

===== Command Center =====

From Pandora FMS version 756, the synchronization system for environments with centralized mode has been redesigned from scratch, making it faster and more efficient, since the changes will be replicated to the nodes automatically without the need for manual synchronization, as it was the case up to now.

This change **deems the previous system outdated**, so in environments where it was active, it will have to go through the previous automatic [[:en:documentation:pandorafms:introduction:03_glossary#merging_code|merging]] system to use the new centralization system and be able to guarantee data integrity**.**

When updating, all already centralized Command Center environments will be forced to go through the new **Merging tool** section located in **Centralised management** to be able to be centralized again correctly.

{{  :wiki:01_system_not_centralised.png  }}

{{  :wiki:02_command_center_menu.png  }}

{{  :wiki:03_command_center.png  }}

The **Merging tool** will merge the different elements of the node and Command Center databases (of those that must be managed from Command Center) in the following way. An order of priority will be established between the Command Center nodes and the Command Center itself, placing the elements with the highest priority at the top of the list and at the bottom those with lower priority.

For example:

{{  :wiki:04_nodes_priority_order.png  }}

<WRAP center round important 60%>

Only the nodes configured in the Command Center that are not disabled are taken into account for the merging process.

</WRAP>

This priority list is used for cases where the **same element** exists in the different nodes but has **different configurations**. For example, for 2 nodes and the Command Center to have the group "Databases". With this priority order, the configuration of the highest priority element will be taken for all, in the example of the Command Center.

In another case, if for example only nodes 1 and 2 had a policy called "Windows", for all nodes and the Command Center the configuration for that policy would be that of Node 1 (we skip the Command Center because it does not have it).

<WRAP center round important 60%>

Only for the policy's own settings (group, description,…). Modules, alerts and other policy elements are considered separate elements independent of the policy and therefore are merged as well.

</WRAP>

The case of policies would be the most particular one out of all the synchronized elements due to how they are configured, since every module, alert, plugin… is dealt with as an independent element and seeing it only with modules, if you have:

\\
{{  :wiki:05_tree_view_not_merged_env.png  }}

The result of the Merging tool would be:

{{  :wiki:06_tree_view_merged_env.png  }}

This allows the result to have as many different configurations as possible so that you can now manage them from the Command Center.

==== Elements centralized by the Merging tool ====

The following elements are those centralized from the new Merging tool:

  * **Users**: It is only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same ID**  will be considered the **same user**  (following the priority rules described previously).
  * **User profiles**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name**  will be considered as the **same user **  (following the priority rules described previously).
  * **Agent groups**: They are only managed from the Command Center. Node management is disabled.
  * By unifying from the Merging tool those with the **same name**  will be considered to be from the **same group **  (following the priority rules described previously).

<WRAP center round info 60%>Make sure you adjust the parameter ''autocreate_group''  from the server configuration files ''pandora_server.conf''  by a valid group ID after unifying from the Merging tool.

</WRAP>

  * **File collections**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same short name**  will be considered to be from the **same collection **  (following the priority rules described previously).
  * **Alert template**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name**  will be considered as the **same template **  (following the priority rules described previously).
  * **Alert commands**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name**  will be considered as the **same command **  (following the priority rules described previously).
  * **Alert actions**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name**  will be considered as the **same action **  (following the priority rules described previously).
  * **//Server plugins//**  : They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name and execution**  will be considered as the **same plugin **  (following the priority rules described previously).
  * **OS**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name **will be considered as the **same OS **  (following the priority rules described previously).
  * **Module tags**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name**  will be considered as the **same tag **  (following the priority rules described previously).
  * **Module categories**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name**  will be considered as the **same category **  (following the priority rules described previously).
  * **Module groups**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name**  will be considered as the **same group **  (following the priority rules described previously).
  * **Component group**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name**  will be considered as the **same group **  (following the priority rules described previously).
  * **Network components**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name and OS**  will be considered as the **same component **  (following the priority rules described previously).
  * **Local components**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name and OS**  will be considered as the **same component **  (following the priority rules described previously).
  * **Component Template**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name**  will be considered as the **same template **  (following the priority rules described previously).
  * **Inventory module**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name and OS**  will be considered as the **same module **  (following the priority rules described previously).
  * **Monitoring policies**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name**  will be considered as the **same policy **  (following the priority rules described previously).
  * **Policy modules**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name within a policy with the same name**  will be considered as the **same module **  (following the priority rules described previously).
  * **Policy inventory modules**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name and OS within a policy with the same name**  will be considered as the **same module **  (following the priority rules described previously).
  * **//Policy plugins//**  : They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same execution within a policy with the same name**  will be considered as the **same plugin **  (following the priority rules described previously).
  * **Policy collections**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same short name within a policy with the same name**  will be considered as the **same collection **  (following the priority rules described previously).
  * **Alerts and policy external alerts**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same template on the same module name within a policy with the same name**  will be considered as the **same alert **  (following the priority rules described previously).
  * **Actions on alerts and policy external alerts**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those with the **same name on the same template on the same module name within a policy with the same name**  will be considered as the **same action **  (following the priority rules described previously).
  * **Agents within policies**: They are only managed from the Command Center. Node management is disabled. By unifying from the Merging tool those within policies with the same name will be considered as the **agents within the same policy**. Agent logs within Command Center policies will be discarded and noly node logs will be taken into account (which is where application becomes effective).
  * **Agents**: Agent management in node is allowed, except for their deletion which should be done from the Command Center.
The sections where those elements are centrally managed can only be managed from the Command Center. In case of accessing those elements from the nodes, you may only list them, and the editing and creating options will disappear. Also a warning that will indicate that the environment is in centralized mode, with a link that will lead to the administrator to the corresponding Command Center section for element configuration.

{{  :wiki:07_policies_centralised.png  }}

=== Database tables used for the Merging tool ===

The following tables are synchronized between the Command Center and nodes within the Merging tool (see also "[[:en:documentation:pandorafms:complex_environments_and_optimization:09_pandorafms_engineering#database_main_tables|Main database tables]]"):

<WRAP group> \\ <WRAP half column>

  * tgroup
  * tcollection
  * tplugin
  * tconfig_os
  * ttag
  * tcategory
  * tmodule_group
  * tnetwork_component_group
  * tnetwork_component
  * tlocal_component
  * tnetwork_profile
  * tmodule_inventory
  * talert_commands
  * talert_actions
  * talert_templates
  * talert_calendar
  * talert_special_days

</WRAP>

  * tprofile
  * tuser
  * tuser_profile
  * tpolicies
  * tpolicy_modules
  * tpolicy_modules_inventory
  * tpolicy_plugins
  * tpolicy_collections
  * tpolicy_alerts
  * tpolicy_alerts_actions
  * tautoconfig
  * tautoconfig_rules
  * tautoconfig_actions
  * tpolicy_agents

<WRAP half column>

</WRAP> \\ </WRAP>

==== Prerequisites for launching the Merging tool database merge ====

  * The **Command Center**  must be able to **connect**  to all **databases**  and all node **APIs**. Make sure that the "**Consoles setup**" configuration is correct and that the indicators are green.

{{  :wiki:08_consoles_setup.png  }}

  * **Node consoles**  must be able to **connect**  to the **Command Center database**. Usually this will not be a problem, unless you have the consoles on computers other than that of Pandora FMS servers. Make sure that the **Setup**  → **Enterprise**  configuration parameters for the Command Center on the nodes is correct.
{{  :wiki:09_node_enterprise_setup.png  }}

  * **Servers **from all nodes must be able to **connect**  to the **Command Center's API. **It is recommended to configure the public URL in the Command Center.
{{  :wiki:10_Command Center_public_url.png  }}

  * Node **servers**  must have their **API configuration **correct in ''pandora_server.conf''  or their **public ****URL **configuration in the console **Setup**. If it is not configured, servers must be hosted in the same machines their consoles are in.
<code>

console_api_url http://localhost/pandora_console/include/api.php
console_api_pass pandora

</code>

{{  :wiki:11_node_public_url.png  }}

  * Each **node **must be able to **connect **to its own **history database.**

{{  :wiki:12_node_historical_database.png  }}

  * All **nodes and the Command Center **must be from the **same version**.
  * All **nodes and the Command Center **must be in the **same MR**.

{{  :wiki:13_version_mr.png  }}

  * All **nodes and Command Center **must have the** same maximum collection size **configured in the **Setup**.

{{  :wiki:14_collection_size.png  }}

  * To avoid errors, the **Command Center and the nodes**  must have the ''memory_limit''  parameter from ''php.ini''  to ''-1'', that is, limitless, but only for the database merging process. After finishing it, it is recommended to set it back to the previous value. That is because a lot of memory is used to merge the nodes, and in very large environments (with many different elements) a large amount of memory can be used, that way you make sure that the system can use all the memory available. If the items to be merged exceed the value of the physical memory available on the server, the Merging tool will fail due to an unexpected error, and in the console/apache logs you will see the line indicating the excess memory reached.
  * All **nodes**  must have a value for the parameter ''post_max_size''  from ''php.ini'' **higher or equal**  to the value configured for the same **Command Center**  parameter. This value must be at least as big as the biggest file collection size there is. It should also be taken into account that this parameter must have a value higher than or equal, both in the nodes and in the Command Center, to the value of ''upload_max_filesize''.
  * All **nodes**  must have a value for the ''upload_max_filesize''  parameter from ''php.ini'' **higher or equal**  to the value configured for the same **Command Center**  parameter. This value must be at least as high as the size of the biggest file collection you have.
  * All **nodes and Command Center**  must have **enough space **in the disk that hosts its **directory "attachment" **  to be able to carry out **backups from the database and collections**.

{{  :wiki:15_attachment_dir_path.png  }}

  * All **nodes and Command Center **must have the computer's **date and time **correctly configured (the use of NTP servers is recommended).

<WRAP center round info 60%> \\ If all those requirements are not met, nodes will not be merged and it will return an error. If you check the result errors, it will return a message with the requirements still pending. \\ </WRAP>

<WRAP center round important 60%>It is important, once the database merging is done, to set again the corresponding value of ''memory_limit''  in file configuration ''php.ini''. Remember that for the change to take effect, the apache ''httpd''  service must be restarted</WRAP>

==== Recommendations prior to launching the Merging tool ====

Although they are notrequisites for database unification process, it is recommended to carry out the following actions too:

  * **Stop the**** pandora_server ****of all nodes and the Command Center for the whole process**. As key elements such as groups are going to be changed, their IDs can be modified, and it is not recommended to have the server process include new references to the environment while it lasts. However, the running server shouldn't be a problem in most cases.
  * **Stop the**** cron **''pandora_db'' ** p****rocess temporarily for the duration of the process**, for the same reasons as the server.
<WRAP center round important 60%> \\ When the merging process starts, both the nodes and the Command Center go into maintenance mode (not for admins). The purpose of this is the same as the recommendation to stop the servers and ''pandora_db'', to prevent a user from modifying elements during the process and for that to cause errors or inconsistencies. \\ </WRAP>

{{  :wiki:pfms-Command Center-command_center_05.png?nolink&  }}

==== Merging process execution ====

The merging process has **2 stages, **a first stage to **synchronyze the different elements **that can be managed from the Command Center and a second stage to **update the references in the events to those centralized elements. **This process is performed that way to allow the console to be accesible again as soon as possible, since event updating is part of the process that can take the longest since it usually entails more information. Both stages are in turn divided into other 2 sub-stages differentiated in 2 progress bars.

=== Stage 1 elements ===

In this stage **elements are synchronized **found in the databases from all nodes that can be managed from the Command Center. It is the merging process as such and it is sub-divided in other 2 stages, each one with its own progress bar:

  * **Initialize:**  It checks all the previous requirements, generates the corresponding backups (if the requirements are met) in case any part of the process fails, and generates the result of the database merging within the memory. If this process fails for any reason, the databases will not have been modified yet, so it will not be necessary to restore backups. Backups are stored in each node/Command Center in the directory: ''/attachment/merge_backups''
  * **Apply:**  If the previous initialization stage was successful, it will be applied to all nodes and the Command Center. This process is sequential in order of priority, so when one is finished, the next one will start.
{{  :wiki:18_merge_success.png  }}

If there is an error during this process (for example, connection loss with a database), the process itself will try to restore the generated backups (a third red progress bar will be seen that will mark the restoration progress). If the reason for the failure prevents the backups from being recovered, the recovery must be done manually.

<WRAP center round important 60%> \\ If the source of the failure prevents the backups from being recovered, the recovering shall be performed manually. \\ </WRAP>{{  :wiki:17_initialize_error.png  }}

  * Synchronization cancelled:

{{  :wiki:19_merge_error.png  }}

<WRAP center round important 60%> \\ Sometimes there might be unexpected failures, for example connection lost for a while between the Command Center and a node's database or the impossibility of creating a backup due to not having aenough disk space, so it is possible the error message shown will be generic. If that's the case and you need it, contact Pandora FMS support team to receive assistance. \\ </WRAP>

=== Stage 2: Event updating ===

In this stage **the existing references to the different synchronized elements in events will be updated **  (for example by groups). The stage is subdivided in the update of the main database events and the history database event update and will only affect those events that existed before launching the merging process. The new generated events after centralizing the nevironment will have all the correct references and won't need to be updated.

  * **Main database:**  As events are a large volume of information that is also affected, this update process takes place in parallel with the normal operation of the already merged environment. At this point, the server and ''pandora_db''  can be started again normally, and standard users are able to access the console again. Of course, you will see in the event view the update process bar for all the events, so that for that part there might still be inconsistencies (regarding filters for example) only for the events that were there before the merge. //New events would be generated normally//. This stage and process is launched by each of the nodes, through a specific task in the console's **cron**. Due to the volume of information, it can be a heavy and time-consuming task, so as far as possible the less load the environment has at that time the better (try to launch it outside of the busiest hours on Pandora FMS).
  * **History database**: It would be the continuation of the previous point, updating the events in the historical database under the same characteristics already indicated.
{{  :wiki:20_events_success.png  }}

==== Environment already centralized through Merging tool ====

Once stage 1 is finished, the environment will be considered centralized, and from there you will be able to manage everything from the Command Center. Element synchronization has also been changed, now the ''pandora_ha''  process of each node is in charge of synchronizing its database with that of the Command Center.

When you make a change in the Command Center (for example, create a user) this queues the necessary queries to the database for the nodes (''INSERTS'', ''UPDATES'', etc.) which ''pandora_ha''  reads in an orderly manner and executes in each server_threshold. This ensures that if a server is down for a while, when it is started again it can catch up correctly.

This list of pending queries can be seen from the Command Center in the **Consoles setup**. If for some reason any query fails, the node will not continue with the rest, you will see an error in **Consoles setup**  and it will be necessary to treat it manually by an administrator. In most cases you should be able to fix it by launching the merging process again in the Merging tool.

{{  :wiki:21_Command Centers_setup_sync_queue.png  }}

{{  :wiki:22_sync_queue.png  }}

=== Including new nodes ===

To add a new node to a centralized environment, go to **Setup → Metasetup → Consoles setup** in the Command Center and click on the **New node** button. All the fields must be filled in to achieve the connection and at the moment of saving, it will depend on whether it is a completely new node, without any data, it will be added with the **Register empty node** button, otherwise the **Register node with data to merge** button must be used.

{{  :wiki:pfms-command_center-new_node.png  }}

  * When using the **Register empty node**  button, a warning window will be displayed indicating that the data in the node will be deleted:

{{  :wiki:node_data_will_be_wiped_out.png  }}

Click **OK**  if you are sure and the new node will be centralized.

  * When using the **Register node with data to merge**  button, a confirmation window will be displayed indicating that the data in the existing node will be centralized:

{{  :wiki:node_data_will_be_merged.png  }}

[[:en:documentation:start|Go back to Pandora FMS index]]