Engineering and development in Pandora FMS

Pandora FMS started as a totally personal open source project back in 2004. I wasn’t even a professional programmer, I was doing Unix security consulting. In fact, I chose PHP but Pandora FMS was my first application with PHP, I knew some things about ASP and my favorite programming language had been C.

A project with a single programmer and no professional users of his software yet is very different from a project with several dozen programmers and hundreds of clients using the software in critical environments. The evolution that Pandora FMS has undergone from 2004 to 2021 is a real case of steady improvement in software engineering.

Fortunately, I did not pay much attention to that subject of the degree, because most of the things that work and that I have learned with practice do not come in a book, nor are they explained at the university, because each software project and each team of people is very different. It may sound cliché, but it is the truth, and it is better to accept it and avoid formulas, because building a solid software product that can grow over time is not trivial at all.

In this article, I am going to talk about our experience, our evolution over time, but above all, about how our engineering processes work today. I have always believed that the most important part of open source is transparency, and that this should apply to everything, not only to software but also to processes and knowledge in general.

Version control system

It is an essential part of any software project. Today the ubiquitous GIT is everywhere (by the way, not everyone knows that Git is the work of Linus Torvalds, original author of the Linux kernel). A version control system helps, in short, a group of developers work without overlapping their jobs.

When the Pandora FMS project started, I was working without version control, because there were no other people. When some people began to collaborate on it, we realized that a simple shared directory was not worth it, because we were overlapping the code and, yes, making backups to save old versions was not a very efficient method.

The first version control system we used was CVS, which we have been using for eight years or more. Around 2008, we started using SVN (Subversion) another slightly more efficient system and it wasn’t until 2013 when we started using GIT and opened our official repository on Github.

Pandora FMS public repository on Github

Since Pandora FMS has an open source version and an Enterprise version -with proprietary code and commercial licenses- we have two GIT projects, one public on GitHub and the other private, which we manage with GitLab. The GitHub version is in sync with our private copy on GitLab at our offices. Some partners who collaborate with us in developing have access to this private repository, and through an extension of our support application (Pandora ITSM) we share all development planning tickets by releases with some of our partners, so that they can see in real time, the development planning based on “releases” and all the details of each ticket.

GitLab ticket view in Pandora ITSM/em>

Release ticket view

Development methodology used in Pandora FMS

At Pandora FMS, we have been using our own methodology from the beginning, although we have borrowed many ideas from agile methodologies, especially from SCRUM. From a life cycle point of view, we use an adaptation of the Rolling Release methodology

These are some important definitions when defining how we work, some of them come from Scrum, others from other methodologies.

Objectives of Pandora FMS work methodology

The objectives involve not only the development members, but also QA, the documentation team and part of the marketing team:

• • Maximum visualization: The entire team must see the same information, and it must flow from bottom to top and from top to bottom. By sharing objectives we will be able to do a more effective job.
  • What is not seen does not exist, which implies that all information relevant to the project must be reflected in the management, implemented with Gitlab. What is not seen does not exist, and what does not exist will not be taken into account for any purpose. Strictly following this methodology will allow everyone to be very aware of the planning:

-Strict deadline compliance.

-Advance planning without last minute modifications.

-Clearer information and in due time.

-Elimination of work peaks and etc.

• Integrity,, with an increasingly large and complex project, it is imperative to keep integrity during development. All code must follow standards..

 

Ticket

The ticket is the minimum work unit. There is a single person responsible for its completion and it is planned to be carried out in a milestone (version release).

A ticket is the way in which the development work is broken down, so a big feature will be made up by different tickets, on which ideally several people can work.

The ticket must contain a functional or description of the requirements, which can include diagrams, specifications, interface diagrams (mockup), test sets, examples, etc. In some cases it may even contain the analysis and design of the whole solution.

A completed ticket must perform as specified in the functional document (ticket) and the changes that have been made to these specifications must be reflected in the ticket.

The functional is key so that QA can validate a ticket or not. QA will have to reopen a ticket if it does not meet any of the functional aspects.

Members and working groups

Product Owner (PO)

The PO defines where Pandora FMS has to go, in contact with customers, support and
the “real” market situation, providing technical and functional guidelines but without getting involved in development as such.

Product Committee

Group of people who will meet permanently with the PO to agree where the product is going to, trying to ensure that all PO decisions are collegiate. It is made up of the leader of each Development, QA, Support, Projects and Documentation team.

Development Manager (DM)

The DM will manage the entire development cycle: define milestones, priorities, manage
individually all members and make operational decisions. The DM reports exclusively to PO and is the leader of the development team.

Development Team

They are in charge of the development of large features and product improvements, complete code refactoring, change development (small features), bug fixes and product maintenance improvements.

QA Team

They verify that each development atomic unit works as defined in the
specifications. They will also create and maintain an ecosystem of automated testing for both backend and user experience.

Support Team

They are the ones who deal directly with the client solving issues. Their experience with the product’s day-to-day means that their opinions must be taken into account, that is why they are part of the product committee.

Project team

They implement it on the end customer and are the ones closest to the customer, since they are often there before the project exists, and they usually offer ideas and all kinds of features in hand, for all purposes they are the “speaker” of the commercial department, therefore they are part of the product committee.

Training and Documentation Team

Responsible for training and the product’s documentation. They coordinate with the marketing team and the translation team.

Remote working

All team members (development, QA, documentation) telework freely. In fact, developers from Europe, Asia and America participate in Pandora FMS, and within Spain they are distributed throughout the national territory. We are a 100% distributed and decentralized company, although with traditional hierarchies.

In order to telework, we need each member to take responsibility for their work, be autonomous and commit to planning. Teleworking entails minimizing the need for oral communication and physical personal meeting, replacing them not with teleconferences, but with a precise use of the tools of the development process.

Development watch-keeping

A developer on the team is especially devoted to solving incidences involving code, in permanent connection with the support team (from 8 am to 8 pm, CEST). This allows not only to have maximum agility when solving a problem on a client, but also code changes are integrated into the code repository in an organized way.

Ticket creation and classification process

Any member of the company (including salespeople) can create a ticket in GitLab. This includes customers and partners, although in their case there is a prior filter by the support team and the sales team respectively.

The more detailed the ticket, the more unequivocal the development will be. Add images, gifs, animations and all the necessary clarifications. As well as the way to access the environment where the problem has been found or the contact persons. A developer will never contact a customer directly. If there is the need to interact with them, it will be done through the support or project team.

Nobody, except for the DM or PO, can change a ticket milestone. On creation, the ticket will not have an assigned milestone or assigned user. The task of defining which release a ticket belongs to is the responsibility of PO and DM exclusively.

When a ticket is finished and the developer thinks it should be reviewed by a colleague, they mention it in the merge request through @xxxxx. The review must be nominal. This review is independent of the code review carried out by the department manager.

General ticket workflow

• The ticket is assigned to a programmer by the DM. If it does not have a ticket assigned, the ticket will be auto-assigned. (See below the terms that regulate this system).
• The developer must understand/solve any questions that may arise after reading the functional document, if necessary, check with the DM or the author of the ticket. This must be done before starting to develop. Once read, you must, in order:

1. Evaluate (by assigning labels) its complexity and size, reaching a prior consensus with the DM.
2. Develop the feature following the ticket specifications
3. Document everything developed in the same ticket or, if required, in a new documentation ticket. This ticket must relate to the “parent” ticket by ticket #ID.
4. The developer must test its functionality at least in:
   -standard docker development environment
   -docker development environment with data.

• When it is deemed complete, it will be tagged ~ QA Pending and placed in the hands of QA.
• For each FEATURE ticket, there will be a reference person, generally from projects, support or even the PO itself. This person will be the one who will define part of the functional (together with the DM and PO), but above all, this person will be the reference person for the developer to ask any details during development, and most importantly, should see the development progress, step by step, so that it is validated.
• Any change to the functional will be reflected by the reference person in the ticket as comments, without altering the original functional.
• If there is a child documentation ticket, QA will validate the ticket using the documentation generated by the reference person, NOT by the functional of the ticket, validating the documentation and the feature at the same time.

Release planning

When creating a ticket, the milestone must be empty (not assigned) like the user. The only ones that can classify a ticket are: DM and PO.

A series of milestones have been defined to support the ticket classification process, some of them, those dated (releases), can be seen as milestones, while the rest should be seen as simple ticket containers.

• (Not allocated): It is the absence of milestones in a ticket. For all intents and purposes, this ticket “does not exist yet.” The DM and PO will validate each and every one of these tickets to see if they make sense in the product roadmap. No developer should take any of these tickets.
• Feature backlog: Tickets that will be made at some indeterminate time in the future that sooner or later will have to be addressed. No developer should take any of these tickets.
• Low priority bugs: Reported bugs with no priority assigned yet by PO/DM. No developer should take any of these tickets.
• STAGE: Tickets proposed by each department for planning in a product release. At each planning meeting, these tickets will be discussed, and moved to other milestones. At the end of the cycle start meeting, this milestone should be empty. The DM is the one who has the final decision as to which STAGE tickets are assigned to a certain release and which are not, relying on the product committee if necessary. No developer should take any of these tickets.
• XXX: Release XXX. Milestone that groups a series of tickets that will be released on a certain date. A milestone has a deadline associated with it. In the case of RRR releases, this date could change, in the case of LTS not.

1. The development of the tickets associated with a release must be finished 5 days before the scheduled day for the release. Tickets not completed before that date will be delayed to the next release and the delay will have to be justified to the DM.
2. There are two types of release milestones:
   -LTS: in April and November. They are 6 months apart.
   -Regular Releases (RRR): There will be 2 to 4 regular releases between LTS releases.

• A developer with no assigned tasks for a release, as long as there are no pending assignment tickets in the release milestones for the developer’s team, can take one of the unassigned tickets from:
  -The closest release, based on date.
  -Second closest release, based on date.

CICD

Pandora FMS developers integrate the code of their branches in a central repository several times a day, causing a series of automatic tests to be executed whose objective is to detect faults as soon as possible and improve the quality of the product.

These tests run dynamically in a series of executors or “runners”, some of them specific, for certain architectures (e.g., ARM), that execute static code analyzers, unit tests, and activate containers to carry out integration tests in a real installation of the application.

The generation of Pandora FMS packages is completely automated. Packages are generated every night from the development branch for manual testing. They can also be generated on demand by any developer or member of the QA or support teams, from any branch through the GitLab web interface.

When a release is made from the stable branch, in addition to package generation, a series of steps are executed that deploy them to Ártica’s internal package server, to SourceForge, to Ártica’s customer support environment, and that, likewise, update the Debian, SUSE and CentOS repositories along with the official Docker images.