Avoid doing the work that can be done by a machine, automate software development!

This article was first published in 2018, some technical aspects may be outdated.

Continuous Integration Software: Learn all about Jenkins

In a previous post we explained all about Continuous Release of Software and today we will see the previous step: the Continuous Integration of software but from the point of view of automated tools . We chose Jenkins for didactic purposes since it is written as a free software license and it is very flexible, just like Pandora FMS.
First of all, we will talk about some practical tips to contrast it with Jenkins, which will also work for any other similar software such as Buildbot or GitLab CI or even more advanced tools such as Concourse or Drone(in fact, there is a Jenkins plug-in for Drone) thus combining Continuous Integration and Continuous Release of software.

Continuous Integration (Change is the only constant)

Continuous Integration is hard to implement but once it is ready, programmers always wonder: how did we manage to live without this in the past? In order to explain its operation, we must briefly look at what a software repository and a version control system are.

How does a software repository work?

A software repository gathers all the necessary files so that a program or system can be compiled or installed on a computer. If a version control software is responsible for its administration, then we will have the perfect combination for a new programmer to start working on the code development (this is known as process chain).

Nowadays, the most widespread version control system written in free software is Git, which allows us to have a distributed copy of all the code used in an application or project. So a new employee can obtain a copy of the whole project, make the modifications in that copy, and once completed, the employee is able to perform compilation tests (convert high-level language with low level language or machine language) and the employee is also able to test databases or data processing.
Once you know that everything is fine, we will have to check if the project has been modified, in this case it will download (automatically) modified files and a summary will be submitted if these updates affect the work that you have just made. In this case, it will correct, assimilate and combine the new code (this usually happens when two programmers work simultaneously in one code)
After all this, it will compile and it will try again and finally it will “upload” its contribution to the repertory, but wait, there’s more: and now let’s say hi to Jenkins, this is the software that will be responsible for carrying out the tests automatically and once approved, its work will be finished (and then the cycle will be repeated).

Tips for the Continuous Integration

We have collected the best tips in order to obtain the best results in the Continuous Integration of software; don’t forget to leave a comment in the comment section if you think there’s something missing.

• Place everything in the main repository: Excepting credentials, passwords and private codes. Everything else must be in the repository, the source code and the test scripts, property files, database structures, installation scripts and third – party libraries. Programmers should be warned that temporary files or collections made by programming environments (which are easily recognizable by their file extensions) are not “uploaded” to the main repository.
• A main trunk with very few branches: Going back to the example of the new employee as a programmer, it is good to create a development branch to assign a job, perhaps, in order to correct a bug. This branch must have short duration and then it must be added to the main trunk, and it will even be tested twice with Jenkins.
• Discipline: The work of programmers must be carried out accurately and it must contribute to the repository as often as possible.
• The Golden rule: The longer the time spent without compiling, the harder it will be to find the bugs , and in the worst possible scenario these could overlap with each other, thus making it difficult to detect them.
• The compilation tests must be quick: Around 10 to 20 minutes after uploading the contribution of the programmer, an approval or rejection must be issued, graphically on a web page or by email. In order to achieve this, we must look at special considerations as soon as we look into Jenkins.

Automating processes with Jenkins

Our Continuous Integration of Software, when applied to our chain of processes, must set us free from the work of pre-compilation, compilation and even installation (creation of databases, configuration of predetermined values, etc.). For this, Jenkins, which is software whose logo shows an English butler who is ready to help us in our daily lives, is written in free software under the permissive MIT license and it runs on GNU / Linux.

It was formerly known as the Hudson project and it was made with the Java language, it was created by the company Sun Microsystems in 2005 and eventually it became the property of the Oracle company. With the privatization of the company, Jenkins was born in 2011.

It’s quite easy to install: you need to add the public key from your website, and then add it to the list of repositories and then install it from the command line. Jenkins offers a web interface, therefore we need to install Apache or Ngix as a web server as well as an email engine for notifications, and once enabled it will let us install the main plugins in order to work.

For Jenkins, most things are plugins, including the chain of processes. Due to the amount of plugins that are available, it is difficult to replicate Jenkins environments on other computers. We can confirm our Continuous Integration through the Apache Groovy language, which is derived and simplified from Java or you can create those you need through the web interface.

Here, we get to see the versatility of Jenkins: we simply create, using a Plugin, a folder or directory called Pandora FMS to store there the customized files related to the Continuous Integration process, but not the source code itself. Then we must make a practical use of it: if we have our hosted project on GitHub, by using a plugin, we can specifically tell our user ID (and password if the code is private) and in one or two hours Jenkins will take care of analysing our repositories through the API of GitHub (in fact this API limits the use by time, that is why Jenkins takes so long when it comes to downloading, respecting the “rules of etiquette” when it comes to the use of resources).

Declaring our identifiers in our source code
Once Jenkins is connected to our repository and the analysis is done, we will see the following :

For example, if we use Python language we must include the following file with the extension “.jenkins”:
/* Requires the Docker Pipeline plugin */
node('docker') {
checkout scm stage('Build') {
docker.image('python:3.5.1').inside {
sh 'python --version' }
}
}
Jenkins needs this in order to create an environment contained in Docker , a software that consumes fewer resources than a virtual machine and offers the same advantages, thus Jenkins can create a test scenario for the specified language and it can run python–version .

Specifying our tests for continuous integration

Now we will add our scripts and although it will take a while to create them, this way we will save a lot of time since it automates the tasks of compilation and testing. Here we have an example on how to instruct Jenkins in GNU / Linux with sh :
pipeline {
agent any stages {
stage('Build') {
steps {
sh 'echo "Hola Mundo"'
sh ''' echo "Múltiples líneas permiten escalonar el trabajo"
ls –lah
'''
}
}
}
}

Private environments also have support, for Microsoft Windows we will use the bat process:
pipeline {
agent any
stages {
stage('Build') {
steps {
bat 'set'
}
}
}
}

Likewise we will make the respective scripts thus allowing each programming language to execute the respective compilers, just to name a few (free and privative):

• Apache Ant for Java language.
• MSBuild for the Visual Studio suite (MsBuild does not need the Integrated Development Environment (IDE) – to perform this task).
• Nant aimed at .NET framework .
• Rake for Ruby.

Hardware for our battery of tests

Today hardware has lagged behind compared to virtual machines; we can run several of them with only one computer. Jenkins needs an infrastructure in order to do its work in a few minutes and now we want to recommend the following:

• A local area network: which must be well managed and planned (automated) with PHIpam .
• A repository of Operating Systems (OS): with ISO images of the OS needed, as well as a repository and/or proxy server with updates for them.
• We must have a manual integration machine: a human being must corroborate Jenkins‘ daily work.
• We must have one or several machines with accumulated compilations: especially if our project works on several platforms or operating systems, we will then make the necessary scripts to partially compile the integrated differences throughout the day, at a first level, this guarantees us fast approvals of Jenkins.
• Several machines for nocturnal executions: on “clean” and also “updated” OS but these must not have received our project (we will make scripts for these cases too) obtaining approvals at a second level.
• Machines with heavy software, which is already installed, and stable: such as database, a Java environment or even web servers, in order to be used by the rest of the computers that carry out tests.
• Model integration machines: which are configured with the minimum, recommended and maximum hardware that we officially recommend for our project.
• Identical machines (as much as possible) to the servers in production: same amount of RAM, cores, video memory, IP address, etc.

In all these scenarios, we can draw on virtual computers, in order to test in real machines through the Continuous Implementation of software.

At this point we recommend Pandora FMS on the issue of monitoring the continuous integration of software as a way to measure the stress on the systems with our new project that is installed. The fact that this is correctly compiled and installed it does not mean that it is doing what it should be doing; in fact in Pandora FMS we offer the Pandora Web Robot (PWR) for web applications and Pandora Desktop Robot (PDR) for desktop applications and in order to test those scenarios with detailed and accurate reports.

Conclusion

As you know, software development in the 21st century is in the pre-stage of software creating software, but that already enters the field of artificial intelligence.
Meanwhile, nowadays, in Pandora FMS from version 7.0 (Next Generation) we use rolling release and we adapt to new technologies thanks to our flexibility, do you want to find out more? Click on this link: https://pandorafms.com

Do you have any questions or comments about Pandora FMS?
Get in touch with us for more information: https://pandorafms.com/company/contact/