New generation of web servers based on HTTP/2 and with TLS by default: ” Caddy Web Server”.

Caddy Web Server. Monitoring of new HTTP/2 plus HTTPS

As we all know, Pandora FMS allows the monitoring of practically any device or application. Let’s talk about web content servers. Even the very popular applications or “apps”, made for the Android operating system of our phones, generally use API commands, which are also hosted on web servers to take advantage of the secure protocol (HTTPS). It is, therefore, that we must keep up with the innovations, and today we will see not only another web server program but also new paradigms in this field. Let’s go!

Let’s talk about HTTP

At the end of the last century, the Hypertext Transfer Protocol (HTTP) reached version 1.1. Since 2015 version 2.0 (improved) is available on 11% of web servers. All right, but what is HTTP?

Imagine a world where all computers have the same operating system, and also the latest version has full backward compatibility with previous versions. Imagine, moreover, that communication protocols have the same characteristics: in this utopian world a computer could perfectly request a given resource from another, say a simple text file. As both computers are completely compatible with each other, the task entrusted would not represent a major problem… in an ideal and perfect world.

Let’s open Pandora’s box and let’s start by “knocking down” half of the transmission of the requested resource to the web server: once the connection is re-established and the resource is requested again, it will be transmitted in its entirety from the beginning. If it is something small we will not waste more time, but if it is something bigger (even in a perfect world) the loss of time is irrecoverable.

Let’s think about something else; now we have the resource we want well stored on our computers and we need to check if there is a newer version on the web server. Again, the complete resource will be transmitted to us… only to discover that it is the same version that we already have stored. This is not only a waste of time, but of resources as well: to begin with, a waste of electricity, and… stop counting any other wasted elements.

We are not going to go any further, you have already imagined it: how good it would be if we had options to “talk” to the web server before it starts sending us any data or information!

For this purpose, protocols were invented, which are nothing more than forms and ways of communication, or if we wanted to abstract ourselves, they are like “languages” spoken by electronic devices. To speak in depth about this would mean to begin a university career to obtain the degree of graduate in the matter, reason why we will continue, narrating as we have been doing it throughout many of our articles.

Let’s look at something else: it’s common to talk about “plain text files”, which we all think is the simplest thing in the field of computing, and which we take for granted that our current operating systems can open without any problem. But even in this it is necessary a previous way of understanding each other, of converting all those zeros and some into letters and numbers of a high level language. These text files do not have a single character encoding; at first ASCII was used (and is still used), then ANSI and now UNICODE, with the popular UTF-8, which saves bytes in transmission and storage.

To solve the first, and also the second and much more, there is the HTTP; they are text messages with commands and questions that follow a specific format, and allows to know the capacities of the web server or to establish the encoding of characters of the resources, among many other options.

We’ll be able to send a single:

GET /index.html HTTP/1.1

(With other additional values, especially the last blank line) where, essentially, we are requesting a text file (by extension, in HTML language) and we are indicating that we “speak” version 1.1 of the protocol. The server will respond with:

HTTP/1.1 200 OK

Next to values such as date, character coding, resource type, a blank line, and the resource itself (the web page in this case) and a last blank line. We must bear in mind that all this is for the transmission of data and information. The manner in which the web browser represents or manages the resource received is subject to another article.

All this explanation was necessary to move on to technical details. In HTTP 1.1, messages that we have sent previously are carelessly sent to the web server. This means that we will have to repeat the sending of the headers, which is redundancy, and we are talking again about a waste of resources, more than anything else of time, which is the most valuable thing. This redundancy happens because it is a stateless protocol: each message is completely independent, so it is said that we cannot log in.

Logging in to any computer system involves identifying yourself and receiving a ticket that we can show each time we “talk” to the server. That’s why the web applications use cookies to identify us, which are values that act as a session (you will have noticed this with the European standards RGPD)

Another derived characteristic is that if we request a web page, we receive it and then we begin to request the images -or any other material- of that web page, each one of those requests are like new clients for the server, they are additional connections, which limit the capacity of response to the computer.

Finally, in this article (because the HTTP 1.1 is very broad) we also note that as “plain text messages” are what we send and receive, and therefore are susceptible to being read by each and every one of the devices that communicate to and from the Internet, which is known as “man-in-the-middle”, so you need the HTTPS protocol to encrypt the content and for our web server to maintain the privacy of our visitors and customers, we must acquire digital certificates, install and configure them. Quite a bit of extra work.

HTTP 2.0

Caddy Web Server handles this protocol by default. This is because this version actually allows many improvements:

• • To get started, only one connection is needed. Everything will be sent and received through this unique “pipe”, which avoids more work to the web server.
  • This was used to eliminate redundant information in each message: now the server “knows” who we are, because when opening the connection this is kept online in the style of an open session (soon we will see that the Caddy Web Server takes even more advantage of this feature).
  • You can compress resources: before you could do it if both client and server had the GZIP library installed and configured; now compress by default with the HPACK algorithm.
  • Messages are no longer plain text files, but binary data for greater efficiency.
  • We can prioritize some messages over others.
  • The server, if configured, can exit from passive mode (waiting for orders) to active mode (sending notifications that we program before certain events) to the user.
  • New vulnerabilities appear and we must configure the values to prevent greater evils. This is where the Caddy Web Server comes in.

Caddy Web Server

In Scotland, the birthplace of golf, the word caddy describes a person who performs sporadic jobs. Such people worked helping the players with their implements and received this name. Also, in this modern world, electric carts that replace people receive that noun. But here it happens that it is a registered name, so with uppercase Caddy, or Caddy web server.

It is written in Go language version 1.10 (or higher), in open source code whose repository is public in Github. We can either compile it from there or we download a batch file, which does the work for us from an especially dedicated domain:

curl -s https://getcaddy.com | bash

Also on the website we can “order on demand”: choose operating system, add-ons and features in a very fluid wizard that prepares the custom package to download and install. It is available for Android, Solaris, BSD, Linux, Windows and Mac and comes ready to use:

• We already said, you have HTTP 2.0 by default installed and configured.
• It has HTTPS support by default thanks to Let’s Encrypt, an organization that provides digital certificates at no cost and through verification mechanisms based on DNS standards: Caddy Web Server is responsible for complying with and delivering those DNS standards and also automatically renews digital certificates for us when they expire (including subdomains). Despite this, we still recommend to keep a constant monitoring of these digital certificates with Pandora FMS
• TLS is strengthened and immune to Heartbleed attacks; it includes several encryption technologies to choose from, such as ECC, ChaCha and AES-GCM, among others.
• It has support, by default, to manage several web domains (virtual hosting): this and other values are changed in a simple file called “Caddyfile”, which we will keep in each directory we need to expose and execute there with the caddy command in a terminal window (we can also execute it as a service or demon).
• It has many complements: for example, we can install one to work with Common Gateway Interface (CGI) and also FastCGI, which opens the doors for us to develop scripts in any language or reuse existing ones, such as those made in Perl language, which are widely known.
• Provides improvements in the automatic management of TLS to encrypt connections.
• For programmers: you can add a Caddy Web Server to be compiled together. I mean, it can be used as a bookstore! (Go language).
• We will be able to write Markdown code and it will be served in HTML.
• As for monitoring, in the “Caddyfile” we will add in a line “log /var/log/acces.log” (or the location we need); by default the records will be rotated. But not only can we save in a file: it also allows STDOUT/STDERR or in a remote register. What’s more, if we assign the Universal Unique Identifier to the requests, we will be able to include them in the HTTP headers and also save them in the records, along with the rest of the data.

All these features can be implemented in other web servers with additional work: with the Caddy Web Server comes everything included.

The only constant is change

In Pandora FMS we strive to keep up to date in the monitoring of all kinds of applications. Do you have any special requirements? Contact us!