The Twitter situation, monitoring and encryption
This post is also available in : Spanish
Supervening event: The Twitter case, monitoring and encryption
An event like the one that happened to Twitter on May 3, 2018, can happen to any company. Through a monitoring process? Let’s study the matter!
Unexpected event: an introduction
Surely you use the popular social network Twitter®, for example to receive Pandora FMS alert messages for your system, in order to know in a timely manner the issues that require human intervention, which is always necessary. On Thursday, May 3, 2018 Twitter® requested in an urgent way (direct message on screen) that we should change our access password. Yes, the message was valid, as we were browsing with secure protocol (TLS). However, we proceeded with caution before taking any action.
Here, we talked about dealing with the crises of Information Technologies and we talked about how Twitter® told us that there was a problem and they were working on its solution. Well, we smiled. They complied with one of our five tips. This article describes – and we will try to analyze – the surprise for the 330 million users that make up the social ecosystem. Let’s go!
Antecedents of this unexpected event
It is not the first time that a glitch software affects us, nor will it be the last. Thanks to monitoring, an incident caused by a power failure can lead us to discover mistakes in the conduction and administration of the systems. It is true: the event is untimely, but could it have been prevented or at least minimized? In this case, Twitter® again meets our third recommendation: be transparent. This is the screenshot of the message we received:
As we can see (if necessary, right click with your mouse to open the image in a separate tab or window), they explain in a direct and sincere way: We recently found a bug that stored passwords unmasked in an internal log. Here’s a link where you can get more information, but guess what? Twitter® meets another one of our recommendations! The fourth premise: good communication. Directly, it indicates the nature of the problem without going into too many technical details, only the minimum necessary. Maybe you’re wondering, where’s the technical detail in this report?
Encrypting, chopping, obfuscating… are these synonyms?
A “non-hidden password” is a password stored naturally. We don’t need anything else to read them: we call that “plain text”. Of course this in no way suits anyone or anything, and Twitter® is no exception. But this company, shines again by offering a good apology. In their Twitter® blog, the same Chief Technology Officer (“CTO”) literally explains the problem in greater detail, in a clear and direct way (you can read their blog entry.
Parag Agrawal, CTO of Twitter®, a software engineer who graduated from Stanford University and has worked for large companies such as Yahoo!®, Microsoft® and AT&T® Laboratories. With the help of the Twitter® blog – hence the importance for all companies to always maintain an official blog – the company was able to offer a good apology by complying with our number one recommendation.
In addition, we also got a glimpse into two key components: the way – and standard – about how the technology industry stores our access passwords and the programming components involved. For this reason, we quote:
“We mask passwords through a process called hashing using a function known as bcrypt, which replaces the current password with a random set of numbers and letters stored in the Twitter® system. This allows our systems to validate your account credentials without revealing your password. This is an industry standard.”
“Due to an error, passwords were written into an internal register before completing the hashing process. We found this error ourselves, removed the passwords and are implementing plans to prevent this error from happening again.”
Yes, they are synonyms for the Royal Spanish Academy
But for us, programmers and system administrators, no. And this does not diminish or invalidate that dictionary. What happens is that the information addressed to the general public is, and always will be, different from that addressed to the specialized public. In computer science we use a process that practically does not allow us to decipher this information. How is this possible? Who can benefit from this process?
This is why we highlight the word hash, which means to cut, divide or chop into pieces, as when one breaks a paper letter into many pieces. Who could read that? But with the advent of computers, splitting a message – or a password, which is the case here – and storing it on different sites also doesn’t guarantee any security; they could intercept our message pieces on the server in charge of identifying our logging in users. The problem remains the same as for Twitter®: the password is still stored in plain text.
Of course, Twitter® saves our hashed passwords, so it actually compares both results with the same function to assume that the user is who they say they are. In other words, when we open a user account on most websites and set up a password, it is hashed and stored in a database. Note that we use an algorithm for this work and we will use it again in the future when we log in again: we re-enter our password, we apply the mathematical formulas and the result (a string of characters) must be the same and identical to the deposited one (the process we use must guarantee that there are no collisions: two different passwords produce the same result -hash-).
This explains why, when we lose our password (event occurred) to the banks, they don’t tell us what it was, because they simply can’t decipher it themselves. It’s easier to erase it and for us to place a new one. We will probably be given a provisional, disposable key, which we will use only once to perform this task, but it is strictly temporary and is stored and handled in a totally different way from the normal identification process. In fact, by using that disposable password we will not enter our bank account, but a message will tell us that the new definitive password was set and that we log in through the regular channels.
We can install bcrypt on any of our GNU/Linux systems, using the command “apt-get install bcrypt”:
Then we can use the man bcrypt command to know how it works (in Debian, since version 8 “Jessie”, the ability to encrypt files has been disabled). We’ll see something like this and we’ll soon notice that your job is to encrypt files using the blowfish algorithm:
But the most surprising thing is that you also DOWNLOAD those files! Here we see two weak points: the use of files to pass information and the fact that information can be decrypted. Bcrypt states that the source files, if successfully encrypted, will be compressed and then rewritten with random characters before being deleted. (Remember the TV series of the 1960s: “This message will self-destruct in five seconds”. Unexpected event.
We are particularly concerned since bcrypt allows passwords to be decrypted and we made this known to Mr Parag via Twitter.
Monitoring the process of “escaping” passwords
We clarify that in the science -and art- of monitoring systems, the intrinsic work consists of “extracting” data to convert them into information and in no way is any other use contemplated for the data. We also insist that Twitter® does not explicitly and directly store in plain text format. In this aspect, we repeat that Pandora FMS selects the data to monitor instead of filtering the data. There is monitoring software that collects all the data stored in the logs of each computer in production and we can infer that it was at this point when they noticed that those discarded files were in plain text.
While it is true, from the programmer’s point of view, when in development mode, recording absolutely everything when moving to the production environment, conditional compilation should be used to prevent private user information from becoming public information. These standards are well established in the world of web application and software developers. The easiest way is to indicate in the “About” option, in each software, whether it is in development mode or in production mode. This is accomplished with specific conditional compilation variables and instructions.
We find it hard to believe that bcrypt was the “culprit” or cause of the event, but rather the way of delivering the files to be encrypted (unless we use the “-c -r” parameters that avoid compressing and deleting). We imagine that each computer stores the user’s password in plain text in a local way -a queue of files- which is then sent protected by TLS to a server in charge of encrypting. The process of encrypting any data or information has an additional work that weighs on the performance of the system, even works in a “light cipher” for “the Internet of Things“. In the article published in your blog about Solaris we indicate that this operating system offers support for computers with auxiliary SPARC M7 chips, dedicated exclusively to the mathematical work necessary to encrypt absolutely everything, including the transmission of data and information about monitoring.
In March 2017 we published an article about the SNMP protocol. Among the features that make this version different is the encrypted communication between computers (in the Pandora FMS NG 722 version we support version 3 in polling mode while maintaining the incredible speed). Obviously encryption and decryption are additional steps and they need the management of credentials, the saving of passwords and here we go again…
We have briefly presented an approach to the mathematical profession of coding and hashing data and information and the unexpected event. The information presented here in no way is, nor is it intended to represent, the position of Pandora FMS and its developers: everything stated here is the sole and exclusive responsibility of its author. Let us know your comments below or, contact us at this link for more information about Pandora FMS.
Programador desde 1993 en KS7000.net.ve (desde 2014 soluciones en software libre para farmacias comerciales en Venezuela). Escribe regularmente para Pandora FMS y ofrece consejos en el foro. También colaborador entusiasta en Wikipedia y Wikidata.
Machacador de hierros en gimnasios y cuando puede se ejercita en ciclismo también. Fanático de la ciencia ficción.
Programmer since 1993 in KS7000.net.ve (since 2014 free software solutions for commercial pharmacies in Venezuela). He writes regularly for Pandora FMS and offers advice in the forum. Also an enthusiastic contributor to Wikipedia and Wikidata.
Crusher of irons in gyms and when he can he exercises in cycling as well. Science fiction fan.