What is eBPF? An in-depth look at Observability in the Linux Kernel

The Linux operating system kernel is the main program that runs after the computer boots up and allows hardware and software to “talk to each other.” Understanding this makes it easier to understand why eBPF technology leverages the operating system to implement observability, security, and network performance, which is critical to the connectivity demanded by the digitization of today’s businesses.

Linux kernel development challenges

Linux has been characterized by openness and collaboration throughout its evolution since its beginnings in the 90s, open to modifications and redistributions over time. This open-source licensing model generated contributions from all over the world, laying the foundation for collaborative development. It is clear that initially there was some skepticism about the viability and potential of Linux, since, at that time, proprietary systems were preferred and the idea of a free and collaboratively developed operating system raised doubts. All in all, the openness and inclusion of Linux attracted developers for the possibility of using, modifying and freely distributing a system, a trend that still continues today.

Solution: Extended Berkeley Packet Filter (eBPF)

The kernel’s capabilities continued to expand, driven by collaboration and collective intelligence by the contributions of developers around the world. Added to this was the adoption of the internet: in the early 90s, the internet was still in its early stages, and engineers were interested in finding ways to better understand the traffic flowing through their networks. BPF was designed to provide a way to do this without having to modify the code of all servers in an organization or fork the kernel code and update it with instrumentation instructions.

What is eBPF?

BPF originally stood for Berkeley Packet Filter and was used to record all incoming and outgoing TCP communications on a server. While this approach requires less work than modifying the code on all servers, it still required significant effort to create the code and deploy it to all servers in the organization, and it could only accept packets with a hash signed in the header. When an incoming packet arrived, the BPF program was started and the header was checked to see if the expected value was present. Depending on the results, the package was accepted or discarded. However, it was no longer just filtering that was needed, so an extended BPF version or eBPF was born.

Differences between eBPF and BPF

As we have already mentioned, BPF is currently used for multiple purposes beyond the simple analysis and filtering of network traffic, expanding its functions and compatibility with 64 bytes, which allows greater collection and storage of information. This is because, since the release of kernel 3.18, Berkeley Extended Packet Filters (eBPF) have been created to log almost all kernel events. eBPFs can store and exchange more data and be used for a variety of purposes, including monitoring, tracking, and optimizing kernel functions.
In the Linux source code, it is observed that the term BPF persists; and in tools and documentation, the terms GMP and eBPF are generally used interchangeably. It has also been seen that the original BPF is sometimes referred to as cBPF (classical BPF) to distinguish it from eBPF.

Operation of eBPF

To understand how eBPF is used, it is important to understand the key concepts:

• Predefined hooks. eBPF programs run based on the events that trigger them: an application (or the kernel) exceeds a threshold known as endpoint. Hooks are predefined and include network events, system calls or syscalls, function input and output, and kernel tracepoints. If there is no predefined binding for a particular requirement, a user probe or kernel (uprobe or kprobe) can be created.
• Program Verification. Once a link is identified, the BPF system call can be used to load the corresponding eBPF program into the Linux kernel. This involves the use of an eBPF library. When a program is loaded into the kernel, it is checked to ensure that it is safe to run, taking into account conditions such as:
  • The program can only be loaded by a privileged eBPF process (unless otherwise specified).
  • The program will not damage or block the system.
  • The program will always run until its completion (it will not remain in an endless loop).
• eBPF Maps. An eBPF program must be able to store its status and share the data collected. eBPF maps can help programs retrieve and store information according to a variety of data structures. Users may access eBPF maps through system calls, both from eBPF programs and applications. Map types are hash tables or arrays, circular buffer, stack tracking, or longer prefix matching, among others.
• Helper calls. An eBPF program cannot arbitrarily call a kernel function because compatibility needs to be maintained, so eBPFs use auxiliary functions to perform function calls. Auxiliary functions are APIs (interfaces) provided by the kernel and can be easily adjusted.
  Helper Calls allow programs to generate random numbers, receive the current date and time, access eBPF maps, manipulate forwarding logic and network packets, among others.
• Function call and tail call. These calls mean that eBPF programs can be made up of several elements. Function calls allow you to define and call functions in a program. Queue calls allow other eBPF programs to run. They can also change the execution context.

eBPF Program Execution Process

eBPF allows developers to write small programs that can be attached to multiple points in the kernel to intercept and process data. Usually written in a subset of the C programming language, programs are then compiled into the eBPF bytecode, which is then loaded into the kernel and executed. Unlike traditional kernel modules, eBPF programs are easy to deploy and manage, requiring no modifications or compilation of kernel source code.

The eBPF program follows a series of steps when running inside the Linux kernel:

• Once the eBPF program has been written, it is compiled into the eBPF bytecode. The result is a platform-separate representation of the eBPF program that can be run within the kernel.
• Next, the eBPF bytecode is loaded into the kernel. The loading process involves checking the security and compatibility of the bytecode with the running kernel, and allocating the necessary resources to run the eBPF program.
• Once the eBPF program has been loaded, it can be attached to a specific kernel event, such as a network packet or a syscall.
• When the attached kernel event takes place, it automatically runs the eBPF program. The eBPF program can access and modify data associated with the event, using standard C programming such as bloops and conditionals.
• After the eBPF program has processed the data, it can return a result to the kernel. For example, an eBPF program attached to a network interface might discard a packet if it matches certain criteria, or an eBPF program attached to a syscall might modify the arguments passed to the call before allowing it to continue.
• When the eBPF program is no longer needed, it can be downloaded from the kernel, freeing the resources allocated to the eBPF program and ensuring that it will no longer run when the attached kernel event takes place.

It should be noted that one of the key features of eBPF is its security guarantees. eBPF programs run on a restricted virtual machine within the kernel and cannot directly access or modify kernel data structures or generate security flaws or vulnerabilities. Also, eBPF programs are subject to different runtime checks, such as limit check and type check, to ensure that they do not compromise memory security or other security properties.

eBPF advantages

La implementación de eBPF tiene beneficios como:

• Speed and Performance: Speed and Performance
• Low Intrusion: Because eBPF programs run within the Linux kernel, they operate with minimal overhead compared to traditional user-space programs.
• Security: eBPF displays and explains system calls, allowing control over network processes.
• Convenience: Technology requires minimal configuration compared to other solutions so that developers can quickly deploy their applications in production environments with low effort.
• Unified traceability: With eBPF, performance data for Linux-based applications and services can be collected through programs running in kernel space. Performance issues can be fixed with this data.
• Programmability: Developers using eBPF can write new functions that previously could not have been performed due to technical limitations or security issues. With eBPF, they can write applications that take advantage of hardware acceleration in embedded devices such as Raspberry Pi (a low-cost microcomputer that can be inserted into a monitor or TV) or other ARM-based systems, which is a processor architecture widely used in mobile devices, servers, and embedded systems.
• Expressivity: Con eBPF es posible extraer datos de observabilidad de seguridad que ayuda a los desarrolladores de aplicaciones a rastrear aplicaciones, proporcionar información para la resolución de problemas de rendimiento, aplicar la seguridad en tiempo de ejecución de las aplicaciones y contenedores, entre otros.

eBPF Best Practices

• Compilation with LLVM and Clang: Clang is used to compile eBPF programs from C code and relies on LLVM as a backend to ensure correctness and security.
• Using the BCC: Toolkit BCC is a set of tools for creating efficient kernel manipulation and tracking programs and includes useful examples.
• Leveraging eBPF Auxiliary Features: Using the connection of small, efficient programs to multiple kernel hooks to gain visibility and capability monitoring, provide security to the network and application containers/microservices, as well as detect/prevent intrusions and enforce security policies.
• Using Maps to Share Data: eBPF maps can be shared between consecutive program runs, between the kernel and user space, or, depending on your use case, between different eBPF programs, no matter what interfaces they are connected to.
• Resource Limit Monitoring: Monitoring of resource capacity allows preventive actions to be implemented by IT employees.
• User Input Validation: Custom eBPF programs can be created to enforce access controls, regulate resource usage, and protect sensitive system components from tampering.
• Updated kernel maintenance: Hardware manufacturers are making steady progress, so the update makes it possible to integrate new technologies, in addition to improving performance.
• Using eBPF Libraries and Frameworks: eBPF frameworks and monitoring libraries enable efficient tracking of network processes and events, in addition to kernel debug symbol analysis.
• Code Documentation: Documentation helps developers track applications and keep track of performance troubleshooting.

Disadvantages of eBPF

• Portability Limitations: Using eBPF to perform CPU-intensive or packet-intensive processing on a per-packet basis would not be efficient because a structure would need to be built and searched for each packet, which can become costly.
• Sandboxed Program Limitations: One thing is certain: eBPF was not designed for security monitoring. Sandboxed programs have their limitations:
  • Memory: They operate within a restricted memory space.
  • Syscall: They can only perform specific syscalls allowed by the eBPF verifier.
  • Access to kernel data structures: Direct access to kernel data structures is restricted.
  • File System Access: They cannot read or write files directly.

eBPF Use Cases

• Real-time system monitoring. Unlike static indicators and counters, eBPF allows generating visibility events and the collection and aggregation of customized metrics in real time. This increases the depth of visibility that can be achieved and drastically reduces the overall system overhead.
• Computer security. eBPF allows you to detect and dig into anomalies in operations, such as an unusual processes initiated in a system and track their performance to determine what resources they are trying to access or what data is being transmitted over the network. That way, eBPF complements and helps support other processes as part of the security systems that protect your server.
• Networks. The ability to inspect all network events and network traffic flows entering and leaving a system and determine which processes interact with them helps administrators troubleshoot network issues. It also provides context that may be useful when seeking to determine whether a performance issue with an application is caused by a network event or by the application itself.
• Monitoring Application Performance. Performance metrics can be calculated for individual applications, microservices, and processes in granular form. This allows performance profiling and continuous monitoring for anomalies. With eBPF you don’t need to deploy and configure agents to track every application or process.
• Versatility. Unlike traditional monitoring tools, which focus primarily on system resource consumption, eBPF profiling provides a granular understanding of how much CPU, memory, and network resources a specific application or process is consuming. This detailed data makes it possible to identify resource-intensive applications, define possible optimizations, and make informed decisions about workload migration.

eBPF Applications in Observability

Access to low-level system events: eBPF allows the collection and aggregation in the kernel of custom metrics, through visibility events and data structures from multiple sources without having to export samples.