Tuesday, 30 April, 2019 UTC
Summary
TL;DR
An introduction to the PHP core
PHP instrumentation can be handled in many ways. When we built our PHP agent at Sqreen we made a series of architectural decisions that enabled us to maximize performance, but also allowed us to access the internals of the language. These methods offer a powerful way to instrument any kind of function, whether directly exposed to PHP users or not.
Instrumenting PHP code
When thinking about instrumenting PHP code, it’s useful to split it up into several categories. The way you likely want to instrument internal functions is different than the way you want to instrument user-defined code, for example.
Let’s take a look at some of the decisions we made around instrumenting PHP at Sqreen. In this post, I’ll share what those decisions were and a bit about why we made them. These decisions require using plenty of PHP internal mechanisms, so let’s start with an introduction to the internals of PHP.
An introduction to the PHP core
PHP’s core is written in C. All functions that are available to the PHP user by default are defined in the PHP core and in C. For instance file_get_contents is defined in the file ext/standard/file.c. It can be used as any function defined in PHP:
It is actually part of the standard extension, which bundles most of the functions available by default to the users. When a PHP program is running, 3 main components are interacting:
- the user code (including PHP frameworks)
- the extensions (often database drivers)
- the PHP core
The request lifecycle in PHP
Generally, PHP productions are run using two main execution modes:
- In the Apache process itself, for instance using mod_php
- As stand-alone processes, using FPM (FastCGI Process Manager). A socket is used for the communication with a reverse proxy (typically Nginx).
We often see productions where a single machine is configured to run up to 1000 PHP processes. The number of processes on the machine also often scales up and down depending on the incoming traffic.
There is no multi-threading. Each process handles one request after the other, in a “fire and forget” way.
Each of these PHP processes has the following lifecycle:
The PHP extension lifecycle
When a PHP process starts, it will initialize all the configured extensions by calling the following functions, whose implementation is left to the extension:
ZEND_MODULE_GLOBALS_CTOR_DMINIT(Module Initialization)- Then several requests can be handled by this process (see below).
When the process is ready to shutdown, it will call the following:
MSHUTDOWNZEND_MODULE_GLOBALS_DTOR_D
The extension can store information that will be accessible across requests during the whole process life.
The PHP request lifecycle
When a request enters a PHP handler:
- The
RINIT(Request Initialization) function is called. - The PHP user code is then executed.
- The response is crafted and sent to the user.
- Eventually, the PHP process calls
RSHUTDOWN.
So let’s take a look at what PHP instrumentation looks like in each of these components.
Instrumenting internal PHP functions
This is the simplest situation. At Sqreen, we replaced the handler of the internal
zend_internal_function structure with a handler of ours called sqreen_function_replacer, and store the original one in sqreen_callbacks. All the internal functions should already be defined by the time we start instrumenting (forgetting for the moment the dl() function, which is irrelevant for our purposes). Then, when our handler is called, it begins by looking up the associated sqreen_callbacks object of the called function in sqreened_function_table. At this point, Sqreen code is wrapping the original function.Instrumenting user-defined PHP code
User-defined code needs to be instrumented using a different behavior, which requires diving deep in the PHP engine. Note that this code can also be dynamically defined at any point of a request rendering (e.g. by using the eval keyword).
In our PHP agent, we hook into a late phase of the compilation of user functions (just before the “second pass”, where, depending on the ABI that PHP was compiled with, certain optimizations involving offsets are made). PHP user functions can be defined at any point during the request and interception at compilation time allows for much better performance than intercepting all the function calls (like PHP debuggers do, with a big performance penalty).
When the compilation happens,
sqreened_function_table already has an entry, which is how we know we should instrument the function being compiled. After interception of the compilation, the pre-second pass user code is used to create a separate function/method, with a different name (i.e., the function being defined is renamed) and the user code of the function being compiled is completely replaced with wrapper code. This wrapper code is expressed in PHP code and compiled at that point (this ensures better cross-version compatibility than hardcoding bytecode). This is an example of the wrapper code (global functions in PHP 5.6+):The wrapper code has the renamed name of the original function hardcoded, so it can call it. Some post-processing is needed on both functions, for instance to reverse the second pass on the wrapper, to do the second pass on the second function, and to support returning and passing by reference.
More information about PHP 7 compilation methods can be found here.
Instrumenting eval and include
The
eval and include are not defined as methods but as opcode. User code is compiled down into very simple instructions by the PHP engine. These instructions are called opcodes. Opcodes are executed instead of the original user source code, and each one has a specific meaning, like calling a function, assigning a value to a variable, or adding 2 numeric values.
PHP extensions can define specific opcode handlers. It means that when the user code is compiled down to opcodes, the extension’s opcode handler will be called, which allows us to use a specific flow for this opcode.
In the case of dynamic instrumentation, any function call could be overridden with another function.
Instrumenting other PHP extensions
Other functions of interest for hooking are not exposed in PHP. For instance, database drivers, such as mysqlnd, expose very high level functions to the user, while we’d rather focus on hooking some lower level helpers. In this case, the best thing to do is to use the hooking capabilities provided by the driver. They require hooks to be written in C.
Recovering from callback errors
Errors that occur in PHP callbacks are easy to recover from given PHP’s internal safeguards and exception handling.
The errors that occur in a C callback are much harder to deal with, unless you’re using signal interception (which you do not want to do on a customer’s machine!).
There are two strategies for avoiding callback errors in C:
- Leverage high quality development methods
- Offload complex business processing to the daemon
Allowing arbitrary callbacks
Given the instrumentation primitives, an interface is needed to make it safe and easy – yet performant – to add arbitrary code. We do this via callbacks.
First, various callbacks can be set on the same instrumented function. They are stored in a list dedicated to this function. The access to this list is critical: adding and removing callbacks should not interfere with the execution.
Special care should be given to the way any callback is executed. Since the callbacks are arbitrary, no assumptions can be made about them. There is a chance that a callback, during its execution, makes use of the instrumented method (e.g., a log method is instrumented, but the callback needs to log something itself). Without proper safeguards, this would enter an infinite instrumentation loop.
Eventually, the callbacks can be set in 3 different positions:
The three different callback positions
This code is an overview of how callbacks can be called on an instrumented method:
Re-instrumenting
If a new callback needs to be added to an already instrumented method, or if a callback needs to be modified, the agent detects that the method is already instrumented and will just replace its callbacks.
De-instrumenting
Once the agent decides to remove all instrumentation, the
sqreened_function_table is iterated and each corresponding item in PHP’s function_table is replaced with its original PHP function.For C code (such as SQL drivers), most of the time only the callback list is emptied – which means the Sqreen hook is still running, but is not doing anything since no callbacks are defined anymore. The overhead in this case is invisible.
Using a daemon
The fact that the extension cannot store any state between requests would require a naive implementation to send information to the Sqreen backend each time a request is processed, which would be terrible for the application performance.
In order to gather all the security signals needed from the request processing, we built a daemon, running close to the processes handling requests, which receives and aggregates the signal.
Detecting the daemon
During
RINIT, the extension tries to connect (via TCP) to the daemon. If the connection cannot be made, the request is left unprotected and the extension tries to launch the daemon. The connection attempt is made with a short timeout in order to prevent hanging a customer request.During the next requests, the extension will try to connect again using an exponential backoff: if the first connection is unsuccessful, it will wait before the next attempt for 10 ms, then 20 ms, then 40 ms, etc. This failsafe mechanism allows the extension to keep a good level of performance while preventing running these connections attempts in a loop.
Launching the daemon
The daemon should be started by the init process at the machine startup. Though in some cases (e.g. Docker) our customers don’t have it, or don’t use init. The PHP extension can launch it automatically if it isn’t launched or doesn’t allow the extension to connect there.
Scaling the daemon
One instance of the daemon can handle up to a certain amount of customers for performance reasons. When the number of connections to the daemon becomes too high, it will fork and the new connections will be handled by this new process.
Communication with the extension
The extension and the daemon use TCP sockets for communication. Messages are transmitted using msgpack (a data format close to JSON, but designed with speed and size in mind). This efficient and flexible format prevents data duplication. Each message is prefixed by the sender version number, so both the agent and the extension know who they are talking to.
Robustness
The robustness of the Sqreen PHP agent relies a lot on the robustness of the daemon. The daemon is itself based on the Python agent, and leverages all the security and robustness measures we put into it. More details about our Python agent can be found here.
Besides dynamic instrumentation, the Sqreen PHP extension relies 100% on existing PHP mechanisms. For instance, the “fire and forget” philosophy is kept as is: all data is flushed to the daemon once the PHP customer’s response is answered, so no state is kept (beside what the extension explicitly decided to store).
Obviously, crashes in compiled languages are hard. We take very special care of programming with failure in mind and leveraging all static and dynamic tools of the C ecosystem, such as the Clang analyzer & address sanitizer, gcov, Valgrind, etc. This approach helps us achieve higher levels of robustness within our agent and daemon.
Memory usage
The PHP core introduces a powerful mechanism to handle memory management. In the per-request business code, the memory doesn’t need to survive the next request. In this case, the PHP memory allocator offers helpers such as emalloc, ecalloc, and efree which are handled by the PHP allocator. Two nice things are offered to the user:
- No need to check if the allocation is successful — if the memory allocation wasn’t successful, then PHP will abort the whole request rendering rather than returning an error code.
- Memory leaks have much less impact since the PHP allocator will recycle the memory at the end of each request.
We’ve described in this post the high-level concepts of an industrial-grade instrumentation agent through the lens of our PHP instrumentation decisions at Sqreen. It’s the details of the implementation though that really make the difference.
At Sqreen, our PHP agent leverages instrumentation techniques in order to protect PHP applications at runtime against security events. Sqreen helps developers get full visibility and protection against security threats. Cyberattacks are blocked at runtime without traffic redirection or code modification. Suspicious and fraudulent activities from/targeting user accounts are identified to detect attackers early.
Feel free to ask questions if you want to know more about PHP instrumentation, or how we do it at Sqreen! If you’re interested in joining us to work on this agent or ones like it, send us a message at [email protected].
About the authors
Jean-Baptiste Aviat spent half a decade hunting vulnerabilities at Apple, helping developers solve them, and developing security software. He is now CTO at Sqreen.
Gustavo Lopes is a software engineer with a broad skillset: from low-level (C, debugging, instrumentation, security), to Java middleware, web frontend (Vue.js) and configuration management. He worked for several years on the PHP engine and is involved in the development of many parts of the Sqreen agents, including Ruby, Java, PHP and V8 bridges.
The post Building a dynamic instrumentation agent for PHP appeared first on Sqreen Blog | Modern Application Security.