Zabbix - A Case Study of Unsafe Session Storage

Introduction

Zabbix is a very popular open-source monitoring platform used to collect, centralize and track metrics like CPU load and network traffic across entire infrastructures. It is very similar to solutions like Pandora FMS and Nagios. Because of its popularity, features and its privileged position in most company’s networks, Zabbix is a high-profile target for threat actors. A public vulnerability broker, a company specialized in the acquisition of security bugs, also publicly announced their interest in this software. 

We discovered a high-severity vulnerability in Zabbix’s implementation of client-side sessions that could lead to the compromise of complete networks. In this article, we give an introduction to the different kinds of session storage and discuss what makes an implementation safe. Then, we describe the technical details of the vulnerability that we discovered in Zabbix, its impact and how it can be prevented. Let’s dive into it!

Client-Side Session Storage 101

Sessions are all about storing a state across several HTTP requests, stateless by design. To this end, applications commonly hand a unique identifier to each client; they have to transmit it alongside future requests. The server can then load the associated information whether it is stored in-memory, in a database, on the local file system, etc. That’s what we usually call server-side session storage.

This historical approach works well but has drawbacks with the way modern web applications are developed and deployed. For instance, it does not scale well: if the backend service is split across multiple servers, how to make sure one’s session is available across services or even the entire server fleet?

As a result, developers introduced the storage of the session on the client-side. Instead of assigning a session identifier to the client, they now have to send a copy of the state with every request. Technology stacks like ASP and Java wrapped this concept in something called View States, but it is now very common to rely on the JSON Web Token (JWT) standard instead. 

The goal of both approaches is to safely store data client-side, but in a way that backend services can still ensure its authenticity and integrity: it requires the use of cryptography to offer these guarantees. Despite the risks of misconfiguration (weak secrets, support for broken cryptographic algorithms) and the inherent difficulty to revoke JWTs, this is mostly a safe way to proceed.

One must not confuse the security guarantees offered by encryption and authentication in such use cases. While the encrypted data may look “secure” to uneducated eyes, the backend service cannot detect if the session data was altered by the client. The use of encryption modes like ECB can even let attackers craft a valid, arbitrary ciphertext without knowledge of the key!

As a demonstration of risks that could arise because of an unsafe design and implementation of client-side session code, let’s look at the technical details of the two vulnerabilities we identified in Zabbix. 

Case Study: Zabbix Web Frontend Vulnerabilities

The monitoring platform Zabbix is commonly deployed on infrastructures with four distinct components:

• Zabbix Agent: service running on all monitored nodes, collecting information when requested by a Zabbix Server;
• Zabbix Server: it connects to Zabbix Agents to collect monitoring data and raise alerts if configured thresholds are reached;
• Zabbix Proxy: associating a single Zabbix Server to hundreds of Zabbix Agents can be very costly and hard to deploy in some network topologies. Zabbix Proxy instances aim to centralize the data of entire zones and report the collected data to the main Zabbix Server;
• Zabbix Web Frontend: an interface to the Zabbix Server, communicating over TCP and a shared database. This dashboard is used by system administrators to access the collected monitoring data and configure the Zabbix Server (e.g. list hosts, run scripts on Zabbix Agents).

During December 2021, we analyzed the external attack surface of the Zabbix Web Frontend to better understand the risks associated with the exposure of this software to untrusted networks. This effort led to the discovery of two critical vulnerabilities, CVE-2022-23131 and CVE-2022-23134. 

These findings are both related to the way Zabbix stores session data on the client-side. We will guide you through their vulnerable implementation, discuss its impact and how it could have been spotted in earlier development stages. 

Impact

The discovered vulnerabilities affect all supported Zabbix Web Frontend releases at the time of our research, up to and including 5.4.8, 5.0.18 and 4.0.36. They do not require prior knowledge of the target, and can be effortlessly automated by attackers. 

We highly recommend upgrading your instances running a Zabbix Web Frontend to 6.0.0beta2, 5.4.9, 5.0.19 or 4.0.37 to protect your infrastructure.

On instances where the SAML SSO authentication is enabled, it allows bypassing the authentication and gaining administrator privileges. This access can be used by attackers to execute arbitrary commands both on the linked Zabbix Server and Zabbix Agent instances with CVE-2021-46088, for which exploitation code is already public. Unlike Zabbix Agent, it is not possible to configure Zabbix Servers to disallow the execution of commands. 

Zabbix’ Client-Side Session Storage Implementation

Server-side sessions are a built-in feature of PHP. The client is assigned a unique session identifier in a cookie, PHPSESSID being the most common one, and has to transmit it with every request. On the server-side, PHP takes this value and looks for the associated session values on the filesystem (/var/lib/php/sessions, sometimes /tmp/) to populate the superglobal variable $_SESSION. Session values cannot be freely modified by clients, as they only control the identifier of the session.

The Zabbix Web Frontend rolls its own client-side storage implementation based on a powerful feature of PHP, custom session handlers. By calling session_set_save_handler() with a class implementing SessionHandlerInterface, all subsequent accesses to $_SESSION will be handled by methods of this class. 

In their case, the goal is to map any access to $_SESSION to cookies. For instance, indexing $_SESSION results in a call to CCookieSession::read(); CCookieHelper::get() is simply a wrapper around $_COOKIE:

ui/include/classes/core/CCookieSession.php

<?php
 
class CCookieSession implements SessionHandlerInterface {
   // [...]
   public const COOKIE_NAME = ZBX_SESSION_NAME;
   // [...]
   public function read($session_id) {
       $session_data = json_decode($this->parseData(), true);
       // [...]
       foreach ($session_data as $key => $value) {
           CSessionHelper::set($key, $value);
       }
   // [...]
   protected function parseData(): string {
       if (CCookieHelper::has(self::COOKIE_NAME)) {
           return base64_decode(CCookieHelper::get(self::COOKIE_NAME));
       }
 
       return '';
   }

Zabbix developers introduced a way to authenticate the data stored in cookies and to ensure they were not tampered with. This feature is implemented in CEncryptedCookieSession:

ui/include/classes/core/CEncryptedCookieSession.php

class CEncryptedCookieSession extends CCookieSession {
  // [...]   
  public function extractSessionId(): ?string {
       // [...] 
       if (!$this->checkSign($session_data)) {
           return null;
       }
       // [...] 
       return $session_data['sessionid'];
   }
   // [...]
   protected function checkSign(string $data): bool {
       $data = json_decode($data, true);
 
       if (!is_array($data) || !array_key_exists('sign', $data)) {
           return false;
       }
 
       $session_sign = $data['sign'];
       unset($data['sign']);
       $sign = CEncryptHelper::sign(json_encode($data));
       return $session_sign && $sign && CEncryptHelper::checkSign($session_sign, $sign);
   }
}

As a side note for advanced readers, there is a big red flag here: the terms “sign[ature]” and “encrypted” are used interchangeably. CEncryptHelper::sign() internally uses AES ECB, prone to malleability and not able to offer security guarantees about the authenticity of the data. Use of this construct also resulted in another security advisory, but it will not be detailed in this article. 

The method CEncryptedCookieSession::checkSign() is only invoked in CEncryptedCookieSession::extractSessionId(), but never in CCookieSessionmethods (e.g. during access in ​​CCookieSession::read()). The authenticity of the session is never validated when fields other than sessionid are accessed.

Since cookies are fully controlled by clients, they basically have control over the session. This is quite uncommon, and breaks most assumptions about the trustworthiness of values stored in it. It could lead to vulnerabilities in parts of the application where the session is used.

CVE-2022-23131 - Bypassing the SAML SSO Authentication

Security Assertion Markup Language (SAML) is one of the most common Single-Sign-On (SSO) standards. Implemented around XML, it allows Identity Providers (IdP, an entity with the ability to authenticate the user) to tell the Service Provider (SP, here Zabbix) who you are. You can configure the Zabbix Web Frontend to allow user authentication over SAML, but it is not enabled by default since it requires the knowledge of the details of the identity provider. This is the most common setup for enterprise deployments. 

The code related to the SAML authentication mechanism can be found in index_sso.php. In a nutshell, its goal is to:

• Redirect the user to the IdP;
• After the user has been authenticated, validate the format and the signature of the incoming SAML payload. A session entry named saml_data is created to remember the user's attributes;
• If an entry named saml_data exists in the session, extract its value and authenticate the user on Zabbix based on the value of username_attribute.

As explained in the previous section, CEncryptedCookieSession::checkSign() is never called in this file, hence the value of the session entry saml_data[username_attribute]can be fully controlled by the client:

ui/index_sso.php

   if (CSessionHelper::has('saml_data')) {
       $saml_data = CSessionHelper::get('saml_data');
       CWebUser::$data = API::getApiService('user')->loginByUsername($saml_data['username_attribute'],
           (CAuthenticationHelper::get(CAuthenticationHelper::SAML_CASE_SENSITIVE) == ZBX_AUTH_CASE_SENSITIVE),
           CAuthenticationHelper::get(CAuthenticationHelper::AUTHENTICATION_TYPE)
       );

The exploitation is straightforward, especially since the Zabbix Web Frontend is automatically configured with a highly-privileged user named Admin. 

Once authenticated as Admin on the dashboard, attackers can execute arbitrary commands on any attached Zabbix Server, and on Zabbix Agents if explicitly allowed in the configuration with AllowKey=system.run[*] (non-default). 

CVE-2022-23134 - Reconfiguring Instances

Another occurrence of the unsafe use of the session was found in setup.php. This script is usually run by system administrators when first deploying Zabbix Web Frontend and later access is only allowed to authenticated and highly-privileged users.

This page normally uses the session to keep track of the progress across the setup steps; again, CEncryptedCookieSession::checkSign() is never called here. Crafting a session with the entry step set to 6 allows re-running the latest step of the installation process.

This step is really interesting for attackers, as its goal is to create the Zabbix Web Frontend configuration file conf/zabbix.conf.php:

ui/include/classes/setup/CSetupWizard.php

   private function stage6(): array {
       // [...] [1] 
       $config = new CConfigFile($config_file_name);
       $config->config = [
           'DB' => [
               'TYPE' => $this->getConfig('DB_TYPE'),
               'SERVER' => $this->getConfig('DB_SERVER'),
               'PORT' => $this->getConfig('DB_PORT'),
               'DATABASE' => $this->getConfig('DB_DATABASE'),
               // [...]  
           ] + $db_creds_config + $vault_config,
           // [...] 
       ];
       $error = false;
       // [...] [2]
       $db_connect = $this->dbConnect($db_user, $db_pass);
       $is_superadmin = (CWebUser::$data && CWebUser::getType() == USER_TYPE_SUPER_ADMIN);
       $session_key_update_failed = ($db_connect && !$is_superadmin)
           ? !CEncryptHelper::updateKey(CEncryptHelper::generateKey())
           : false;
       if (!$db_connect || $session_key_update_failed) {
           // [...]  
           return $this->stage2();
       }
       // [...]  
       if (!$config->save()) {
           // [...]  

At [1], a new CConfigFile object is created to store and validate the new configuration values. The method CSetupWizard::getConfig() is simply a wrapper around the current session, hence these values are fully controlled by the attacker. 

At [2], the code tries to identify if the new database configuration is valid by attempting a connection to it. As this code is only supposed to be called during the initial setup process, when user accounts and database settings like the encryption key are not yet provisioned, attackers with control over the session will be able to go through the various checks.

As a result, existing configuration files can be overridden by attackers even if the Zabbix Web Frontend instance is already in a working state. By pointing to a database under their control, attackers can then gain access to the dashboard with a highly-privileged account:

It is important to understand that this access cannot be used to reach Zabbix Agentsdeployed on the network: the Zabbix Web Frontend and the Zabbix Server have to both use the same database to be able to communicate. It could still be possible to chain it with a code execution vulnerability on the web dashboard to gain control of the database and pivot on the network.

Other exploitation scenarios are possible in non-hardened or old environments. For instance, PHP’s MySQL client implements the LOAD DATA LOCAL statement, but now disabled by default for 3 years. Another lead could be the presence of calls to file_exists() with a fully-controlled parameter when validating the database configuration, which is known to be a security risk because of potentially dangerous scheme wrappers like phar://. 

Patch

Both vulnerabilities were addressed separately by the Zabbix maintainers:

• An additional signature field is introduced in the SSO authentication flow to prevent users from altering the SAML attributes stored in the session (0395828a)
• The way session cookies are authenticated is now done using an HMAC construct instead of AES ECB (eea1f70a). 
• The setup process now bails out earlier if the instance is already installed and the current user does not have the Super Administrator role (20943ae3). 

They also took the decision to not enforce cookie signature checks: the main drawback of this approach is that new features relying on the session can introduce a similar vulnerability if the call to CEncryptedCookieSession::checkSign() is forgotten. There is also no way to detect potential security regressions. 

Timeline

Summary

In this article we introduced common security issues when implementing client-side session storage. As a case study, we described high-severity vulnerabilities that we discovered in Zabbix, a popular open-source monitoring platform. The vulnerabilities CVE-2022-23131 and CVE-2022-23134, both with the same root cause, can lead to a bypass of authentication and enable remote attackers to execute arbitrary code on a targeted server instance.

When writing and reviewing code related to important security features, it is easy to make the same assumptions as the original developer who introduced the vulnerability. Here, there were no integration tests related to the client-side session storage that could have spotted this behavior.

Always provide access to sensible services with extended internal accesses (e.g. orchestration, monitoring) over VPNs or a restricted set of IP addresses, harden filesystem permissions to prevent unintended changes, remove setup scripts, etc. 

We would like to thank the Zabbix maintainers for their responsiveness and robust disclosure process. 

Related Blog Posts

• Pandora FMS 742: Critical Code Vulnerabilities Explained 
• phpBB 3.2.3: Phar Deserialization to RCE 
• 10 Unknown Security Pitfalls for Python 
• elFinder - A Case Study of Web File Manager Vulnerabilities