By Ariel Zelivansky and Aviv Sasson
The Kubernetes Patch Release Team on Tuesday released new builds that patch Kubernetes vulnerabilities CVE-2019-16276 and CVE-2019-11253, which have been publicly disclosed over the past month. These vulnerabilities have the potential to be highly dangerous under some Kubernetes configurations.
We highly recommend upgrading to Kubernetes builds 1.14.8, 1.15.5 or 1.16.2, regardless of your Kubernetes configuration.
Here is an overview of the now-resolved issues:
CVE-2019-16276, or HTTP Protocol Violation in Go’s net/http Library
This vulnerability has its roots in the Go language’s standard HTTP library, net/http. The library is used for HTTP request parsing in Kubernetes. Its implementation normalized invalid HTTP requests in a way that caused a potential danger for Kubernetes.
HTTP requests are comprised of a field-name, followed by a colon, then its value. Per the HTTP/1.1 specification, no whitespace is allowed between the header’s field-name and colon. As mentioned in the GitHub issue of this problem, the net/http library interpreted headers with this whitespace the same as valid headers, in violation of the HTTP RFC.
This seemingly innocent problem becomes dangerous when a Go server relying on the library is used in combination with a reverse proxy that filters request headers. The proxy may ignore invalid headers, but forward them to the Go server, which would interpret these headers as valid.
For example, if a reverse proxy server is used for authentication, attackers could exploit this bug to authenticate as any user by crafting an invalid header that would go through to the server.
This scenario is not at all hypothetical. The Kubernetes API server can be configured to work with an Authenticating Proxy and identify users through request headers. Such a proxy may be used to allow users to authenticate to the API server using an existing SSO scheme, or other access control configurations.
To exploit the attack, assuming X-Remote-User is used as the identifying user field for the API server, an attacker may send the following request to the proxy:
X-Remote-User : admin
If the proxy is designed to filter X-Remote-User headers but doesn’t recognize the header because it’s invalid and forwards it to the Kubernetes API server, the attacker would successfully pass the API request with the roles of the “admin” user.
The diagram below demonstrates how such an attack may take place.
In this case, the proxy is highly naive. It both accepts anonymous connections and forwards invalid headers. The attacker sends a malicious request with an invalid header with including a whitespace. The API server accepts this header and performs the attacker’s requests with Bob’s roles. Of course, the details of an actual exploit would rely on the implementation of the proxy server.
On the bright side, we were unable to find any open-source authenticating proxy that is commonly used. Using an authenticating proxy is a niche configuration, and special flags need to be passed to the API server to enable it. If your Kubernetes setup does use an Authenticating Proxy, make sure to upgrade immediately, especially because exploitation of this problem is very simple.
CJ Cullen from the Kubernetes Security Committee published a security advisory on this issue. The new releases were built with the fixed Go version (Go 1.12.10), rendering them immune to this attack.
CVE-2019-11253, or Kubernetes YAML Parsing Vulnerable to DoS
In an unusual manner, the discovery of this vulnerability was made by a curious Kubernetes user over a Stack Overflow question. The user found that the Kubernetes API server was vulnerable to a specific kind of attack on YAML parsers called YAML bomb attack. The attack, sometimes dubbed “Billion laughs attack” for historical reasons, intends to crash the YAML parser by exponentially loading it with references. The user worried how such an attack could be prevented to protect his API server.
The original exploit attached to the question is shown below.
Rory McCune, a Stack Overflow user, realized that the fact this attack works is a potentially dangerous vulnerability in Kubernetes, and directed it to the Kubernetes security team. The security team suggested publishing the details of this vulnerability on its own GitHub issue, since the details of this attack were already public.
Although the CVE was assigned to Kubernetes, and a patch was released to its source, the real culprit of this vulnerability is the YAML parser library that enabled this attack. Fortunately, another patch was written to resolve this problem at the go-yaml library level, preventing this attack in other projects that rely on its code.
After we experimented with exploitation of this vulnerability, we determined it should be considered high severity. Any user or container with roles that allow delivering YAML to the API server may crash it. Although it may be brought up depending on its restart policy and restart limit, the attacker may abuse the attack and deliver it continuously.
We recommend reviewing each role given to users, pods or anonymous users to determine if it is required, especially if it allows sending POST requests with a YAML file.
Fixed Releases
Kubernetes releases 1.14.8, 1.15.5 and 1.16.2 include the fixes to these issues. We recommend that you update clusters to the safe versions regardless of your configuration.
We would like to thank Jordan Liggitt from the Kubernetes release team for directing us to the correct releases.
Learn more about our previous research on Kubernetes vulnerabilities.