On Dec. 10, three vulnerabilities in the Envoy proxy were made public, one of which was classified as “high severity” and two as “medium severity,” affecting all versions up to and including Envoy 1.12.1.
Istio, which relies on Envoy, is also directly affected by these issues. The vulnerabilities may affect many Kubernetes deployments using Envoy, including many that are managed by cloud providers. On Dec. 13, Google issued an email alert to all its Google Kubernetes Engine (GKE) users, urging them to upgrade all Envoy instances in their clusters. In this article, I will shed more light on these three issues, their impact and how they were fixed. I’ll also present a proof-of-concept video of a full bypass of Envoy rules.
Before diving into the details, let’s go over the fundamentals of the affected components:
Service Mesh
A service mesh is a transparent software infrastructure layer designed to improve networking between microservices. The service mesh ensures communication is fast, reliable and secure. It provides useful capabilities such as load balancing, traceability, encryption and more.
Envoy
Envoy is a high-performance proxy designed for cloud-native applications. It was originally developed by the engineers at Lyft, and it grew to become open sourced and graduated to be a Cloud Native Computing Foundation (CNCF) project. Additionally, it is a key component in the most popular service mesh – Istio.
Istio
Istio is a service mesh, originally developed by Google, IBM and Lyft. It uses Envoy as a sidecar proxy, which means every microservice or pod has an Envoy running beside it and all the communication in the cluster goes through these sidecar components.
CVE-2019-18838 – Denial of Service and Potentially Other Issues
Envoy can be customizable with different encoding filters. If it is configured with an encoder, containing a specific router manager API call, and it receives an HTTP request without the "Host" header, it will send back an "Invalid request" response. This response is dispatched through the configured encoder filter chain before being sent to the client. If an encoder filter access requests "Host" header, it will cause a NULL pointer to be dereferenced and result in abnormal behavior of Envoy.
It may cause Envoy to fail and crash, but under certain circumstances, an attacker may be able to craft an exploit even more dangerous and execute code on Envoy instances.
CVE-2019-18801 – Heap Overflow
Envoy can be configured to accept HTTP 1, and while doing so, it assumes that all the HTTP header value sizes are less than 4KB. However, in HTTP 2 there are no restrictions on the size of the values, and as a result, untrusted input can overflow the buffer allocated for the encoding of the headers and crash Envoy.
So far, an RCE exploit was not published, but a proof-of-concept exists in which two malicious requests are able to result in bypassing Envoy’s access control.
CVE-2019-18802 – Policy Bypass and Potentially Other Issues
The security problem here is that Envoy’s parser incorrectly fails to trim whitespace after the HTTP header value. This can result in many security issues like policy bypass, privilege escalation or information disclosure. For example, one of Envoy’s features is the ability to reject HTTP requests by filtering the hostname header. The problem is that Envoy will treat “forbiddenSite[.]com ” as a different string from “forbiddenSite[.]com” (notice the space character after the dot com) and forward the request instead of denying it. After that, the HTTP server will trim the whitespace to send a response.
Proof of Concept
Mitigation
On the same day the vulnerabilities were disclosed, Envoy released version 1.12.2, which contained the fixes for these issues. Istio released version 1.4.2, which contained the new Envoy as the sidecar proxy.
Although all these vulnerabilities require special configuration to successfully exploit, I would still highly recommend all Envoy and Istio users upgrade their instances to the latest versions as fast as possible. This is not just because these vulnerabilities can lead to a denial of service, but because their impact is not yet fully understood, and some exploits can potentially lead to much more damage. In a worst-case scenario, this could even potentially lead to a remote code execution.
The original advisories can be found on Envoy’s Github security page.