This advisory is to inform Instaclustr customers of Instaclustr’s risk assessment and mitigation steps in relation to the Spectre (CVE-2017-5715 and CVE-2017-5753) and Meltdown (CVE-2017-5754) vulnerabilities disclosed last last week. We have considered these vulnerabilities from two perspectives, Cross-VM leakage and Host (cross process) leakage.
For Cross-VM leakage, our cloud providers have published advice that the underlying services are patched and no longer vulnerable. Since the updates to the underlying hypervisors, we have observed an increase in CPU usage of up to 20% across nodes under our management.
For Host (cross process) leakage, the risk of compromise of cluster data is low as we run each cluster as a single tenant. Instaclustr is in the process of testing the impact of and planning the roll out of the host OS patches. We expect this to be completed within one month.
Each of these scenarios is covered in more detail below.
Cross-VM Leakage
Cross-VM leakage is a significant risk for any workload running in public cloud environments. This risk is managed by the cloud providers and their quick response to these vulnerabilities demonstrates how seriously they take these risks. We use three providers for virtual instances, Amazon Web Services, Google Compute Platform and Microsoft Azure. Our Softlayer offering is run on bare metal, so cross-VM leakage is not relevant to that service. Each of our virtual infrastructure providers have provided assurances that their hypervisors are patched and no longer vulnerable to these vulnerabilities.
Performance impacts of this patching have been observed since being fully rolled out by the cloud providers. We have seen up to 20% increases in CPU utilization and a small increase in latency across all clusters in AWS and Azure (GCP impact appears to be negligible). For clusters with sufficient processing capacity, noticeable user impact will be small at most (we’ve generally seen 1-2ms latency increase in synthetic monitoring). For clusters which were already running close to processing capacity, you may experience overall higher latency, or periods of higher latency, and may need to add nodes to regain previous performance.
From our providers:
Amazon Web Services
“While all customer instances are protected, we recommend that customers patch their instance operating systems. This will strengthen the protections that these operating systems provide to isolate software running within the same instance. For more details, refer to specific vendor guidance on patch availability and deployment.”
Google Cloud Platform
“Infrastructure patched against known attacks. Customers must patch/update guest environment.”
Microsoft Azure
“This Azure infrastructure update addresses the disclosed vulnerability at the hypervisor level and does not require an update to your Windows or Linux VM images. However, as always, you should continue to apply security best practices for your VM images.”
Host (cross process) leakage
We have assessed these vulnerabilities against our environment and consider the overall risk to information leakage as low. Our environments are all single tenant, and customer access is limited to the application layer. While most of our applications are not modifiable, Spark and Zeppelin can run arbitrary code. We have determined that it is possible that an authorised user could execute one of these attacks against their own hosted systems. In the event that either of these vulnerabilities are exploited by one of our customers, they only gain access to their own information.
Even though the risk of exploitation is low, it does open another vector for attack, and patches have been released for the operating systems that we use. As such, Instaclustr is preparing to patch our systems using the following process.
- Performance benchmarking
Operating system patches have been released, and we are currently undertaking benchmarking to determine if there is any additional impact on cluster performance. We will release the results of our benchmarking prior to commencing the upgrade process. - Plan out OS upgrades
Node OS upgrades will be carefully planned, and where possible we will upgrade non-production systems first. Customers host a Development and/or Test environment with us in addition to Production, we will work with those customers to upgrade the lower environments and allow time for testing prior to upgrading Production clusters. - Advise and upgrade
We will be patching all clusters over the next month. We will provide updates on progress on https://status.instaclustr.com. The OS upgrade will be completed without downtime, but will require a rolling restart of each cluster.
Should you have any questions regarding this advisory, please contact us by email [email protected].
If you wish to discuss scheduling of the upgrade to your system or have any other questions regarding the impact of these patches, please contact [email protected].
This advisory is based on currently available information. We have made efforts to ensure that the information is correct at the time of release. In the event that information changes we will update this advisory. This information is provided to support our customers own risk assessment and mitigation processes.