Please contact your support team if you have a question or need assistance for any Rackspace products, services, or articles.
Earlier this week, we were notified of a potential hypervisor vulnerability (Xen Security Advisory 133: http://xenbits.xen.org/xsa/advisory-133.html and http://venom.crowdstrike.com/) that affects a portion of our First and Next Generation Cloud Servers fleet, as well as Cloud Big Data. Please note that OnMetal Cloud Servers are not affected.
Server Types that ARE Impacted
* FirstGen Cloud Servers running Windows
* NextGen Cloud Servers built from a PVHVM image
Server Types that are NOT Impacted
* FirstGen Cloud Servers running Linux
* NextGen Cloud Servers built from a PV image
We patched the portion of our infrastructure that supports the Cloud Virtual Machine (VM). For the patch to be effective in resolving the vulnerability, the customer VM must be power cycled, either by the customer or by Rackspace. Our preference is that customers do this themselves, and we strongly recommend that customers take this action as quickly as possible.
Given the severity of the vulnerability, customers have less than 24 hours to perform the power cycle themselves. After that window closes, for customers who have not completed this maintenance, Rackspace will force conduct the power cycle. As a number of our customers deploy across multiple regions, regional maintenance events will be staggered so no two regions are affected at the same time. We understand that many customers deploy in a single region. To help customers plan accordingly, a detailed timeline during which the forced Rackspace power cycle will take place for individual VM's will be made available via the First and Next Gen Cloud Servers APIs and Cloud Control Panel on Thursday, 5/14/15.
IMPORTANT NOTE: A SOFT REBOOT IS INSUFFICIENT to make the patch fully effective and resolve this security vulnerability. Details of the recommended power cycle process are outlined at: https://community.rackspace.com/products/f/25/t/5188
We recommend that customers ensure their applications and environments are able to withstand a short interruption in service prior to completing the power cycle of their VM(s). This means that there are no single points of failure in the configuration and that applications are able to gracefully resume service after a server pause process. For a comprehensive description of how customers can prepare for a power cycle, see Community page: https://community.rackspace.com/products/f/25/t/4319.
Thanks for the update Stuart.
Is there any indication as to how long the update is likely to take or if reboots will be required?
Thanks, sorry but could I ask for clarification on a few points? I've checked with the live chat support, but haven't been 100% convinced by the answers.
1) Are all VMs going to be forcibly power cycled after 24 hours, just ones that have been affected and haven't been rebooted, or all servers that haven't been rebooted since the patch?
2) Related to 1, is there a way see if a box is in need of a reboot? The article linked to in the advisory shows the (!) icon next to boxes, but I'm unclear if that would be present or if that's what will appear after 24 hours.
3) I can't see any way of determining if a box was built with a PV or PVHVM image. Is there any way to determine this? One of the live chat support team mentioned that if a box disk configuration was marked as "automatic" then it was a PV, but this wasn't an "official" determination?
Thanks for the questions. I've answered them below.
1) Are all VMs going to be forcibly power cycled after 24 hours, just ones that have been affected and haven't been rebooted, or all servers that haven't been rebooted since the patch?A. We will only forcibly power cycle the VMs have are affected and have not rebooted. If you power cycled the server before the maintenance window, then we will not power cycle the instance again.
2) Related to 1, is there a way see if a box is in need of a reboot? The article linked to in the advisory shows the (!) icon next to boxes, but I'm unclear if that would be present or if that's what will appear after 24 hours.A. The Cloud Control Panel team is working on getting the notifications into the server details by tomorrow morning. The icons will show up once that information has been loaded.
3) I can't see any way of determining if a box was built with a PV or PVHVM image. Is there any way to determine this? One of the live chat support team mentioned that if a box disk configuration was marked as "automatic" then it was a PV, but this wasn't an "official" determination?A. A simple way of determining if an instance is PV or PVHVM is to click on the server name in the control panel and look at the "system image" line item. It sill state whether it is "(PV)" or "(PVHVM)" at the end of the image name.
You can also check your server with this one-liner from the command line on the VM:
if [ `dmesg | egrep -i 'xen|front' | grep 'HVM' | wc -l` -eq 0 ] ; then echo "PV Not Impacted" ; else echo "PVHVM Impacted" ; fi
After running that command you'll receive back either: "PV Not Impacted" or "PVHVM Impacted"
That's great Stuart, thanks very much for your help!
Few more questions seeking clarity: