Earlier this week, we were notified of a potential hypervisor vulnerability (Xen Security Advisory 133: http://xenbits.xen.org/xsa/advisory-133.html and http://venom.crowdstrike.com/) that affects a portion of our First and Next Generation Cloud Servers fleet, as well as Cloud Big Data. Please note that OnMetal Cloud Servers are not affected.
Server Types that ARE Impacted
* FirstGen Cloud Servers running Windows
* NextGen Cloud Servers built from a PVHVM image
Server Types that are NOT Impacted
* FirstGen Cloud Servers running Linux
* NextGen Cloud Servers built from a PV image
We patched the portion of our infrastructure that supports the Cloud Virtual Machine (VM). For the patch to be effective in resolving the vulnerability, the customer VM must be power cycled, either by the customer or by Rackspace. Our preference is that customers do this themselves, and we strongly recommend that customers take this action as quickly as possible.
Given the severity of the vulnerability, customers have less than 24 hours to perform the power cycle themselves. After that window closes, for customers who have not completed this maintenance, Rackspace will force conduct the power cycle. As a number of our customers deploy across multiple regions, regional maintenance events will be staggered so no two regions are affected at the same time. We understand that many customers deploy in a single region. To help customers plan accordingly, a detailed timeline during which the forced Rackspace power cycle will take place for individual VM's will be made available via the First and Next Gen Cloud Servers APIs and Cloud Control Panel on Thursday, 5/14/15.
IMPORTANT NOTE: A SOFT REBOOT IS INSUFFICIENT to make the patch fully effective and resolve this security vulnerability. Details of the recommended power cycle process are outlined at: https://community.rackspace.com/products/f/25/t/5188
We recommend that customers ensure their applications and environments are able to withstand a short interruption in service prior to completing the power cycle of their VM(s). This means that there are no single points of failure in the configuration and that applications are able to gracefully resume service after a server pause process. For a comprehensive description of how customers can prepare for a power cycle, see Community page: https://community.rackspace.com/products/f/25/t/4319.
Thanks for the update Stuart.
Is there any indication as to how long the update is likely to take or if reboots will be required?
Thanks, sorry but could I ask for clarification on a few points? I've checked with the live chat support, but haven't been 100% convinced by the answers.
1) Are all VMs going to be forcibly power cycled after 24 hours, just ones that have been affected and haven't been rebooted, or all servers that haven't been rebooted since the patch?
2) Related to 1, is there a way see if a box is in need of a reboot? The article linked to in the advisory shows the (!) icon next to boxes, but I'm unclear if that would be present or if that's what will appear after 24 hours.
3) I can't see any way of determining if a box was built with a PV or PVHVM image. Is there any way to determine this? One of the live chat support team mentioned that if a box disk configuration was marked as "automatic" then it was a PV, but this wasn't an "official" determination?
Thanks for the questions. I've answered them below.
1) Are all VMs going to be forcibly power cycled after 24 hours, just ones that have been affected and haven't been rebooted, or all servers that haven't been rebooted since the patch?A. We will only forcibly power cycle the VMs have are affected and have not rebooted. If you power cycled the server before the maintenance window, then we will not power cycle the instance again.
2) Related to 1, is there a way see if a box is in need of a reboot? The article linked to in the advisory shows the (!) icon next to boxes, but I'm unclear if that would be present or if that's what will appear after 24 hours.A. The Cloud Control Panel team is working on getting the notifications into the server details by tomorrow morning. The icons will show up once that information has been loaded.
3) I can't see any way of determining if a box was built with a PV or PVHVM image. Is there any way to determine this? One of the live chat support team mentioned that if a box disk configuration was marked as "automatic" then it was a PV, but this wasn't an "official" determination?A. A simple way of determining if an instance is PV or PVHVM is to click on the server name in the control panel and look at the "system image" line item. It sill state whether it is "(PV)" or "(PVHVM)" at the end of the image name.
You can also check your server with this one-liner from the command line on the VM:
if [ `dmesg | egrep -i 'xen|front' | grep 'HVM' | wc -l` -eq 0 ] ; then echo "PV Not Impacted" ; else echo "PVHVM Impacted" ; fi
After running that command you'll receive back either: "PV Not Impacted" or "PVHVM Impacted"
That's great Stuart, thanks very much for your help!
Few more questions seeking clarity:
Here are the answers to your questions.
I received a support ticket listing the effected servers (which is all of them) at 4:34 p.m. Eastern. Between 6 and 7:30 p.m. Eastern, I power cycled all of them (shutdown -h, wait for pings to drop/console connect error, hard reboot via control panel).
I've received a second support ticket at 10:39 p.m. Eastern listing the same servers as being effected. The text of the tickets seems identical except for the blurb changes from "you have 24 hours" to "you have 18 hours". Might this be a duplicate or already queued notification, or should I assume that everything's going to need to reboot again?
I'll answer your question.
Q. I received a support ticket listing the effected servers (which is all of them) at 4:34 p.m. Eastern. Between 6 and 7:30 p.m. Eastern, I power cycled all of them (shutdown -h, wait for pings to drop/console connect error, hard reboot via control panel).
A. If you have already power cycled the servers on your account, then you will not need to do it again. We are communicating through ticket to remind customers of the time remaining on the self-service window. You may receive some additional notifications but you will not need to take further action and we will not power cycle your servers when the maintenance window starts.
Hi Stuart Bankey
I received this vulnerability error, as one of my server is got impacted by this. After reading the community support pages I came to know about reboot. so I started the reboot process and around 30 minutes passed but it still showing as continue process. Would you suggest me how long it will take to complete it.
I reboot all of my servers, and only one shows ... "applying update operation...." in VNC console from more than one hour... others servers reboot in a few minutes,
can you tell me if there is any way to know if the patch is already installed on my servers?
I may have missed it, but when exactly does the 24 hour window for self-reboots start and end? I would like to reboot at a quiet business time (0400 GMT on 15-May) and want to see if that is still within the allowable window.
I've also rebooted my servers and now can't connect to one of them even using rackpsace java console.
Thank you for reaching out to us for assistance with this! I'd be glad to answer you question here, the server should be just fine good since the server was rebooted. As long as you have performed a hard reboot your server should have been properly patched.
If you have any other issues, questions or concerns please give us a call, chat or via ticket to let us know and we'd be glad to do all we can to help!
The Rackspace Community (“Community”) is provided “AS IS” without warranty of any kind. The information on the Community sites is created by members of the Community and is intended for reference and general discussions only. Although some of the content may contain information provided by Rackspace employees, it does not represent an assessment of a particular customer environment or an assessment of any specific compliance with laws or regulations or constitute advice. We recommend that you engage additional expertise in order to further evaluate applicable requirements for your specific environment. For customer specific support issues please contact your Rackspace Support Team.READ MORE
RACKSPACE MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, AS TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THE RACKSPACE OPEN CLOUD COMMUNITY SITE. RACKSPACE RESERVES THE RIGHT TO DISCONTINUE OR MAKE CHANGES TO ITS SERVICES OFFERINGS AND SPECIFICATIONS AT ANY TIME WITHOUT NOTICE. USERS MUST TAKE FULL RESPONSIBILITY FOR APPLICATION OF ANY SERVICES AND/OR PROCESSES MENTIONED IN ANY COMMUNITY DISCUSSIONS. EXCEPT AS SET FORTH IN RACKSPACE GENERAL TERMS AND CONDITIONS, CLOUD TERMS OF SERVICE AND/OR OTHER AGREEMENT YOU SIGN WITH RACKSPACE, RACKSPACE ASSUMES NO LIABILITY WHATSOEVER, AND DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO ITS SERVICES INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT.
ALTHOUGH PART OF THE COMMUNITY GENERATED CONTENT MAY EXPLAIN HOW RACKSPACE SERVICES MAY WORK WITH THIRD PARTY PRODUCTS, THE INFORMATION CONTAINED IN THE COMMUNITY DISCUSSIONS IS NOT DESIGNED TO WORK WITH ALL SCENARIOS. ANY USE OR CHANGES TO THIRD PARTY PRODUCTS AND/OR CONFIGURATIONS SHOULD BE MADE AT THE DISCRETION OF YOUR ADMINISTRATORS AND SUBJECT TO THE APPLICABLE TERMS AND CONDITIONS OF SUCH THIRD PARTY. EVEN THOUGH RACKSPACE EMPLOYEES MAY PARTICIPATE IN THE COMMUNITY DISCUSSIONS, RACKSPACE DOES NOT PROVIDE TECHNICAL SUPPORT FOR THIRD PARTY PRODUCTS, OTHER THAN SPECIFIED IN YOUR HOSTING SERVICES AGREEMENT YOU HAVE SIGNED WITH RACKSPACE AND RACKSPACE ACCEPTS NO RESPONSIBILITY FOR THIRD-PARTY PRODUCTS.READ LESS