Thank you for visiting the Rackspace Community
The The Community is live! Post new content or topics so our teams can assist.

Please contact your support team if you have a question or need assistance for any Rackspace products, services, or articles.

IMPORTANT NOTICE – QEMU "VENOM" Vulnerability

  • Earlier this week, we were notified of a potential hypervisor vulnerability (Xen Security Advisory 133:  http://xenbits.xen.org/xsa/advisory-133.html and http://venom.crowdstrike.com/) that affects a portion of our First and Next Generation Cloud Servers fleet, as well as Cloud Big Data. Please note that OnMetal Cloud Servers are not affected. 

     

    Server Types that ARE Impacted

    * FirstGen Cloud Servers running Windows

    * NextGen Cloud Servers built from a PVHVM image

     

    Server Types that are NOT Impacted

    * FirstGen Cloud Servers running Linux

    * NextGen Cloud Servers built from a PV image

     

    We patched the portion of our infrastructure that supports the Cloud Virtual Machine (VM). For the patch to be effective in resolving the vulnerability, the customer VM must be power cycled, either by the customer or by Rackspace. Our preference is that customers do this themselves, and we strongly recommend that customers take this action as quickly as possible. 

    Given the severity of the vulnerability, customers have less than 24 hours to perform the power cycle themselves. After that window closes, for customers who have not completed this maintenance, Rackspace will force conduct the power cycle. As a number of our customers deploy across multiple regions, regional maintenance events will be staggered so no two regions are affected at the same time. We understand that many customers deploy in a single region. To help customers plan accordingly, a detailed timeline during which the forced Rackspace power cycle will take place for individual VM's will be made available via the First and Next Gen Cloud Servers APIs and Cloud Control Panel on Thursday, 5/14/15.

    IMPORTANT NOTE: A SOFT REBOOT IS INSUFFICIENT to make the patch fully effective and resolve this security vulnerability. Details of the recommended power cycle process are outlined at: https://community.rackspace.com/products/f/25/t/5188 

    We recommend that customers ensure their applications and environments are able to withstand a short interruption in service prior to completing the power cycle of their VM(s). This means that there are no single points of failure in the configuration and that applications are able to gracefully resume service after a server pause process. For a comprehensive description of how customers can prepare for a power cycle, see Community page: https://community.rackspace.com/products/f/25/t/4319.

  • Thanks for the update Stuart.

    Is there any indication as to how long the update is likely to take or if reboots will be required?

  • Hi,

    Thanks, sorry but could I ask for clarification on a few points? I've checked with the live chat support, but haven't been 100% convinced by the answers.

    1) Are all VMs going to be forcibly power cycled after 24 hours, just ones that have been affected and haven't been rebooted, or all servers that haven't been rebooted since the patch?

    2) Related to 1, is there a way see if a box is in need of a reboot? The article linked to in the advisory shows the (!) icon next to boxes, but I'm unclear if that would be present or if that's what will appear after 24 hours.

    3) I can't see any way of determining if a box was built with a PV or PVHVM image. Is there any way to determine this? One of the live chat support team mentioned that if a box disk configuration was marked as "automatic" then it was a PV, but this wasn't an "official" determination?

    Many thanks,

  • Hi leftbrained,

    Thanks for the questions. I've answered them below.

    1) Are all VMs going to be forcibly power cycled after 24 hours, just ones that have been affected and haven't been rebooted, or all servers that haven't been rebooted since the patch?
    A. We will only forcibly power cycle the VMs have are affected and have not rebooted. If you power cycled the server before the maintenance window, then we will not power cycle the instance again. 

    2) Related to 1, is there a way see if a box is in need of a reboot? The article linked to in the advisory shows the (!) icon next to boxes, but I'm unclear if that would be present or if that's what will appear after 24 hours.
    A. The Cloud Control Panel team is working on getting the notifications into the server details by tomorrow morning. The icons will show up once that information has been loaded.

    3) I can't see any way of determining if a box was built with a PV or PVHVM image. Is there any way to determine this? One of the live chat support team mentioned that if a box disk configuration was marked as "automatic" then it was a PV, but this wasn't an "official" determination?
    A. A simple way of determining if an instance is PV or PVHVM is to click on the server name in the control panel and look at the "system image" line item. It sill state whether it is "(PV)" or "(PVHVM)" at the end of the image name. 

    You can also check your server with this one-liner from the command line on the VM:

    if [ `dmesg | egrep -i 'xen|front' | grep 'HVM' | wc -l` -eq 0 ] ; then echo "PV Not Impacted" ; else echo "PVHVM Impacted" ; fi

    After running that command you'll receive back either: "PV Not Impacted" or "PVHVM Impacted"

  • That's great Stuart, thanks very much for your help!

  • Few more questions seeking clarity:

    1. I rebooted my instances. Does that prevent this vuln from being used against me by an unpatched "neighbor" who has yet to reboot?
    2. Is Rackspace monitoring for signs of VENOM being exploited on their network? Has Rackspace seen any evidence of this attack being used "in the wild"?
  • Hi gabehammersmith,

    Here are the answers to your questions.

    Q. I rebooted my instances. Does that prevent this vuln from being used against me by an unpatched "neighbor" who has yet to reboot?
    A. All instances on the host must be patched for the host to be clear from this vulnerability. We are providing a window for customers to reboot their own instances before we reboot on their behalf because we want to make this process as accommodating as possible. Any instances that are not rebooted by the customer past the 24 hour window will be rebooted by Rackspace. We are doing this to ensure the safety of our customers data.  
    We are also monitoring the network and overall infrastructure for any use of an exploit for this vulnerability.   If there were to be a known exploit, we will immediately reach out directly to those impacted customers to contain and resolve the issue.
     
    Q. Is Rackspace monitoring for signs of VENOM being exploited on their network? Has Rackspace seen any evidence of this attack being used "in the wild"?
    A. Rackspace continuously monitors our infrastructure to ensure the security and stability of our environment. We have not detected this vulnerability being used in any attempted exploit.  There are no reports of any exploit to date.   If there were to be a known exploit, we will immediately reach out directly to those impacted customers to contain and resolve the issue. 
  • Thanks Stuart!

  • I received a support ticket listing the effected servers (which is all of them) at 4:34 p.m. Eastern. Between 6 and 7:30 p.m. Eastern, I power cycled all of them (shutdown -h, wait for pings to drop/console connect error, hard reboot via control panel).


    I've received a second support ticket at 10:39 p.m. Eastern listing the same servers as being effected. The text of the tickets seems identical except for the blurb changes from "you have 24 hours" to "you have 18 hours". Might this be a duplicate or already queued notification, or should I assume that everything's going to need to reboot again?

  • Hi NickP,

    I'll answer your question.

    Q. I received a support ticket listing the effected servers (which is all of them) at 4:34 p.m. Eastern. Between 6 and 7:30 p.m. Eastern, I power cycled all of them (shutdown -h, wait for pings to drop/console connect error, hard reboot via control panel).


    I've received a second support ticket at 10:39 p.m. Eastern listing the same servers as being effected. The text of the tickets seems identical except for the blurb changes from "you have 24 hours" to "you have 18 hours". Might this be a duplicate or already queued notification, or should I assume that everything's going to need to reboot again?

    A. If you have already power cycled the servers on your account, then you will not need to do it again. We are communicating through ticket to remind customers of the time remaining on the self-service window. You may receive some additional notifications but you will not need to take further action and we will not power cycle your servers when the maintenance window starts.

  • Hi Stuart Bankey

    I received this vulnerability error, as one of my server is got impacted by this. After reading the community support pages I came to know about reboot. so I started the reboot process and around 30 minutes passed but it still showing as continue process. Would you suggest me how long it will take to complete it.

    Thanks

    Neha Singhania

  • Hi,

    I reboot all of my servers, and only one shows ... "applying update operation...." in  VNC console from more than one hour...  others servers reboot in a few minutes,

    can you tell me if there is any way to know if the patch is already installed on my servers?

  • I may have missed it, but when exactly does the 24 hour window for self-reboots start and end?  I would like to  reboot at a quiet business time (0400 GMT on 15-May) and want to see if that is still within the allowable window.

  • Hello,


    I've also rebooted my servers and now can't connect to one of them even using rackpsace java console.

    Thanks

  • Hello there,


    Thank you for reaching out to us for assistance with this! I'd be glad to answer you question here, the server should be just fine good since the server was rebooted. As long as you have performed a hard reboot your server should have been properly patched.

    If you have any other issues, questions or concerns please give us a call, chat or via ticket to let us know and we'd be glad to do all we can to help!