Thank you for visiting the Rackspace Community
The Community is live! Post new content or topics so our teams can assist.

Please contact your support team if you have a question or need assistance for any Rackspace products, services, or articles.

IMPORTANT NOTICE – First and Next Gen Server Vulnerability - October 17, 2015

  • Rackspace was notified of a host level security vulnerability that affects a portion of our Cloud Servers fleet. We will patch and reboot the portion of our infrastructure that is impacted by thisvulnerability. Where possible, we will Live Migrate customer instances to patched infrastructure to reduce the impact of this maintenance on your environment. In some cases, Live Migration is not an option - these customer instances will be impacted during the patch and reboot process. If you have a Cloud Server instance that requires a reboot, you will receive a ticket with the specific date and time that your instance will be rebooted.

    As more information is available, we will share with you via this post and in direct customer communication as appropriate.   

    Q: Which products and services are affected?

    A: First Generation and Next Generation Cloud Servers, as well as Cloud Big Data, are affected by this vulnerability.  No other Rackspace products or services, including OnMetal servers, are impacted.

    Q: Is my server being rebooted? 

    A: We are leveraging a variety of techniques to resolve this vulnerability. Our preference is to Live Migrate Cloud Server Instances, making this completely seamless to customers and avoiding disruption. However, some servers must be rebooted to apply the patch. If the host on which your Cloud Server resides requires a reboot, you will receive a separate ticket notification with a specific date and time when the reboot will be performed.

    Q: When will the control panel show the maintenance windows? 

    A: If we determine an instances is unable to be Live Migrated and the host machine needs to be rebooted, we will send a ticket to inform you of the maintenance window. Your control panel will be updated to reflect this window within a few hours of receiving the ticket.

    Q: Do you have Live Migrate capabilities to transparently move customers to patched host machines?

    A: We do have live migration capabilities in portions of our fleet, and will utilize this functionality to move instances to patched host machines throughout the maintenance period. If your cloud server is live migrated to a remediated host, a reboot of your instance will not be required.

    Q: Prior to my maintenance window, can I launch new cloud servers on patched hosts so I can pre-migrate my application?

    A: Our engineering teams have been focused on evaluating the Xen vulnerability, as well as creating and testing patches.  We are now beginning to patch empty hosts.  Depending on region and flavor class, the amount of patched host capacity may be limited. While it is possible for a newly created cloud server to land on a patched host, we do not have sufficient capacity to guarantee this for all builds.

    Q: If I reboot my cloud server prior to its scheduled maintenance window, will that allow me to avoid the scheduled Live Migration or reboot?

    A: No. Although you can reboot your cloud server at any time, the security patches that must be applied require a reboot of the underlying host to take effect. Rebooting your cloud server early will not allow you to avoid the scheduled host reboot.

    Q: Will I experience down time as part of the Live Migrate process?

    A: No. You should not experience any down time as part of a Live Migration. The Live Migration process is a seamless experience for moving instances between host machines.  We have completed thousands of Live Migrations since bringing this feature to our cloud, including moving instances between geographically diverse facilities without interruptions in connectivity. When live migrated, instances should experience only a minor blip in connectivity that, in our testing, typically lasts less than one second.

    Q: When will my specific Live Migration event begin?

    A: We are not able to provide the specific start time of the Live Migration of individual cloud server.

    Q: How long does the Live Migration process take?

    A: A number of factors can influence the duration of a Live Migration. Due to the diversity of customer environments and the elastic nature of cloud infrastructure, we are unable to provide a per-instance estimate of the duration of the Live Migration process.

    Q: How will I know when Live Migration of my instance(s) is complete?

    A: Live Migrations are designed to allow us to migrate customer instances between hosts without noticeable impact to operability.  During the live migration process, instances might experience a few seconds of packet loss in the instant that we switch from old host to new host. As a result, we will notify impacted customers only when Live Migration of a particular instance is unsuccessful.

    Q: Will my IP addresses change?

    A: No. All attributes associated with your cloud server will remain the same, including your IP address.

    Q: One of the instances is a database server. How will this event affect my database read/writes/updates?

    A: All updates made to your server during the Live Migration process will persist throughout the migration. Live Migrate does not require that you halt any processes, including the modification of data.

  • Q: If my instance is scheduled for a reboot, how can I find the reboot window?

    A: ​We have provided several different methods to determine when your instance reboot is scheduled.


    Cloud Control Panel

    The Cloud Servers list in our Cloud Control Panel (https://mycloud.rackspace.com) will display a visual indicator next to each server that is affected by an upcoming reboot.



    The same details are available for each individual server on the Cloud Servers detail page.



    If you create a new Cloud Server, it is possible that it will be placed on a host machine that still requires maintenance. If so, you will see the visual indicator for the reboot window appear within 30 minutes of the build completion.

    A CSV export of all Cloud Servers with pending reboots can be downloaded from the Cloud Servers list. Only those servers shown in the Cloud Servers list will be included in the CSV so, if needed, adjust your region selector to include all regions in which you have servers prior to downloading the CSV. Note: The CSV download is not available to Internet Explorer users at this time.



    The visual indicator notification will be removed from each server following a successful reboot of the host machine during the maintenance window.



    First and Next Gen Cloud Servers APIs

    The reboot window for each Cloud Server affected by this maintenance is populated in the server metadata under a key named "rax:reboot_window". The metadata value is a semi-colon separated set of timestamps, representing the start and end of the server's maintenance window in UTC. For example, a reboot window of 02:00 to 04:00 UTC on March 1st would be represented as:

    - Key: rax:reboot_window
    - Value: 2015-03-01T02:00:00Z;2015-03-01T04:00:00Z

    If you create a new Cloud Server, it is possible that it will be placed on a host machine that still requires maintenance. If so, you will see a visual indicator of the reboot window appear within 30 minutes of the build completion. The metadata key and value will be removed from each server following a successful reboot of the host machine during the maintenance window.


    Command Line Tool

    We have created a cross-platform command line tool that can be used to retrieve the reboot window(s) for all of your Cloud Servers and optionally generate a CSV file. Additional details are available in the README file of the GitHub repository.

    https://github.com/rackerlabs/cs-reboot-info

  • We have successfully completed the maintenance effort associated with this vulnerability.  In some cases, we were able to patch hosts without impacting those customers who may have received a ticket notifying of a scheduled reboot.  

     

    To date, we’ve learned of no data compromise among Rackspace customers. We apologize for any inconvenience that you may have experienced. 

     

    If you notice any issues with your Cloud Server, or if you determine that there is an issue with the applications or  services running on your Cloud Server, please contact your Fanatical Support Team.