Thank you for visiting the Rackspace Community
The The Community is live! Post new content or topics so our teams can assist.

Please contact your support team if you have a question or need assistance for any Rackspace products, services, or articles.

March 2015 Critical Maintenance - First & Next Gen Cloud Servers

  • Para Portugués: https://community.rackspace.com/general/f/53/t/4989
    Para Español: https://community.rackspace.com/general/f/53/t/4986

    Recently, several critical Xen hypervisor vulnerabilities have been discovered that affect a portion of our First and Next Generation Cloud Servers fleet. Remediating these issues will unfortunately require a reboot of affected host machines. To protect the security of your data, we will be patching our infrastructure prior to when the details of these exploits will be released publicly. The purpose of this communication is to give you details about this maintenance event with as much advance notice as possible. Please note that OnMetal Cloud Servers are not affected.

    While we do everything we can to minimize service interruptions, the security of your environment is our highest priority. We understand that any downtime impacts your business and we do not make this decision lightly. In preparation for a potential reboot, we recommend that you take proactive steps to ensure your environment is configured to return to proper operations. For a comprehensive description of how you can prepare for a reboot, please visit this Community page: https://community.rackspace.com/products/f/25/t/4319

    As a number of our customers deploy across multiple regions, regional maintenance is staggered so no two regions are affected at the same time. The broad regional maintenance schedule can be found later in this thread.

    We understand that many customers deploy in a single region. To help you plan accordingly, more granular per-instance reboot windows are available via the Cloud Control Panel, First and Next Gen Cloud Servers APIs, and a Command Line Tool (details later in this thread).

    We know that these kinds of maintenances are not desired, but when they are required, our goal is to partner with you to minimize impact to your application. Our entire Rackspace team is standing by ready to assist.

    For the most up to date information on status and additional details, please visit the Rackspace Cloud Status page: https://status.rackspace.com and the Securities & Vulnerabilities section of our Rackspace Community page: https://community.rackspace.com/general/f/53

    If you have any further questions or need assistance, please don't hesitate to contact a member of your Fanatical Support team.  We are here to serve you.

    Sincerely,

    Rackspace

  • Regional Maintenance Schedule

    The First & Next Gen Cloud Servers host infrastructure will be patched and rebooted to address several critical Xen vulnerabilities.  Reboots will occur on a per region basis, beginning Monday, March 2, 2015 (local time) and ending Monday, March 9, 2015 (local time) according to the following schedule: 

    First Generation Cloud Servers

    • LON: Tuesday, March 3rd 22:00 - Wednesday, March 4th 04:00 UTC
    • DFW: Friday, March 6th 01:00 - Friday, March 6th 06:00 CST; Friday, March 6th 22:00 - Saturday, March 7th 06:00 CST
    • ORD: Saturday, March 7th 22:00 CST - Sunday, March 8th 07:00 CDT; Sunday, March 8th 23:00 - Monday, March 9th 06:00 CDT [Update 2015-03-08 04:55 CDT: An error was discovered in our daylight savings calculations for these windows and the times have now been corrected. We apologize for the error.]


    Next Generation Cloud Servers

    • LON: Monday, March 2nd 22:00 - Tuesday, March 3rd 06:00 UTC
    • IAD: Tuesday, March 3rd 01:00 - Tuesday, March 3rd 05:00 EST
    • DFW: Tuesday, March 3rd 22:00 - Wednesday, March 4th 06:00 CST
    • SYD: Thursday, March 5th 01:00 - Thursday, March 5th 05:00 AEDT
    • ORD: Wednesday, March 4th 22:00 - Thursday, March 5th 06:00 CST; Thursday, March 5th 22:00 - Friday, March 6th 02:00 CST
    • HKG: Friday, March 6th 00:00 - Friday, March 6th 04:00 HKT

     

    Per Instance Maintenance Schedule

    The above regional maintenance schedule is provided for broad planning purposes only. More granular per instance reboot windows are available via the Cloud Control Panel, First and Next Gen Cloud Servers APIs, and a Command Line Tool.


    Cloud Control Panel

    Our Cloud Control Panel at https://mycloud.rackspace.com will display a visual indicator in the Cloud Servers list next to each server with an upcoming reboot.



    The same details are available for each individual server on the Cloud Servers detail page.



    If you create a new Cloud Server, it is possible that it will be placed on a host machine that still requires maintenance. If so, you will see the reboot window appear within 30 minutes of the build completion.

    A CSV export of all Cloud Servers with pending reboots can be downloaded from the Cloud Servers list. Only servers shown in the Cloud Servers list will be included in the CSV, so adjust your region selector to include all regions where you have servers prior to downloading the CSV. Note: The CSV download is not available to Internet Explorer users at this time.



    The notification will be removed from each server following a successful reboot of the host machine during the maintenance window.



    First and Next Gen Cloud Servers APIs

    The reboot window for each Cloud Server affected by this maintenance is populated in the server metadata under a key named "rax:reboot_window". The metadata value is a semi-colon separated set of timestamps representing the start and end of the server's maintenance window in UTC. For example, a reboot window of 02:00 to 04:00 UTC on March 1st would be represented as:

    - Key: rax:reboot_window
    - Value: 2015-03-01T02:00:00Z;2015-03-01T04:00:00Z

    If you create a new Cloud Server, it is possible that it will be placed on a host machine that still requires maintenance. If so, you will see the reboot window appear within 30 minutes of the build completion. The metadata key and value will be removed from each server following a successful reboot of the host machine during the maintenance window.


    Command Line Tool

    We have created a cross-platform command line tool that can be used to retrieve the reboot window(s) for all of your Cloud Servers and optionally generate a CSV file. Additional details are available in the README file of the GitHub repository.

    https://github.com/rackerlabs/cs-reboot-info

  • Frequently Asked Questions
     

    Q: Which products and services are affected?

    A: First Generation and Next Generation Cloud Servers as well as Cloud Big Data. Cloud Big Data customers are being notified separately. No other Rackspace products or services are impacted.


    Q: Prior to my maintenance window, can I launch new cloud servers on patched hosts so I can pre-migrate my application?

    A: We are patching empty host machines as quickly as we can prior to the scheduled regional maintenance windows. Depending on region and flavor class, the amount of patched host capacity may be limited. It is possible for a newly created cloud server to land on a patched host, but we do not have sufficient capacity to guarantee this for all builds.


    Q: Should I expect all of my cloud servers to complete their reboots and be back online prior to the end of the instance-specific reboot windows?

    A: We strive to complete all host reboots within the instance-specific reboot windows. In extremely rare circumstances, hosts may not come back online until after the conclusion of the instance-specific reboot window, reboots may be initiated just outside the instance-specific window, or rescheduling of an instance-specific window may be required.


    Q: How do I know if a new cloud server that I've just launched will need to undergo a reboot?

    A: First, check the Regional Maintenance Schedule at https://community.rackspace.com/general/f/53/p/4978/11947#11947 to determine if reboots are complete in the region in which you are creating a server. If the region's maintenance window has not yet passed, it is possible that it will be placed on a host machine that still requires maintenance. If so, you will see the reboot window appear within 30 minutes of the build completion, as described in the post above.


    Q: If I reboot my cloud server prior to its scheduled maintenance window, will that allow me to avoid the scheduled reboot?

    A: No. Although you can reboot your cloud server at any time, the security patches that must be applied require a reboot of the underlying host to take effect. Rebooting your cloud server early will not allow you to avoid the scheduled host reboot.


    Q: Can I change the time of my reboot window?

    A: We cannot modify regional or per instance schedules due to the scope of this issue and our desire to patch all affected hosts prior to when the details of the vulnerabilities will be released publicly.


    Q: Do you have live migrate capabilities to transparently patch host machines?

    A: We do have live migration capabilities in newer portions of our fleet. We are actively using live migrate to transparently patch capable host machines and will continue this process throughout the entire maintenance period. If your cloud server is live migrated to a remediated host, the maintenance notice will be removed from your server and a reboot will not be required. While our goal is to transparently patch as many hosts as possible, you should not rely on this occurring and should still plan accordingly for a reboot during the specified window. We are unable to provide more specifics on which cloud servers may or may not be live migrated.


    Q: Can I opt out of the maintenance?

    A: The nature of this issue requires that we patch and reboot all affected host machines. As such, impacted cloud servers can not opt out of the maintenance.

     

  • Our Cloud Server reboot windows are now complete. There are a small number of hosts that were unable to be patched in their original windows and will be patched in the next 24 hours. Affected customers have received a ticket communication and will see the new window reflected on their server(s) as described above. We will update this post again tomorrow once all patching is complete.

  • All Cloud Server reboots are now complete, including the small number of hosts that had to be patched over the past 24 hours in rescheduled windows.

    If you have any further questions or need assistance, please don't hesitate to contact a member of your Fanatical Support team.