Thank you for visiting the Rackspace Community
The Community is currently in read-only mode. All content is available, but the ability to post new content or topics is not available at this time.

Please contact your support team if you have a question or need assistance for any Rackspace products, services, or articles.

Ports take about a minute to build

This question is answered.

Hi community,

Hoping someone here (maybe jamesdenton) will quickly recognize this issue. We're seeing ports take about a minute to build when our iptables length reaches 8000 or so. This means if we're spinning up several VMs at once, the third or fourth time out waiting for a DHCP response.

I've found https://bugs.launchpad.net/neutron/+bug/1314189 which mentions both the ipset and _find_last_entry() improvements in Juno, which we're running (OSAD 10.1.12, upgraded from RPC 9.0.6).

In logs we see something like this (notice the timestamps):

2015-10-30 10:17:18.704 33315 INFO neutron.agent.securitygroups_rpc [req-49aa84e3-1bb5-4865-9a3f-a794fc9bfd63 None] Preparing filters for devices set(['tap23273962-1f', 'tape67d396f-3d', 'tap5aeaaf61-47', 'tapedd8d084-e8', 'tap9d968d11-52', 'tap68609379-9c'])
2015-10-30 10:22:55.874 33315 INFO neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-49aa84e3-1bb5-4865-9a3f-a794fc9bfd63 None] Port tapedd8d084-e8 updated. Details: {u'profile': {}, u'admin_state_up': True, u'network_id': u'17eeef21-7ea9-41b8-9344-985b62474a21', u'segmentation_id': 4, u'device_owner': u'compute:nova', u'physical_network': None, u'mac_address': u'fa:16:3e:20:8e:d9', u'device': u'tapedd8d084-e8', u'port_id': u'edd8d084-e8', u'fixed_ips': [{u'subnet_id': u'39eb0e4d-2cdf-42e7-9f1d-b65f9ed409a9', u'ip_address': u'172.16.1.125'}], u'network_type': u'vxlan'}
2015-10-30 10:22:56.864 33315 INFO neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-49aa84e3-1bb5-4865-9a3f-a794fc9bfd63 None] Port tap9d968d11-52 updated. Details: {u'profile': {}, u'admin_state_up': True, u'network_id': u'9c848337-c969-4cd9-9747-204c6b00f17a', u'segmentation_id': 11, u'device_owner': u'compute:nova', u'physical_network': None, u'mac_address': u'fa:16:3e:d9:d8:8d', u'device': u'tap9d968d11-52', u'port_id': u'9d968d11-52', u'fixed_ips': [{u'subnet_id': u'e70e8631-eda5-493d-938e-baf5a1b6482e', u'ip_address': u'172.16.2.96'}], u'network_type': u'vxlan'}
2015-10-30 10:22:59.570 33315 INFO neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-49aa84e3-1bb5-4865-9a3f-a794fc9bfd63 None] Port tap23273962-1f updated. Details: {u'profile': {}, u'admin_state_up': True, u'network_id': u'17eeef21-7ea9-41b8-9344-985b62474a21', u'segmentation_id': 4, u'device_owner': u'compute:nova', u'physical_network': None, u'mac_address': u'fa:16:3e:58:e9:a0', u'device': u'tap23273962-1f', u'port_id': u'23273962-1f', u'fixed_ips': [{u'subnet_id': u'39eb0e4d-2cdf-42e7-9f1d-b65f9ed409a9', u'ip_address': u'172.16.1.121'}], u'network_type': u'vxlan'}
2015-10-30 10:23:00.571 33315 INFO neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-49aa84e3-1bb5-4865-9a3f-a794fc9bfd63 None] Port tape67d396f-3d updated. Details: {u'profile': {}, u'admin_state_up': True, u'network_id': u'17eeef21-7ea9-41b8-9344-985b62474a21', u'segmentation_id': 4, u'device_owner': u'compute:nova', u'physical_network': None, u'mac_address': u'fa:16:3e:1e:47:e8', u'device': u'tape67d396f-3d', u'port_id': u'e67d396f-3d', u'fixed_ips': [{u'subnet_id': u'39eb0e4d-2cdf-42e7-9f1d-b65f9ed409a9', u'ip_address': u'172.16.1.122'}], u'network_type': u'vxlan'}
2015-10-30 10:23:01.717 33315 INFO neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-49aa84e3-1bb5-4865-9a3f-a794fc9bfd63 None] Port tap68609379-9c updated. Details: {u'profile': {}, u'admin_state_up': True, u'network_id': u'9c848337-c969-4cd9-9747-204c6b00f17a', u'segmentation_id': 11, u'device_owner': u'compute:nova', u'physical_network': None, u'mac_address': u'fa:16:3e:9d:f9:06', u'device': u'tap68609379-9c', u'port_id': u'68609379-9c', u'fixed_ips': [{u'subnet_id': u'e70e8631-eda5-493d-938e-baf5a1b6482e', u'ip_address': u'172.16.2.99'}], u'network_type': u'vxlan'}
2015-10-30 10:23:04.459 33315 INFO neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-49aa84e3-1bb5-4865-9a3f-a794fc9bfd63 None] Port tap5aeaaf61-47 updated. Details: {u'profile': {}, u'admin_state_up': True, u'network_id': u'9c848337-c969-4cd9-9747-204c6b00f17a', u'segmentation_id': 11, u'device_owner': u'compute:nova', u'physical_network': None, u'mac_address': u'fa:16:3e:91:43:e0', u'device': u'tap5aeaaf61-47', u'port_id': u'5aeaaf61-47', u'fixed_ips': [{u'subnet_id': u'e70e8631-eda5-493d-938e-baf5a1b6482e', u'ip_address': u'172.16.2.94'}], u'network_type': u'vxlan'}

We've seen these take even longer when there are more VMs, ports, and iptables rules.

Does this sound normal? Is there a way to improve this?

Thanks!

Verified Answer
  • Two updates, for those who land here:

    1. On improving the performance of iptables processing in the Neutron agent: 

    2. On the number of iptables rules I had: over 4000 of them were stale on a single host! Here's the quick and dirty script I wrote to tell me which chains didn't have matching tap interfaces:

    #!/bin/bash
    RULE_DEVICES=`sudo iptables-save | grep :neutron-linuxbri-i | sed 's/:neutron-linuxbri-i//' | awk '{print $1}'`
    TOTAL_COST=0
    for DEV in $RULE_DEVICES; do
    IFACE=`ip a | grep $DEV`
    if [ -z "$IFACE" ]; then
    COST=`sudo iptables-save | grep $DEV | wc -l`
    TOTAL_COST=$(($TOTAL_COST + $COST))
    echo "Device $DEV has no matching interface, cost = $COST rules"
    fi
    done
    echo "Total cost of orphaned rules = $TOTAL_COST"

    And then the resolution: restart neutron-linuxbridge-agent. We'll be putting checks in place tomorrow to compare the output of

    iptables-save | grep :neutron-linuxbri-i | wc -l

    with

    ip a | grep tap | wc -l

    Hope this saves someone from the frustration I experienced.

All Replies
  • Hi evanstoner,

    I tried to contact jamesdenton, but he is out of the office until next week. Is this an urgent issue that I need to escalate through another group, or can I bring this to his attention when he gets back into the office?

    Thanks,

    Stuart

  • Hey Stuart, it's not too urgent -- we would like to resolve it within the next few weeks, but for now we are able to work around it by tearing down older experiments. If you could bring it up when he gets back in, that would be great.

    Thanks!

  • Hi evanstoner,

    I sent him a note asking to look at this thread. I'll check back in next week to see if he has replied.

    Thanks,

    Stuart

  • Two updates, for those who land here:

    1. On improving the performance of iptables processing in the Neutron agent: 

    2. On the number of iptables rules I had: over 4000 of them were stale on a single host! Here's the quick and dirty script I wrote to tell me which chains didn't have matching tap interfaces:

    #!/bin/bash
    RULE_DEVICES=`sudo iptables-save | grep :neutron-linuxbri-i | sed 's/:neutron-linuxbri-i//' | awk '{print $1}'`
    TOTAL_COST=0
    for DEV in $RULE_DEVICES; do
    IFACE=`ip a | grep $DEV`
    if [ -z "$IFACE" ]; then
    COST=`sudo iptables-save | grep $DEV | wc -l`
    TOTAL_COST=$(($TOTAL_COST + $COST))
    echo "Device $DEV has no matching interface, cost = $COST rules"
    fi
    done
    echo "Total cost of orphaned rules = $TOTAL_COST"

    And then the resolution: restart neutron-linuxbridge-agent. We'll be putting checks in place tomorrow to compare the output of

    iptables-save | grep :neutron-linuxbri-i | wc -l

    with

    ip a | grep tap | wc -l

    Hope this saves someone from the frustration I experienced.