Thank you for visiting the Rackspace Community
The The Community is live! Post new content or topics so our teams can assist.

Please contact your support team if you have a question or need assistance for any Rackspace products, services, or articles.

Basic IPTABLES troubleshooting. Fix it now!

Basic IPTABLES troubleshooting.

This is generally a in depth issue Iptables can be complicated. But bare with us. We will try to simplify it.Troubleshooting iptables is time consuming and can really just flat out be frustrating.  Get used to it, this won't ever change no matter how experienced with iptables you are. I will take this time to note I wrote this with centos/fedora/rhel in mind. Though a chunk of this can be applied to Ubuntu, and debian servers.

Quick guide to iptables:

  1. Rules start at the top and go down. Unless told to do otherwise.
  2. A rule will match exactly, if you don't give a rule enough information, unwanted packets will sneak in.
  3. Once iptables accepts, rejects, or drops a packet it will not process any further rules on it.

Generally there are two things you will be troubleshooting iptables for. 
A. It is not accepting traffic and it should not.
B. It is accepting traffic and it really shouldn't be doing that. 

If the server is randomly blocking or accepting traffic this may take more troubleshooting, sigh. Worst yet it may not even be related to iptables. 


Try to stay focused on the problem and don't per-maturely come to conclusions. Troubleshooting is a time consuming process and it takes patience. Don't guess. If you have a setup of 800 rules, guess what? You will need to look at all 800 rules until you find the problem. 

Alright,before we get started you need to know a few things about traffic. It's dangerous don't play in the street. Ok in all seriousness let me list these out for you:

  • What is the source ip or range that you are having problems connecting to? 
  • What is the destination ip address of the website, or service?
  • What is the port or port range affected or what type of traffic is it? (tcp, udp,icmp, etc)
  • Is it supposed to block it or allow it through? 

Going through this list will get us started going through your problem child that is iptables. Accept in those rare instances it goes outside the scope of this article. 

Keep these things in mind (Especially if you didn't write every rule yoruself.)
You will troubleshoot up to down. Depending on the type of traffic we are looking at.

  • Iptables has three chains built in. 
    • These are for INPUT. - Traffic going to the server. 
    • OUTPUT - Traffic leaving the server.
    • FORWARD - Traffic that is not intended to go to or from the server.(Typically this is set to act as a firewall for other servers). 
  • Target = This is the action that is used if a rule matches. This could be a custom chain. So if you see a rule with another chain as the target that matches exactly, be sure to look at each rule in that chain. In the following example you will see BLACKLIST2 sub-chain that applies to the traffic on port 80, it will be diverted to this other chain.
  • Return - This target indicates that you should return to the parent chain. If you see a rull that matches with a return target, stop all your troubleshooting on the current chain and return the rule that is directly after the rule referenced in the custom chain. If there are no matches the default policy is applied.                                 Read more about the confusing RETURN here.
  • Tables - the following tables may have a rule that blocks or diverts your traffic. "nat", "mangle", or "raw". Usually all the rules will be in the "filter" table but you might run into situations where this most definitely isn't the case. One way to check this is to check the following: 
  • iptables -t mangle -nL ; iptables -t nat -nL; iptables -t raw -nL
  •  Be aware of the policy. If the policy is ACCEPT all traffic that does not match a rule will be accepted. Vice-Versa if the policy is DROP or REJECT all traffic that does not match a rule is rejected. 

Here is an example rule set we will use: 

Chain INPUT (policy DROP)
target prot opt source destination
BLACKLIST2 tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:50
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:1010



Chain BLACKLIST2 (1 references)
target prot opt source destination
REJECT * -- 123.123.123.123 0.0.0.0/0
REJECT * -- 45.34.234.234 0.0.0.0/0
ACCEPT * -- 0.0.0.0/0 0.0.0.0/0


Ok Here is the issue: You are serving ssh traffic to everyone. But you want to only allow your ip 1.1.1.1 to have access on port 22. Since we know this is inbound traffic it's obvious it will affect the INPUT chain. 

We are looking for:

  • Source ip: any
  • destination ip: any
  • Protocol(prot):tcp
  • Port: 22

  1. Step 1: The first denotes that any source IP and any destination IP on the destination port 80. Since we are working with port 22 this rule doesn't match, so we will continue to the next rule. If traffic here was on port 80 it would invoke the BLACKLIST2 sub chain.
  2.  Step 2: The second rule denotes any source IP and any destination IP on destination port 50. Since this is regarding port 22, this rule does not match, so let’s continue on.
  3. Step 3: This third rule denotes any source IP and any destination IP on the port 22. Since this is talking about port 22 this rule matches perfectly so it continues on. Whoop!
  4. Step 4: This fourth rule denotes any source IP and any destination IP on port 22. Since this also regards port 22 it is an exact match. The target ACCEPT is applied to the traffic. 
 We found the problem now we need to come up with a solution.
Alright, let's save the iptable rule set to a file.
iptables-save > current-iptables-rules
Now  edit the current-iptables-rules file in your favorite editor and find the rule that looks like this:
-A INPUT -p tcp --dport 22 -j ACCEPT
Now we have this line we need to load the iptables configuration from this file for testing.
iptables-restore <  current-iptables-rules
You should not edit the /etc/sysconfig/iptables as you may *really* put yourself in a bad situation. And you get locked out of your server. Thank goodness for Rackspace Control Panel Terminal Console. should this happen to you it will  allow you to get into your server to fix your iptables. YAY for Rackspace! Any way by just copying the file current-iptables-rules to the iptables-restore if you do get locked out just reboot and be able to log in, since  we didn't run a /sbin/service iptables save it's not permanently set yet. 
Now ruleset should look like this:
Chain INPUT (policy DROP)
target prot opt source destination
BLACKLIST2 tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:50
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT tcp -- 111.111.111.111 0.0.0.0/0 tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:1010

Chain BLACKLIST2 (1 references)
target prot opt source destination
REJECT * -- 123.123.123.123 0.0.0.0/0
REJECT * -- 45.34.234.234 0.0.0.0/0
ACCEPT * -- 0.0.0.0/0 0.0.0.0/0


 The Chain Drop will now drop any connections except port 22 since it specifies the source ip. Also note that 0.0.0.0 is in the sub-chain BLACKLIST2 so anything with the IP 0.0.0.0 will be rejected. 

Once we know that the rule is doing what it should (test this from a separate IP, I will give further details on this below) once tested we can write the tables to the system configuration (i.e /etc/sysconfig/iptables).

service iptables save


Testing Your firewall:
For this example we will use port 80.
 # 1 hit it with a hammer if it doesn't break. You are good to go. 

  • To find out if your ports are open or closed run the following:
  • netstat -tulpn
  •  
  • To find out if tcp port is open or not:
  • netstat -tulpb |grep :80
  •  
  •  If the port isn't open start your apache service:
  • service httpd start
  •  
  • Make sure iptables is running:
  • iptables -L  INPUT  -v  -n |grep 80
  •  
  • If you do not see port 80 open for ALL users:
  • iptables -A INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT
  • /sbin/service iptables save

From a remote location or server use telnet to connect to your server:
telnet x.x.x.x 80

Sample:

Trying 75.126.153.206...
Connected to www.cyberciti.biz.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
 
 
 
Try to see if the port is open from a remote location with nmap:
nmap -sS -p 80 x.x.x.x
 
Sample:
Starting Nmap 5.00 ( http://nmap.org ) at 2011-12-13 13:19 IST
Interesting ports on www.cyberciti.biz (75.126.153.206):
PORT   STATE SERVICE
80/tcp open  http
Nmap done: 1 IP address (1 host up) scanned in 1.00 seconds 
 

Great IPTABLES resources: