Every now and then at Rackspace, as with any hosting provider, we do occasionally have issues where customers have left themselves open to attack. Either by failing to lock down their server, or by missing the additional steps we advise to secure the server. In such cases sometimes customers find their server is sending spam email, and is even prone to other malware.

"One of the greatest issues with this kind of event is effectively auditing; the process of tracking down the source of traffic, specifically which process which the packet originates from, specifically."

Due to AUP and other obligations, it can become a critical issue for both the uptime, and reputation of your site. In many cases, customers do not necessarily have forensic experience, and will struggle very hard to remove the malware. In some cases, the malware keeps on coming back, or, like in my customers case, you could see lots of extra network traffic still using tcpdump locally on the box.

The NetFilter package is included in the Linux 2.4.x kernel, it is a replacement for ipchains.

Image credit: csh.rit.edu.

Enter, netfilter, part of the Linux Kernel, and it is able, if you ask it, to track down where packets are coming from, on a process level. This is really handy if you have an active malware or spam process on your system, since you can find out exactly where it is, before doing more investigation. Such a method, also allows you to trace down any potential false positives, since the packet address is always included, you get a really nice overview.

To give you an idea, I needed to install a kernel with debuginfo, just to do this troubleshooting, however this depends on your distribution.

Updating your Kernel may be necessary to use netfilter debug

$yum history info 18

Transaction performed with:
    Installed     rpm-4.11.3-21.el7.x86_64                               @base
    Installed     yum-3.4.3-150.el7.centos.noarch                        @base
    Installed     yum-plugin-auto-update-debug-info-1.1.31-40.el7.noarch @base
    Installed     yum-plugin-fastestmirror-1.1.31-40.el7.noarch          @base
Packages Altered:
    Updated kernel-debuginfo-4.4.40-202.el7.centos.x86_64               @base-debuginfo
    Update                   4.4.42-202.el7.centos.x86_64               @base-debuginfo
    Updated kernel-debuginfo-common-x86_64-4.4.40-202.el7.centos.x86_64 @base-debuginfo
    Update                                 4.4.42-202.el7.centos.x86_64 @base-debuginfo


As below, you could use a similar process using netfilter.ip.local_in, for monitoring packets coming in and the application they come from. If this is something you think you will need to use often, and you wish to avoid the overhead of enabling debug in your kernel, you could potentially consider using netdata instead. But here's how you do it directly with netfilter and your own BASH script.

The NetFilter Script

#! /usr/bin/env stap

# Print a trace of threads sending IP packets (UDP or TCP) to a given
# destination port and/or address.  Default is unfiltered.

global the_dport = 0    # override with -G the_dport=53
global the_daddr = ""   # override with -G the_daddr=127.0.0.1

probe netfilter.ip.local_out {
    if ((the_dport == 0 || the_dport == dport) &&
        (the_daddr == "" || the_daddr == daddr))
            printf("%s[%d] sent packet to %s:%d\n", execname(), tid(), daddr, dport)
}



Executing the Script


[root@pirax-test-new hacked]# chmod +x dns_probe.sh
[root@pirax-test-new hacked]# ./dns_probe.sh
Missing separate debuginfos, use: debuginfo-install kernel-3.10.0-514.2.2.el7.x86_64
swapper/3[0] sent packet to 78.136.44.6:0
sshd[25421] sent packet to 134.1.1.1:55336
sshd[25421] sent packet to 134.1.1.1:55336
swapper/3[0] sent packet to 78.136.44.6:0



I was a little bit concerned about the above output, it looks like process 'swapper' with 'pid (process id) 3', is doing something it wouldn't normally do. Upon further inspection though, we find it is just the outgoing cloud monitoring call which relates to the Rackspace Cloud Monitoring Agent;

# nslookup 78.136.44.6
Server:        83.138.151.81
Address:    83.138.151.81#53

Non-authoritative answer:
6.44.136.78.in-addr.arpa    name = collector-lon-78-136-44-6.monitoring.rackspacecloud.com.

This should summarise well the process of how you can troubleshoot this stuff in a more advanced way using the Kernel. Certainly, if deploying from scratch is not an option, this is one way to get an idea of the processes and the process ID's in the linux Kernel, utilizing the TCP/UDP Network stack.

Remember when enabling debug, that it is important to disable it afterwards as there is a performance overhead from using a debug kernel which is significant and there might be some security implications.

After you have finished working be sure to reinstall your reference kernel again, this can normally be done by running an apt-get remove against the kernel you installed, but check with your technical contact first before proceeding.