Remote Syslog - IPTables - Tigers and Bears OH MY

I have been working on a simple SYSLOG project to gather scanning data off of host around the internet and centralize it. The no telling what I will do with the data at that point. Its a project I thought of a while back and I nicknamed it TopTalkers. Basically this is the setup I am going for, very simple a remote syslog server with two host sending it traffic.

I would grow the idea from there with more host and then I wanted to write the session data to a PostgresDB. Kind of a poor mans SEIM or a smart mans method of not paying Cisco for Splunk or Microsoft for Sentinel.

I had worked on a project like this a couple of years/versions of Debian ago and it was always a debate between rsyslog and syslog-ng. After setting up the three servers I logged into the syslog01 server and went into /var/log/ and things looked different. First off there was a README file in the directory!

I am going to move the sidebar conversation to the blog instead of taking up this whole technical discussion with journald vs syslog debate. Alright lets get back to some remote logging shall we???

So I am going to start this off and load rsyslog and UFW on the logger servers then I will go from there. I am using UFW for basic firewall configs it makes things way easier when building basic rules. <cli>root@logger01:~# apt install rsyslog ufw</cli>

I started this with the plans of using UFW to keep it simple things got pretty detailed pretty quick so I ended up switching directly to iptables. I may end up doing another install with netfilter to keep up with the times.

Alright so now I am actually going to use a little bit of help from Google Gemini. At one time in life I loved spending the whole afternoon messing with iptables rules these days I am just going to ask Google and let Gemini kick it out. A bit of warning here the first method/policy Gemini gave me locked me out of the system on the first line. The firewall rules are only temporary and are lost on reboot unless you load the “iptables-persistent” package after you have things applied and tested.

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j LOG --log-prefix "FIREWALL_LOG: PERMIT "
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j LOG --log-prefix "FIREWALL_LOG: DROP "
-A OUTPUT -o lo -j ACCEPT

After you have tested access back into your device over SSH you can now issue the persistent command to maintain the rules at startup. Turns on iptables-persistent isn't installed by default. So install it then run it and accept the warning to save the current policy.

apt install iptables-persistent

After iptables-persistent is install you can now run iptables-save to your policy file and it will load again after reboots.

root@logger02:~# iptables-save > /etc/iptables/rules.v4

While digging into the rules I was just using the iptables -L command to list the policy. This command gives some basic down and dirty output and I have found using iptabels -L -v with the verbose option gives much better results with a packet counter and the interface information that is left out of the basic -L command.

root@logger02:~# iptables -L -v
Chain INPUT (policy DROP 14136 packets, 706K bytes)
 pkts bytes target     prot opt in     out     source               destination         
19294 4901K ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED
   12   680 LOG        tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh LOG level warn prefix "FIREWALL_LOG: PERMIT "
 1250  110K ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh
   12   804 ACCEPT     all  --  lo     any     anywhere             anywhere            
  247 11960 LOG        all  --  any    any     anywhere             anywhere             LOG level warn prefix "FIREWALL_LOG: DROP "

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 21944 packets, 5353K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   24  1944 ACCEPT     all  --  any    lo      anywhere             anywhere            
root@logger02:~# 

Alright I have the basics running now but need to address a couple of things.

1. We are logging a ton of traffic like all of my SSH connectivity. I need to tune this down and get into some rate limiting or things will get messy really quick. DONE
2. Even though I have loaded RSYSLOG all logs are still going to the systemd logs. I want to keep the system logs pointed to systemd (let it do its thing) and have all of the firewalls logs sent to /var/log for text based processing. I thought this was going to be an option, turns out log flexibility has gone away now that the world of Linux has push to systemd

Important considerations:

Order of rules: The LOG rule must be at the very beginning of the chain to log all packets. Log level: Adjust the –log-level to suit your needs and logging system. Levels range from 0 (highest severity) to 7 (debug). Rate limiting: Logging all packets can generate a lot of log data. Use the –limit and –limit-burst options to rate-limit logging if needed.

By implementing this iptables policy, a firewall that allows SSH on port 22, denies all other inbound traffic, and logs both accepted and denied packets with the desired prefix can be created.