Remote Syslog - IPTables - Tigers and Bears OH MY
I have been working on a simple SYSLOG project to gather scanning data off of host around the internet and centralize it. The no telling what I will do with the data at that point. Its a project I thought of a while back and I nicknamed it TopTalkers. Basically this is the setup I am going for, very simple a remote syslog server with two host sending it traffic.
I would grow the idea from there with more host and then I wanted to write the session data to a PostgresDB. Kind of a poor mans SEIM or a smart mans method of not paying Cisco for Splunk or Microsoft for Sentinel.
I had worked on a project like this a couple of years/versions of Debian ago and it was always a debate between rsyslog and syslog-ng. After setting up the three servers I logged into the syslog01 server and went into /var/log/ and things looked different. First off there was a README file in the directory!
I am going to move the sidebar conversation to the blog instead of taking up this whole technical discussion with journald vs syslog debate. Alright lets get back to some remote logging shall we???
Configuring iptables (the hell of logging denies)
So I am going to start this off and load rsyslog and UFW on the logger servers then I will go from there. I am using UFW for basic firewall configs it makes things way easier when building basic rules.
<cli>root@logger01:~# apt install rsyslog ufw</cli>
I started this with the plans of using UFW to keep it simple things got pretty detailed pretty quick so I ended up switching directly to iptables. I may end up doing another install with netfilter to keep up with the times.
Alright so now I am actually going to use a little bit of help from Google Gemini. At one time in life I loved spending the whole afternoon messing with iptables rules these days I am just going to ask Google and let Gemini kick it out. A bit of warning here the first method/policy Gemini gave me locked me out of the system on the first line. The firewall rules are only temporary and are lost on reboot unless you load the “iptables-persistent” package after you have things applied and tested.
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p tcp -m tcp --dport 22 -j LOG --log-prefix "FIREWALL_LOG: PERMIT " -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -j LOG --log-prefix "FIREWALL_LOG: DROP " -A OUTPUT -o lo -j ACCEPT
Saving iptables config and setting persistence after reboots
After you have tested access back into your device over SSH you can now issue the persistent command to maintain the rules at startup. Turns on iptables-persistent isn't installed by default. So install it then run it and accept the warning to save the current policy.
apt install iptables-persistent
After iptables-persistent is install you can now run iptables-save to your policy file and it will load again after reboots.
root@logger02:~# iptables-save > /etc/iptables/rules.v4
Displaying iptables rules
While digging into the rules I was just using the iptables -L command to list the policy. This command gives some basic down and dirty output and I have found using iptabels -L -v with the verbose option gives much better results with a packet counter and the interface information that is left out of the basic -L command.
root@logger02:~# iptables -L -v Chain INPUT (policy DROP 14136 packets, 706K bytes) pkts bytes target prot opt in out source destination 19294 4901K ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED 12 680 LOG tcp -- any any anywhere anywhere tcp dpt:ssh LOG level warn prefix "FIREWALL_LOG: PERMIT " 1250 110K ACCEPT tcp -- any any anywhere anywhere tcp dpt:ssh 12 804 ACCEPT all -- lo any anywhere anywhere 247 11960 LOG all -- any any anywhere anywhere LOG level warn prefix "FIREWALL_LOG: DROP " Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 21944 packets, 5353K bytes) pkts bytes target prot opt in out source destination 24 1944 ACCEPT all -- any lo anywhere anywhere root@logger02:~#
The drama of logging in Linux
I didn't know in this journey to consolidate logs to a centralized server I was going to end up blogging so much. Turns out I needed to start another post to talk about the usage of systemd's journald vs rsyslog. I went with rsyslog but you can read the post if you want to know why.
Going against the Linux Gods
Now we need to shut off kernel logging in the journald. I go into details about this in the blog posting mentioned above, but I don't like the idea of cutting off kernel logs going to journald. You're hands are kind of tied in this scenario and loading rsyslog takes you back to the legacy method of logging which isn't that bad. Everything is in /var/log again.
Install rsyslog and disable kernal logging in journald
Basic rsyslog install is fine
apt install rsyslog
Here is a my /etc/rsyslog.conf file from the log servers. This prevents FW logs from showing up in the kernel log and sends them to their own log file (easier to work with)
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
###############
#### RULES ####
###############
# Forward messages with "FIREWALL_LOG:" prefix to a separate file
:msg, contains, "FIREWALL_LOG:" /var/log/fw.log
# This will forward all of the Firewall Logs to the syslog01 server from /etc/hosts over UDP 514
:msg, contains, "FIREWALL_LOG:" @syslog01:514
# Discard messages with the "FIREWALL_LOG:" prefix
:msg, contains, "FIREWALL_LOG:" stop
# Send all other messages to the journald socket - not sure this is needed I am doing it all in syslog now might remove later.
*.* action(type="omuxsock" Socket="/run/systemd/journal/syslog")
#
# Log anything besides private authentication messages to a single log file
#
*.*;auth,authpriv.none -/var/log/syslog
#
# Log commonly used facilities to their own log file
#
auth,authpriv.* /var/log/auth.log
cron.* -/var/log/cron.log
kern.* -/var/log/kern.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
Then edit /etc/systemd/journald.conf and uncomment and change the ReadKMsg=yes flag to ReadKMsg=no
root@logger02:~# vim /etc/systemd/journald.conf
# Use 'systemd-analyze cat-config systemd/journald.conf' to display the full config.
#
# See journald.conf(5) for details.
[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
MaxLevelSyslog=warn
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
ReadKMsg=no
#Audit=no
Once you have rsyslog configured and logging disabled going to journald you need to restart both logging services. Then you should start seeing firewall logs (fw.log) in your /var/log/ directory.
systemctl restart systemd-journald.service systemctl restart rsyslog.service
Alright I have the basics running now but need to address a couple of things.
The Syslog Server - syslog01
Now you should see both servers “logger02 and logger01” in the log file on your syslog server.
2025-06-11T04:32:24+00:00 logger02 kernel: [514514.807037] FIREWALL_LOG: DROP IN=eth0 OUT= MAC=62:4b:80:04:8f:5d:fe:00:00:00:01:01:08:00 SRC=5.188.206.54 DST=209.38.71.201 LEN=40 TOS=0x00 PREC=0x00 TTL=235 ID=22090 PROTO=TCP SPT=8080 DPT=3344 WINDOW=1024 RES=0x00 SYN URGP=0 2025-06-11T04:32:24+00:00 logger01 kernel: [ 3309.873056] FIREWALL_LOG: DROP IN=eth0 OUT= MAC=f6:5a:cb:13:e6:33:fe:00:00:00:01:01:08:00 SRC=79.124.8.112 DST=137.184.122.230 LEN=40 TOS=0x00 PREC=0x20 TTL=242 ID=54321 PROTO=TCP SPT=41609 DPT=34567 WINDOW=65535 RES=0x00 SYN URGP=0
We are logging a ton of traffic like all of my SSH connectivity. I need to tune this down and get into some rate limiting or things will get messy really quick.DONE- Even though I have loaded RSYSLOG all logs are still going to the systemd logs. I want to keep the system logs pointed to systemd (let it do its thing) and have all of the firewalls logs sent to /var/log for text based processing.
I thought this was going to be an option, turns out log flexibility has gone away now that the world of Linux has push to systemd
Important considerations:
Order of rules: The LOG rule must be at the very beginning of the chain to log all packets. Log level: Adjust the –log-level to suit your needs and logging system. Levels range from 0 (highest severity) to 7 (debug). Rate limiting: Logging all packets can generate a lot of log data. Use the –limit and –limit-burst options to rate-limit logging if needed.
By implementing this iptables policy, a firewall that allows SSH on port 22, denies all other inbound traffic, and logs both accepted and denied packets with the desired prefix can be created.
CONTACT ME: if you think I missed something, need to update anything, or you just have questions. Follow the contact rules or spam filters will get you.