Securing your server with Syslog-ng

There are many programs that help you secure your servers. There are simple firewalling programs and there are complex network intrusion detection systems. Depending on your demands, there seems to be a solution for everyone. Unless you’re picky and you think you can do better – like I did, when I wrote my own combination of syslog-ng and Linux iptables, skipping the middle man.

August 2013 update. There seems to be a problem using the syslog-ng file() destination on /proc in Linux. See https://valentijn.sessink.nl/?p=458 for more information.

Lately, we had a two similar security breaches. One had to do with a problem in the Ubuntu pam settings, where pam will ignore everything, including wrong passwords, bad configurations, account expired and the kitchen sink, the other was a — rather stupid — default password that we left in a testing machine that got upgraded to production. Luckily, no animals were harmed; but we decided it to be time to fence off attackers by searching the log files for attacks.

There are many solutions for this. In the past, we had used logcheck, which had a lot of patterns but was not very good at dissecting them (i.e. you could either have or have not something mailed, but no chance of doing something special with the IP address of the attacker). For easy analysis and security there are fail2ban (written in Python), Sagan (written in C), SSHguard (written in Yacc and C). (If you are looking for a complete (and maybe separate) intrusion detection system you may want to use something like Snort).

The problem I had with these solutions is, that they all have a quite complicated scanning path. If, for example, sshd logs an error message, this message will go to 1) your system log; from there to 2) the analysis program; which will, most likely, 3) spawn a script that, in turn, 4) will execute IPtables.  Eek. There are faster ways: SSHguard will call IPtables right away, but at a cost: it’s written in C, with (as far as I could see) the patterns written in Yacc; so getting a new pattern to work would mean a) writing a Yacc pattern, then b) recompile SSHguard. And while I understand the value of recompiling, I normally rather leave that to BSD users.

Luckily, there is an alternative. Syslog-ng has got its own on board pattern recognition logic. That simplifies the path enormously: from sshd to the system logger; and, with a bit of trickery as outlined below, straight to the IPtables firewalling, without calling an external command. Here is what I built.

Step 0. Install syslog-ng

Easy enough.  On our Ubuntu Lucid servers, I added the “backports” repository and installed syslog-ng 3.1 from there. Please note that you do need version 3.1 or newer for the patterndb logic to work.

Step 1.  Get a pattern.

To get started, get the sshd pattern from the Balabit git web frontend:

wget -O sshd.pdb http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=access/sshd.pdb;hb=HEAD

Put it in /etc/syslog-ng/patterndb.d/, then run

pdbtool merge -D /etc/syslog-ng/patterndb.d/ -p /var/lib/syslog-ng/patterndb.xml

Step 2: setup syslog-ng

Add, in the appropriate sections of your /etc/syslog-ng/syslog-ng.conf file:

# a destination; we have a dual destination here, a file to see the
# blocked hosts, and an iptables-destination in /proc to block them.
destination d_syslogblock { file("/proc/net/xt_recent/syslogblock"
    template("+${usracct.device}\n")); file("/var/log/syslogblock"); };
# a parser for the pattern-DB we made in step 1
parser pattern_db {
    db_parser( file("/var/lib/syslog-ng/patterndb.xml")); };
# a filter to filter the parser results
filter f_syslogblock { tags("secevt") and match("REJECT"
    value("secevt.verdict")); };
# and finally, the log itself:
log { source(s_src); parser(pattern_db); filter(f_syslogblock);
destination(d_syslogblock); };

Step 3. Setup firewalling.

A minimalistic firewall could look like:

ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
ip6tables -A INPUT -m state --state INVALID -j DROP
ip6tables -A INPUT -m recent --rcheck --name block \
    --seconds 3600 -j DROP   # see note (1) below
ip6tables -A INPUT -m recent --rcheck --name syslogblock \
    --seconds 900 --hitcount 15 -m recent --name block --set \
    -j LOG --log-prefix "syslogblocked: "

Step 4. Repeat for IPv4

Now repeat step 3, but use “iptables” instead of “ip6tables”. (I used ip6tables on purpose, because it’s time you get ready for IPv6 if you aren’t).

Ready.

Not too hard, isn’t it? It works as follows. Syslog-ng will put the IP-address of any wrong password or username into the /proc/net/xt_recent/syslogblock file, with a “+” in front of the address. This is the iptables way of telling the recent module it should add a certain address. Now the IPtables rules tell, that if someone is seen in this “syslogblock” list more that 15 times within 15 minutes, then (s)he will be added to the “block” list.

The double “-m recent …. -m recent” logic works because iptables does so called “short circuit” evaluation, If someone is not 15 times in the “syslogblock” list, then iptables won’t even bother to evaluate the rest of the line, i.e. it won’t call the “-m recent –name block” test. However, if someone is 15 times there (and it all happened within the last 15 minutes), only then it will see if “-m recent –name block –set” evaluates to “true” as well. And, by definition, it does, because an xt_recent “set” command is always true.

Once you’re in the “block” list, you will be blocked for an hour. Note (1): if you replace the “–rcheck” here with an “–update” statement, the block will last even longer. The “–rcheck” option says: we will block you the next hour. While “–update” says: we don’t want to see you for an hour, but if we see you again during this time, we’ll block you again. So the latter means that you actually need to be quiet for 60 minutes to be able to log in again.

There is only one potential problem in this whole setup: if  syslog-ng inadvertedly sends garbage to the /proc/net/xt_recent/syslogblock file, then this file will be closed. Syslog-ng will reopen it after the “reopen” timeout, so your message will probably come through; but should something change in the working of syslog-ng, like should syslog-ng try to re-send potentially broken messages, then this will, effectively, block the rest of the blocking messages. (I don’t think this will happen, because “fprintf” is not a reliable transport, so to say, but you’ll never know).

I’ll implemented this solution successfully for a couple of servers; I also added a profile for asterisk SIP scanning and I’ll be adding more servers and more profiles in the coming weeks.

Some final remarks: xt_recent has a default tables size of 100 IP addresses, so it won’t eat up all your computer memory. However, this means that having IPv6 addresses in syslogblock may or may not be an advantage. As most of these IPv6 netblocks are /48 or /64, an attacker can easily change source addresses for every connection, thus slowly flushing your syslogblock file. I haven’t seen any IPv6 scans so far, but you might want to keep an eye on the /var/log/syslogblock file. Also, this file is not rotated by default and you may want to change that, too.

Happy blocking!