Log4j syslogAppender and Rsyslog

This is how I workarounded an annoying situation, working with log4j.syslogAppender (the syslog module from the Java logging library) and Rsyslog togheter, in order to sends Tomcat logs to a remote log analyzer (Splunk, on the same machine with Rsyslog), that understands 'log4j' format

The article is just meant to show some Rsyslog hacks

Of course, I noticed the problem from the log analyzer, which wasn't working because of the confused log format. So firstly, I wanted to have a look how the logs were cominig to Rsyslog

To show all the fields, I've enabled the RSYSLOG_DebugFormat, next to the entry in /etc/rsyslog.conf

local1.info /var/log/log_analyzer/application.log;RSYSLOG_DebugFormat

This the awfull result:

Debug line with all properties:
FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: '2013-10-25', PRI: 142,
syslogtag '19:', programname: '19', APP-NAME: '19', PROCID: '', MSGID: '-',
TIMESTAMP: 'Oct 25 19:38:51', STRUCTURED-DATA: '-',
msg: '38:51,915 INFO

As shown, those fields are totally messed up (HOSTNAME: '2013-10-25', MSGID: '-' ..) The reasons behind this "unmatching" log4j/rsyslog fields, could be either a misconfiguration log4j side (don't ask me where since the configuration is minimal) or the broken syslogAppender module. Reading on the web I was convincing myself much for the latter

So, my decision at the moment is to bypass as much as I can, the Rsyslog handling of those logs in order to store the logs exactly as they come from log4j

These are the steps

1 - Let's define a custom template
Rsyslog has its default template, so for first we need to setup a custom template (Log4j, which basically will avoid any formatting for our logs) and make it point to our "local1.info" entry

$template Log4j, "%rawmsg%\n"
local1.info /var/log/log_analyzer/application.log;Log4j

2 - Turn the control character escaping off
With %rawmsg%, our logs will look like this

<142>2013-10-25 20:20:53,862 INFO  ["http-80"] org.apache.cxf.interceptor: Inbound Message#012Content-Type: xxx#012Headers: xxx

As we can see the string has many #012 in the middle. This represents a form feed control character. The default global option in Rsyslog is meant to escape the control characters, so let's disable it

$EscapeControlCharactersOnReceive off

Verify the Rsyslog configuration

lurch~# rsyslogd -N1
...
rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad imuxsock
rsyslogd: End of config validation run. Bye.

And restart the service

lurch~# service rsyslog restart 

Let's have a look to log again

<142>2013-10-25 20:20:53,862 INFO  ["http-80"] org.apache.cxf.interceptor: Inbound Message
Content-Type: xxx
Headers: xxx

Much better but

3 - Remove some chars from the string
Our string still begins with a <142>. This is the PRIO value in Rsyslog. Since the message is a RAW message, I assume that the PRIO is sent directly from log4j, so it wont be possible to play with Rsyslog fields, in order to remove it
So I decide to cut off the first characters in the string(<142>), in the template definition

$template Log4j, "%rawmsg:6:$%\n"

Finally:

2013-10-25 20:20:53,862 INFO  ["http-80"] org.apache.cxf.interceptor: Inbound Message
Content-Type: xxx
Headers: xxx

LXC and cgroup.memory on Debian

Two days ago on Lurch, I was trying to show/set a memory limit for a container (LXC), using "lxc-cgroup -n <container> memory.limit_in_bytes"

Unfortunately, I got the message "lxc-cgroup: missing cgroup subsystem", that I've firstly intended as "I couldn't mount this cgroup at this session"

Briefly, asking about memory cgroup to LXC, everything was ok

while asking to linux not

Another confusing point to me, was the check of the dmesg output, that showed memory cgroup between the others

So, after a little of googling, I have understood like, the memory cgroup  is just not enabled on Debian by default. That because having the cgroup.memory enabled, costs around 15Mb of ram, that is obviously a waste if you don't use that cgroup

In order to have the availability of said cgroup, you need to instruct the Grub by /etc/default/grub with the boot parameter cgroup_enable=memory

The amount of memory reserved to the cgroup nos is printed out during the boot time

In the end I could set my cgroup memory limit

Pxe with Dnsmasq

Just a couple of words..

For a much too long period of my life I have always manually changed the "pxelinux" entry in the "dnsmasq.d/domain.conf" file, to achieve the boot with this and with that image depending to the needed install distro

I just have finally found a way to serve each pxe images in one shot and even with a confortable menu list

That is accomplished adding something like:
Needless to say that you could even make point each pxe entry to a different tftp servers

VirtualBox port-forwarding

Today on my Munich-Florence train, I've waisted some time just trying some ways for "sshing" my puppet VirtualBox guest. Usually DHCPclient does everything and, once the ip address is got by the guest, I can easily "ssh" into it. This time, I had not a wifi connection and I've encoutered some annoying bounces on my ssh attempts. It looked like the guest was unreachable. I've instinctively tried every "nat", "bridged", "host-only" options.. and in the end .. it touched me to read the documentation :) (by the way, a very good paper) 

..and it was so i've discovered this official best practice to port-forward services on a VirtualBox guest

In this example (exactly extracted from the doc), we are going to portforwarding the SSH service, from our HOST 8888 port, to the GUEST (vm name: "Puppet Test Machine") on 22 port:

"sshService" is just a label.

In this way, our HOST will keep the forward on each interfaces. Anyway, it's possible to bind a specific interface though.

Now that we have our forward ready, we can connect the loopback on the given port
This rule will be permanet unless you explicitly remove it. See the rule properties:
Then delete the rule: