Tuesday, March 18, 2014

Nice kdump hack: get dmesg only

Last week during a kernel debugging trainig, I was asked by a participant if it would be possible to get only the dmesg of the crashed kernel, without capturing the whole crash dump.
The possibility is clear, since both current RHEL/CentOS versions as well as SLES11SP3 already put a "dmesg.txt" next to the vmcore in the crash dump directory.
But how would you achieve to get only the dmesg?
And why would one want that?
Well, the second question is easily answered: in order to deploy crash dump capturing in a large hardware pool, quite some preparation needs to be done. In my daily work, servers most of the time have more RAM than they have local disk storage, so you need to store the dumps on the network. Then you need to make sure that a large amount of crashing servers (a famous example was the leap second bug) does not fill up the storage and leads to further problems like machines not coming up again due to full storage etc. All solvable, but to be considered before deployment. If you just capture the dmesg, you can almost certainly store that locally without creating problems. Another reason would be to get the servers up again as soon as possible, while still capturing some useful information (dumping a few hundreds of gigabytes of RAM can take quite some time).

So how to do it?
SUSE's kdump infrastructure (tested on SLES11SP3) has a configuration option KDUMP_PRESCRIPT which allows to give a custom script which will be run before the crash dump is captured. This script now needs to call vmcore-dmesg and save the output somewhere for later inspection, then unmount the rootfs and issue reboot -f. Since this script never returns, the regular core-collector will not run. Problem solved.

The script is actually pretty trivial, so that it can be pasted here:
#!/bin/sh
# small script which can be used as KDUMP_PRESCRIPT in SLES
# it *only* saves the dmesg of the crashed kernel and then
# reboots immediately, *no* crash dump is saved.
# benefits:
# * get the machine up ASAP, while still collecting
# some useful information.
# * can be always enabled without worrying about storage etc
#
# License: WTFPL v2
NOW=`date +%Y-%m-%d-%H%M`
OUT=/root/var/crash/vmcore-dmesg-${NOW}.txt
# in SUSE kdump initrd, real rootfs is mounted to /root
PRG=/root/usr/sbin/vmcore-dmesg # SLES12
test -x $PRG || PRG=/root/sbin/vmcore-dmesg # SLES11SP3
$PRG /proc/vmcore > $OUT
umount /root
reboot -f # do not continue the kdump initrd


It is slightly more complicated than absolutely necessary, but it should work in newer releases which now put the tools in /usr/sbin, too.
In my case, I saved it to /usr/local/sbin/vmcore-dmesg.sh and then changed the following in /etc/sysconfig/kdump:
KDUMP_REQUIRED_PROGRAMS="/usr/local/sbin/vmcore-dmesg.sh"
KDUMP_PRESCRIPT="/usr/local/sbin/vmcore-dmesg.sh"

After restarting kdump, the next crash gave me a nice:
sles11sp3:~ # ls -l /var/crash/
total 36
-rw-r--r-- 1 root root 33383 Mar 18 09:33 vmcore-dmesg-2014-03-18-0911.txt

and no crash dump, mission accomplished.