Multiple stage servers in open build service

 The open build service publisher has a configuration variable in BSConfig.pm, where you can define a rsync server to publish the built repos to.

Unfortunately, the documentation apart from the actual code (in src/backend/bs_publish function sync_to_stage) seems scarce, so let's document one non-standard usage here.

"Standard" (IMO) usage, one rsync staging server:

our $stageserver = 'rsync://my-rsync/obs-repos-module';

 But now, for a transition phase, I need multiple rsync servers that are all synced to. The format for this is a perl array variable reference that contains pairs of "project name regex, array of sync servers". This also allows to sync different repositories to different rsync servers for example.

The simplest use of this is "sync all repos to multiple rsync servers" and is configured like this:

our $stageserver = ['.*', ['rsync://rsync1/module1', 'rsync://rsync2/module2', 'rsync://rsync3/module3']];

 With this configuration, bs_publish will send all repos to the three mentioned rsync URLs in turn.

Fun with grub2-install "not a directory"

These days, I came across spontaneous kiwi image build failures in a private OBS instance.
The images were SLES15-SP1, they had not been touched for quite some time, rebuilds were only triggered due to new packages in the update channel.
The error was grub2-install failing with the error message "not a directory".
Looking at the recent changes in the update repo showed no obvious reason (some python packages that had nothing to do with grub2-install), so I started to investigate...

... 3 days later, after following some detours, I finally found the issue.

grub2-install scans the installation device for filesystems, and probes all known (to grub2) fs types. The probe of "minix_be" fails fatally. Sometimes.

After building my own grub2 package with lots of debug-printf's, I finally found out, that the minix fs detection of grub2 is a little "fragile". It does the following (pseudo code):

• grub_mount_minix(device) || return "not minix fs"
• grub_minix_find_file("/") || fatal_error()

The problem is, that grub_mount_minix() only does pretty simple magic numbers checks, which can lead to false positives.

Comparing the superblock structures of ext[234] and minix filesystems (from the grub2 source code) side by side, you see this:

struct grub_minix_sblock         |struct grub_ext2_sblock
{                                |{
  grub_uint16_t inode_cnt;       |  grub_uint32_t total_inodes;
  grub_uint16_t zone_cnt;        |
  grub_uint16_t inode_bmap_size; |  grub_uint32_t total_blocks;
  grub_uint16_t zone_bmap_size;  |
  grub_uint16_t first_data_zone; |  grub_uint32_t reserved_blocks;
  grub_uint16_t log2_zone_size;  |
  grub_uint32_t max_file_size;   |  grub_uint32_t free_blocks;
  grub_uint16_t magic;           |  grub_uint32_t free_inodes;
};

This already hints at the issue: at the same disk location where ext2 stores the free inodes number, minix stores its magic number, which is used by grub to detect if it is a minix file system.

Now if you happen to have an ext3 file system with a free_inodes number whose lower 16 bits resemble one of the GRUB_MINIX_MAGIC numbers, chances are grub_mount_minix() will succeed, but then the attempt to acces the root directory will fail with a fatal error.

This is a plain grub2 bug, which I will probably report upstream and try to get fixed.
However, I need a fix to have my images build again, and the chances of getting a fix into SLES15-SP1 are ... low (and it is a daunting task, even if you are reporting this bug as a big SLES customer), so I built a workaround in my (locally built, lucky me...) python-kiwi package.

It basically does the following, before calling the "chroot grub2-install ...".

• statvfs() to get the free_inodes number
• check if the lower 16 bits resemble one of the MINIX_MAGIC numbers
• if it does, touch a temporary file in
• unmount and mount again to update the superblock (I missed this at first and wondered why it did not work)
• unlink the temporary file
• continue as before

This workaround is ugly as hell, but it does work for me.

P.S.: the detours included first noticing that almost every change I made to the image, like wrapping grub2-install into a wrapper script for debugging) made the issue go away (because of a different free_inodes number), so I always needed to check after every change that the issue was still present, then finding that copying the locales in grub2-install actually triggers an ENOTDIR - "Not a directory", because it misses special handling the /usr/share/locale/locale.alias file. Of course I thought "this is the issue" and patched it out of grub2, just to find that the original problem still persisted... then overnight package updates in SLES15-SP1 making this problem go away and reappear seemingly random... you guess it 😄

grub2-set-default and submenus

At work, I was investigating why "grub2-set-default" apparently did not work for booting older kernels on a SLES12-SP1 system (openSUSE has the same configuration). A colleague found out, that it worked once he removed the "submenu" wrap around the additional installed kernels.

Today, after some googling, I found out that even though you have unique "menuentry" titles, a plain "grub2-set-default my\ menu\ entry" still does not work, unless you give the path to the submenu.

This is done in grub2 syntax like this:

grub2-set-default "1>openSUSE Leap 42.2, with Linux 4.4.46-11-default"

The "1>" tells grub2 to look for the menuentry in the submenu, which is the second toplevel item. For SUSE / openSUSE the second toplevel item is always, AFAICT, the "Advanced options for $VERSION" menu, where the additional kernels live.

An alternative for my case would have been

grub2-set-default "1>1"

Which would be "the second entry from the submenu which is the second toplevel item" (counting from zero). But you need to look at the config file and count the entries.

The entries have an additional ID that looks like it is costructed like:

gnulinux-$(uname -r)-advanced-${UUID_OF_ROOTFS}

 in my case:

gnulinux-4.4.46-11-default-advanced-b073628b-5ddc-4a2d-9943-0f2999dfdaaa

Still looks unwieldy, but you might be able to automatically determine that from a script.

When "# needsrootforbuild" in OBS does not work...

...always remember, that you also need to change /usr/lib/obs/server/BSConfig.pm:

# Allow to build as root, exceptions per package
# the keys are actually anchored regexes
our $norootexceptions = {
        "my-project/root-package" => 1,
        "dev-projects.*/other-package" => 1,
};

I already forgot that and wondered why it worked for "root-package", but not for "other-package" (which was not yet added...)

"Ghost" keystrokes with libvirt/KVM, SPICE and Windows guests

After offline resizing the image and file system of a Windows guest VM running on KVM, like this:

dd if=/dev/zero of=wxp.img bs=1M seek=10240 count=0
fdisk -c=dos wxp.img # resize partition, activate(!)
losetup -Pf wxp.img
ntfsresize /dev/loop0p1
losetup -d /dev/loop0

Windows (as expected) wanted to run a file system check on next boot. And on the following boot. And... every time.

I investigated and found out, that the CHKDSK prompted for "skip this check with any key press" and apparently a key was pressed at every boot, even though I did not touch anything.

Long story short: apparently the SPICE drivers, which this VM is using, are creating "ghost" devices and events during boot, which are interpreted as key presses by Windows. The solution was pretty simple: shut down the VM, switch the configuration from "SPICE server" to "VNC server", boot, wait for the CHKDSK to finish, shut down, switch back to "SPICE server".

My KIWI/OBS talk from oSC'16

Last Friday, at openSUSE Conference 2016, I was giving a talk together with Christian Schneemann about KIWI and OBS (the events.opensuse.org software is not able to manage "two speakers for one talk", this is why I am not listed in the schedule).

The slides from that talk are now available from the B1-Systems website.

Use your distro's kernel in OBS

The Open Build Service has the nifty feature that you can tell it to use a specific kernel to boot the worker VMs that build your software. To use that, you don't need any special setup, just a package which contains a kernel and an initrd:

   /.build.kernel.kvm # used by KVM workers
   /.build.kernel.xen # used by Xen workers
   /.build.initrd.kvm
   /.build.initrd.xen

So you just need this package and make sure it is installed in the VM using the VMinstall: tag in the project config.
If the build service worker script detects that after preparing the VM, such a kernel and initrd are present, they will be used for booting the worker VM that finally builds your package or image. If it is *not* detected, then the kernel the worker server is running with (usually a SUSE kernel) will also be used for the VM.

In the openSUSE Buildservice instance, all "recent" SUSE distributions are configured for that: they use the kernel-obs-build package, which gets created automatically when building the kernel rpms.

Now I am right now using a buildservice instance for cross-distribution package- and imagebuilds. The challenges of trying to build RHEL/CentOS 7 images with KIWI in OBS warrant at least one additional blog post, but one thing I noticed was, that some of the kiwi stuff, when done with a CentOS 7 userland, apparently also needs a CentOS kernel, otherwise kiwi's parted calls, for example, will exit with code 1 (without issuing an error message, btw).
So I have built a kernel-obs-build from the CentOS 7 kernel and configured my OBS instance to use it, which brought me quite some steps further to building CentOS images with KIWI in OBS.
The code (or rather: the spec files) to "convert" the CentOS kernel to an OBS kernel is at https://github.com/seife/kernel-obs-build, a short README on how  to use it is included.

Note that right now it only works with KVM workers as I was not able to get the worker code to boot the kernel correctly in a Xen VM, even though drivers are all there, the reason is probably that the obs worker scripts rely on some of the specifics of a Xen-specific kernel (e.g. the device name of the block devices being passed through to the VM from the config, which is not true for a generic PV-capable kernel).
But I guess this will improve soon, now that openSUSE has dropped the kernel-xen package, they will face the same issues and hopefully someone will fix them ;)

FITRIM/discard with qemu/kvm for thin provisioning

My notebook computer is running with an SSD, and usually I'm creating logical volumes for the KVM VM's I install on it for testing purposes. On my normal file systems, I regularly run "fstrim" manually, to help the SSD firmware figure out which blocks can be reused. However, the LV's of the virtual machines usually stayed un-TRIM'ed. I had heard, that KVM/QEMU now supports the discard commands, but had not yet gotten to testing it.

I finally got to figuring out how it works:

First, you need to switch the VM to using virtio-scsi instead of virtio-blk:

Before:

<disk type='block' device='disk'>
  <driver name='qemu' type='raw'/>
  <source dev='/dev/main/factory'/>
  <target dev='vda' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/
</disk>

After:

<disk type='block' device='disk'>
  <driver name='qemu' type='raw'/>
  <source dev='/dev/main/factory'/>
  <target dev='sda' bus='scsi'/>
  <address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='scsi' index='0' model='virtio-scsi'/>

Note the added scsi controller, and the only things you need to change are "target" and "address", if your source is different, that's ok.

Now check that your VM still boots. If it does not, then it is missing the virtio-scsi driver in the initrd. Reboot with the old configuration and build an initrd that includes all drivers, or at least the virtio-scsi driver. Another possible problem is the change from "/dev/vda1" to "/dev/sda1", check your fstab and use UUID or filesystem label for booting. Both problems did not occur to me on a stock Factory install (it uses UUID by default and had all drivers in initrd), but a hand-built kernel (built with "make localmodconfig"...) failed to boot, so be prepared.

Now you are using virtio-scsi for your device, but fstrim will still give you a "operation not supported" message. You'll need another parameter in your VM's configuration:

<driver name='qemu' type='raw' discard='unmap'/>

Restart the VM, and...

factory-vm:~ # fstrim -v /
/: 8,7 GiB (9374568448 bytes) trimmed
factory-vm:~ # 

Now what about thin-provisioning?

I converted the same VM from LV to a plain raw file.

This is the file on the host, it is sparse:

susi:/local/libvirt-images # ls -lh factory.raw
-rw-r----- 1 qemu qemu 20G Mar 15 14:05 factory.raw
susi:/local/libvirt-images # du -sh factory.raw
12G     factory.raw

Now let's delete some stuff inside the VM and run fstrim:

factory-vm:~ # du -sh /home/seife/linux-2.6/
3.9G    /home/seife/linux-2.6/
factory-vm:~ # rm -rf /home/seife/linux-2.6/
factory-vm:~ # fstrim -v /
/: 12.7 GiB (13579157504 bytes) trimmed

Checking again on the host:

susi:/local/libvirt-images # ls -lh factory.raw
-rw-r----- 1 qemu qemu 20G Mar 15 14:08 factory.raw
susi:/local/libvirt-images # du -sh factory.raw
6.4G    factory.raw

So this is really neat, as you now can free up space on the host after cleaning up in the VM. Maybe I should reconsider my "put all VMs into logical volumes" strategy again, as this wastes quite some valuable SSD space in my case.

Is anyone still using PPP at all?

After talking to colleagues about how easy it is to contribute to the Linux Kernel simply by reporting a bug, I was actually wondering why I was the first and apparently the only one to hit this bug.
So there are two possible reasons:

• nobody is testing -rc kernels
• nobody is using PPP anymore

To be honest, I'm also only using PPP for some obscure VPN, but I would have expected it to be in wider usage due to UMTS/3G cards and such. So is nobody testing -rc kernels? This would indeed be bad...

Building Yocto/Poky on openSUSE Factory

Since a few weeks, openSUSE Factory no longer is labeled as "openSUSE Project 13.2", but as:

seife@susi:~> lsb_release -ir
Distributor ID: openSUSE project
Release: 20140918

When trying to build the current Yocto poky release, you get the following Warning:

WARNING: Host distribution "openSUSE-project-20140918" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.

Now I know these warnings and have ignored those before. The list of tested distributions is hard coded in the build system configuration and in general it would be a bad idea to add not yet released versions (as 13.2) or rolling releases. And since the Factory release number changes every few days, it is clearly impossible to keep this up to date: once you have tested everything, the version has increased already. But apart from this, purely cosmetic warning, there is a really annoying consequence of the version change: the configuration cache of bitbake (the build tool used by Yocto poky/OpenEmbedded) is rebuilt on every change of the host distribution release. Updating the cache takes about 2 minutes on my machine, so doing a simple configuration check on your already built Yocto distribution once a week can get quite annoying. I looked for a solution and went for the "quick hack" route:

• bitbake parses "lsb_release -ir"
• I  replace "lsb_release" with a script that emits filtered output and is before the original lsb_release in $PATH

This is what I have put into ~/bin/lsb_release (the variable check is a bit of paranoia to let this only have an effect in a bitbake environment):

#!/bin/bash

if [ -z "$BB_ENV_EXTRAWHITE" -o "x$1" != "x-ir" ]; then

        exec lsb-release $@

fi

printf "Distributor ID:\topenSUSE project\nRelease:\t2014\n"

Then "chmod 755 ~/bin/lsb_release" and now the warning is

WARNING: Host distribution "openSUSE-project-2014" has not been validated...

And more important: it stays the same after updating Factory to the next release. Mission accomplished.

UPDATE: Koen Kooi noted that "Yocto" is only the umbrella project and what I'm fixing here is actually the "poky" build system that's part of the project, so I edited this post for clarity. Thanks for the hint!

Fix openSUSE's grub2 for Virtualization and Servers

After installing current openSUSE Factory in a VM, I found that the old GRUB option was removed from YaST2. I knew this from the mailing list, but now I actually realized that this happened. I still prefer GRUB over GRUB2, because for me it is easier to manage. But being lazy, I just went with the default.
Everything went well, until I added a customized kernel (I had installed the VM to do some kernel experiments after all). The boot menu suddenly was not very useful anymore. After selecting "advanced options", I got the following:

Well, which one of the four is now my hand-built, brand new kernel?
There is no such thing as in old GRUB where "Esc" got you out of gfxboot mode and into text mode. The command keys, like "e" for editing the current selection and "c" for a GRUB2 shell (something even more hellish than the old GRUB shell apparently) work, but you really need to know this, as there is no indication of that.

So I wanted to get rid of the gfxboot stuff. I don't need fancy, I need it usable.
Booted the VM, logged in. "zypper rm grub2-branding-openSUSE" followed by "grub2-mkconfig > /boot/grub2/grub.cfg". Much better:

But still it is in graphics mode, which I do not care about now, but once I have to deploy this stuff on something like an HP server where you can get a text console via SSH, but only if it is in plain VGA mode, I will not be amused. So boot that VM again, and look further. Finally, the solution is in /etc/default/grub: "GRUB_TERMINAL=console". The comment above says just uncommenting the original "gfxterm" setting would be enough, but it is not. After recreating the config file and rebooting, it looks quite useful:

And it is not even missing information, compared to the gfxterm version... no idea why this stuff is default.

Now that "Distribution" string in there looks completetly redundant, so getting rid of that will help, too.
Again, it is in /etc/default/grub, variable GRUB_DISTRIBUTOR. I see that in the grub2 rpm package, there is only "openSUSE" instead of  "openSUSE Factory Distribution", so it might be put into the config by the installer or something. I'll change it to just "Factory" (to distinguish between other openSUSE installations). After grub2-mkconfig, it looks almost good:

Now the important information (Kernel version) is completely visible. Much better than the original "bling bling" screen, which had no useful information at all...
Just fixing the Factory string would probably have helped also, but it still would fail the server test, so plain console will stay my favorite for now.

Nice kdump hack: get dmesg only

Last week during a kernel debugging trainig, I was asked by a participant if it would be possible to get only the dmesg of the crashed kernel, without capturing the whole crash dump.
The possibility is clear, since both current RHEL/CentOS versions as well as SLES11SP3 already put a "dmesg.txt" next to the vmcore in the crash dump directory.
But how would you achieve to get only the dmesg?
And why would one want that?
Well, the second question is easily answered: in order to deploy crash dump capturing in a large hardware pool, quite some preparation needs to be done. In my daily work, servers most of the time have more RAM than they have local disk storage, so you need to store the dumps on the network. Then you need to make sure that a large amount of crashing servers (a famous example was the leap second bug) does not fill up the storage and leads to further problems like machines not coming up again due to full storage etc. All solvable, but to be considered before deployment. If you just capture the dmesg, you can almost certainly store that locally without creating problems. Another reason would be to get the servers up again as soon as possible, while still capturing some useful information (dumping a few hundreds of gigabytes of RAM can take quite some time).

So how to do it?
SUSE's kdump infrastructure (tested on SLES11SP3) has a configuration option KDUMP_PRESCRIPT which allows to give a custom script which will be run before the crash dump is captured. This script now needs to call vmcore-dmesg and save the output somewhere for later inspection, then unmount the rootfs and issue reboot -f. Since this script never returns, the regular core-collector will not run. Problem solved.

The script is actually pretty trivial, so that it can be pasted here:

#!/bin/sh
# small script which can be used as KDUMP_PRESCRIPT in SLES
# it *only* saves the dmesg of the crashed kernel and then
# reboots immediately, *no* crash dump is saved.
# benefits:
# * get the machine up ASAP, while still collecting
#   some useful information.
# * can be always enabled without worrying about storage etc
#
# License: WTFPL v2
NOW=`date +%Y-%m-%d-%H%M`
OUT=/root/var/crash/vmcore-dmesg-${NOW}.txt
# in SUSE kdump initrd, real rootfs is mounted to /root
PRG=/root/usr/sbin/vmcore-dmesg                 # SLES12
test -x $PRG || PRG=/root/sbin/vmcore-dmesg     # SLES11SP3
$PRG /proc/vmcore > $OUT
umount /root
reboot -f       # do not continue the kdump initrd

It is slightly more complicated than absolutely necessary, but it should work in newer releases which now put the tools in /usr/sbin, too.
In my case, I saved it to /usr/local/sbin/vmcore-dmesg.sh and then changed the following in /etc/sysconfig/kdump:

KDUMP_REQUIRED_PROGRAMS="/usr/local/sbin/vmcore-dmesg.sh"
KDUMP_PRESCRIPT="/usr/local/sbin/vmcore-dmesg.sh"

After restarting kdump, the next crash gave me a nice:

sles11sp3:~ # ls -l /var/crash/
total 36
-rw-r--r-- 1 root root 33383 Mar 18 09:33 vmcore-dmesg-2014-03-18-0911.txt

and no crash dump, mission accomplished.

Fix coolstream neo tuner voltage problem

After trying to use the EN50494 feature of neutrino-mp, I found that as soon as I attached the coolstream neo to my coax-"bus", all other receivers on the same bus could no longer tune any frequency. After quite some investigation, I found out, that the tuner does not lower the voltage to 13V if there is less than about 10mA of current load on the "LNB in" coax output.
I tried to work around the problem in software. However, that did not work too well, because if the neo was the only active receiver on the bus, then the SCR matrix would shut down if there was no voltage applied. So I checked if there is a way to simply fix the broken hardware.
There is. Next to the tuner "tin box" there is a transistor that has the 14/19V at one of its terminals. Just adding a 1.1kOhm resistor from there to ground made the voltage switch correctly even without any coax cable attached.

1.1kOhm resistor fixes tuner voltage problem

 The picture is not very good but it shows where the LNB power can be tapped.Note: use this at your own risk, soldering the resistor to the wrong terminal or shortening the wrong solder points might very well kill your box. You have been warned.
It is also very well possible that better soldering points exist, however this solution can be implemented without disassembling the box completely by just soldering on the top side of the PCB.

It would be interesting if this modification also solves the huge amount of DiSEqC switching problems that were reported last spring after some software update, it surely solved my EN50494 (aka unicable) bus blocking problem. After all the vendor has issued no statement until today, even though this clearly looks like broken (by design) hardware...

"rdmsr" implemented in perl

Today I needed the "rdmsr" tool to determine if machines are configured correctly for Hypervisor usage (VT-X enabled). Just after learning how to detect this (by looking at the cpu-checker ubuntu package), I found out that msr-tools is not available on SLES11. Instead of building the package, I just implemented a minimal version in perl (it will be integrated into a perl tool checking other aspects of the system anyway):

#!/usr/bin/perl
# License: WTFPLv2
# minimal "rdmsr" implementation
$msr = shift
        or die "need msr as parameter";
$msr = hex($msr) if ($msr =~ m/^0x/i);
open(FD, "/dev/cpu/0/msr")
        or die "open /dev/cpu/0/msr: $!";
sysseek(FD, $msr, SEEK_SET)
        or die "sysseek: $!";
sysread(FD, $reg, 8 ) == 8
        or die "sysread: $!";
$reg = reverse($reg);
$hex = unpack('H*', $reg);
$hex =~s/^0*//;
print "$hex\n";

Yes, i know, my perl is horrible :-) but maybe this is useful for someone else anyway.

openSUSE Kernel debuginfo weirdness

Just because I fell into the same trap twice, once when trying kdump a year ago, now when working on systemtap, a short reminder for everyone:

If you want to do something that needs kernel-default-debuginfo installed (like, say, "kdump"/"crash" or "systemtap"), then make sure that you also have kernel-default-devel-debuginfo on your system.

The reason for this is, that the kernel-default-debuginfo package has only the debuginfo for the kernel modules, but it misses the debuginfo for /boot/vmlinux. This debuginfo is in kernel-default-devel-debuginfo package.

This is at least strange, since the vmlinux binary is not in the devel package but in the main kernel package.
But in practice it does not matter if this is a bug or not: you need both debuginfo packages installed to make kdump analysis with "crash" or systemtap work.

ExpressCard hotplug with kernel 3.11

(Executive summary: boot with acpiphp.disable=1)

Kernel 3.11rc has fixed the mei driver suspend problem for me. It brought another quirk, however. ExpressCard hotplug does not work anymore with my trusty old Thinkpad X200s. I need this to use my USB3 card. With 3.11 it only works if it is already plugged in during boot.
To be honest, this has never been 100% automatic before: I always needed to manually load the "pciehp" module to get the slot to work. The other possible driver, "acpiphp" refused to load.
With 3.11, both the acpiphp and the pciehp drivers can no longer be built as modules but are both built in statically.
In addition, now the acpiphp driver claims to support my slot, which it in fact does not, but it still claims the slot and prevents the pciehp driver (which is initialized later) from working. The kernel hackers are working to fix up this mess, but it is probably not going to "just" work in 3.11, so for now the way to get the ExpressCard slot working on a Thinkpad X200s is to pass the boot option

acpiphp.disable=1

to the kernel.

"mei" driver suspend regression in linux-3.10

For all those running kernels from Kernel:HEAD, where 3.10-final has arrived today, be warned that a suspend / resume regression crept in (I spotted and reported that after -rc4 or such, but it has not been fixed).
This regression makes my Thinkpad X200s not resume from suspend to RAM (I have not tried suspend to disk, but it might be affected, too), the machine just freezes hard on resume.
Fortunately, it is relatively easy to work around.
The affected driver is the "mei_me" driver for Intel's Management Engine, which AFAICT allows you to use management functions of the chipset like serial-over lan and similar. I don't need this, but I need a suspend to RAM that works :-)

The work around is to simply unbind the driver after boot, then the machine will suspend and resume fine (blacklisting the module does not work since the driver is built in in the openSUSE kernel).

Find out, which device is bound to the driver:

# find /sys/bus/pci/drivers/mei_me/ -type l|sed 's#^.*/##'
0000:00:03.0

Now unbind (as root):

# echo 0000:00:03.0 > /sys/bus/pci/drivers/mei_me/unbind

...and your next resume will work better.

Using virtio console with KVM

Yesterday I finally figured out how to use the virtio console on KVM, which is very easy once you know what to do.

A short explanation of what a "virtio console" is:
On paravirtualized environments like Xen the VM (guest) has no direct hardware access, but also no emulated hardware but instead needs special drivers that know it is running on a hypervisor. The advantage of this approach is that you don't need to emulate hardware and that the driver for the guest is often much simpler (it does not need to probe hardware, for example, as it talks to the hypervisor via a special interface). This makes for a very convenient default Xen setup where you can just do "xm console " on the hypervisor and get a functional console without any fiddling.
On fully virtualized environments, usually an emulated serial port is used for that, which is harder to set up. It is not impossible, but definitely not that easy and there is quite some stuff to configure to get it working.
Nowadays even fully virtualized systems often have some paravirtualized drivers available, mostly for performance reasons. One of those paravirtualized drivers available with KVM is the virtio console driver.

So back on topic: how to set up the virtio console driver for a KVM guest
First, edit the configuration of your guest, in my case with "virsh edit factorytest" (factorytest is the name of my guest). Add the following snippet of configuration before the closing "</devices>" tag:

    <console type='pty'>
      <target type='virtio' port='0'/>
    </console>

Then a few things need to be done on the KVM guest. In my case of an openSUSE 12.2 guest, the virtio_console module was loaded automatically. However, it surely won't hurt to put it into INITRD_MODULES in /etc/sysconfig/kernel.
The second thing is "console=hvc0" needs to go onto the Kernel command line.
The easiest way to accomplish this is to use the YaST bootloader module and add it in the "Boot Loader Options" tab. You need to actually make sure there is "console=tty console=hvc0" there, because without the "console=tty" the system did not boot up cleanly for me. I guess that the kernel is not happy with only a console that gets added later via a module, but I need to investigate that.

All the rest was handled by systemd and friends just fine, after rebooting the VM guest, I was immedately able to do "virsh console factorytest", hit "Enter", get the login prompt and log in.

Kdump Talk / Paper

This time in english :-)
Tomorrow I'm holding the kdump talk at LinuxTag in Berlin, and since I had submitted it in english, I had to translate my short paper anyway, so here it is. It is a rather short "beginner's guide" but it should get you started. And it's much better than the slides as there is almost no technical details in them due to the short timeslot (30 minutes) I got.

Kdump Vortrag / Paper

(German only, the paper is also only available in german right now).

Zur Zeit halte ich Vorträge über kdump, unter anderem auf dem GUUG Frühjahrsfachgespräch 2012 und den Chemnitzer Linuxtagen 2012.

Da die Dauer der Vorträge meist auf ca. 45 Minuten begrenzt ist, enthalten die Vortragsfolien nur wenig technische Details. Für den Tagungsband des FFG 2012 habe ich jedoch einen Artikel geschrieben, der als Einstieg in das Thema besser geeignet ist. Diesen Artikel gibt es hier.

Viel Erfolg!