Multiple stage servers in open build service

 The open build service publisher has a configuration variable in BSConfig.pm, where you can define a rsync server to publish the built repos to.

Unfortunately, the documentation apart from the actual code (in src/backend/bs_publish function sync_to_stage) seems scarce, so let's document one non-standard usage here.

"Standard" (IMO) usage, one rsync staging server:

our $stageserver = 'rsync://my-rsync/obs-repos-module';

 But now, for a transition phase, I need multiple rsync servers that are all synced to. The format for this is a perl array variable reference that contains pairs of "project name regex, array of sync servers". This also allows to sync different repositories to different rsync servers for example.

The simplest use of this is "sync all repos to multiple rsync servers" and is configured like this:

our $stageserver = ['.*', ['rsync://rsync1/module1', 'rsync://rsync2/module2', 'rsync://rsync3/module3']];

 With this configuration, bs_publish will send all repos to the three mentioned rsync URLs in turn.

Fun with grub2-install "not a directory"

These days, I came across spontaneous kiwi image build failures in a private OBS instance.
The images were SLES15-SP1, they had not been touched for quite some time, rebuilds were only triggered due to new packages in the update channel.
The error was grub2-install failing with the error message "not a directory".
Looking at the recent changes in the update repo showed no obvious reason (some python packages that had nothing to do with grub2-install), so I started to investigate...

... 3 days later, after following some detours, I finally found the issue.

grub2-install scans the installation device for filesystems, and probes all known (to grub2) fs types. The probe of "minix_be" fails fatally. Sometimes.

After building my own grub2 package with lots of debug-printf's, I finally found out, that the minix fs detection of grub2 is a little "fragile". It does the following (pseudo code):

• grub_mount_minix(device) || return "not minix fs"
• grub_minix_find_file("/") || fatal_error()

The problem is, that grub_mount_minix() only does pretty simple magic numbers checks, which can lead to false positives.

Comparing the superblock structures of ext[234] and minix filesystems (from the grub2 source code) side by side, you see this:

struct grub_minix_sblock         |struct grub_ext2_sblock
{                                |{
  grub_uint16_t inode_cnt;       |  grub_uint32_t total_inodes;
  grub_uint16_t zone_cnt;        |
  grub_uint16_t inode_bmap_size; |  grub_uint32_t total_blocks;
  grub_uint16_t zone_bmap_size;  |
  grub_uint16_t first_data_zone; |  grub_uint32_t reserved_blocks;
  grub_uint16_t log2_zone_size;  |
  grub_uint32_t max_file_size;   |  grub_uint32_t free_blocks;
  grub_uint16_t magic;           |  grub_uint32_t free_inodes;
};

This already hints at the issue: at the same disk location where ext2 stores the free inodes number, minix stores its magic number, which is used by grub to detect if it is a minix file system.

Now if you happen to have an ext3 file system with a free_inodes number whose lower 16 bits resemble one of the GRUB_MINIX_MAGIC numbers, chances are grub_mount_minix() will succeed, but then the attempt to acces the root directory will fail with a fatal error.

This is a plain grub2 bug, which I will probably report upstream and try to get fixed.
However, I need a fix to have my images build again, and the chances of getting a fix into SLES15-SP1 are ... low (and it is a daunting task, even if you are reporting this bug as a big SLES customer), so I built a workaround in my (locally built, lucky me...) python-kiwi package.

It basically does the following, before calling the "chroot grub2-install ...".

• statvfs() to get the free_inodes number
• check if the lower 16 bits resemble one of the MINIX_MAGIC numbers
• if it does, touch a temporary file in
• unmount and mount again to update the superblock (I missed this at first and wondered why it did not work)
• unlink the temporary file
• continue as before

This workaround is ugly as hell, but it does work for me.

P.S.: the detours included first noticing that almost every change I made to the image, like wrapping grub2-install into a wrapper script for debugging) made the issue go away (because of a different free_inodes number), so I always needed to check after every change that the issue was still present, then finding that copying the locales in grub2-install actually triggers an ENOTDIR - "Not a directory", because it misses special handling the /usr/share/locale/locale.alias file. Of course I thought "this is the issue" and patched it out of grub2, just to find that the original problem still persisted... then overnight package updates in SLES15-SP1 making this problem go away and reappear seemingly random... you guess it 😄

When "# needsrootforbuild" in OBS does not work...

...always remember, that you also need to change /usr/lib/obs/server/BSConfig.pm:

# Allow to build as root, exceptions per package
# the keys are actually anchored regexes
our $norootexceptions = {
        "my-project/root-package" => 1,
        "dev-projects.*/other-package" => 1,
};

I already forgot that and wondered why it worked for "root-package", but not for "other-package" (which was not yet added...)

My KIWI/OBS talk from oSC'16

Last Friday, at openSUSE Conference 2016, I was giving a talk together with Christian Schneemann about KIWI and OBS (the events.opensuse.org software is not able to manage "two speakers for one talk", this is why I am not listed in the schedule).

The slides from that talk are now available from the B1-Systems website.

Use your distro's kernel in OBS

The Open Build Service has the nifty feature that you can tell it to use a specific kernel to boot the worker VMs that build your software. To use that, you don't need any special setup, just a package which contains a kernel and an initrd:

   /.build.kernel.kvm # used by KVM workers
   /.build.kernel.xen # used by Xen workers
   /.build.initrd.kvm
   /.build.initrd.xen

So you just need this package and make sure it is installed in the VM using the VMinstall: tag in the project config.
If the build service worker script detects that after preparing the VM, such a kernel and initrd are present, they will be used for booting the worker VM that finally builds your package or image. If it is *not* detected, then the kernel the worker server is running with (usually a SUSE kernel) will also be used for the VM.

In the openSUSE Buildservice instance, all "recent" SUSE distributions are configured for that: they use the kernel-obs-build package, which gets created automatically when building the kernel rpms.

Now I am right now using a buildservice instance for cross-distribution package- and imagebuilds. The challenges of trying to build RHEL/CentOS 7 images with KIWI in OBS warrant at least one additional blog post, but one thing I noticed was, that some of the kiwi stuff, when done with a CentOS 7 userland, apparently also needs a CentOS kernel, otherwise kiwi's parted calls, for example, will exit with code 1 (without issuing an error message, btw).
So I have built a kernel-obs-build from the CentOS 7 kernel and configured my OBS instance to use it, which brought me quite some steps further to building CentOS images with KIWI in OBS.
The code (or rather: the spec files) to "convert" the CentOS kernel to an OBS kernel is at https://github.com/seife/kernel-obs-build, a short README on how  to use it is included.

Note that right now it only works with KVM workers as I was not able to get the worker code to boot the kernel correctly in a Xen VM, even though drivers are all there, the reason is probably that the obs worker scripts rely on some of the specifics of a Xen-specific kernel (e.g. the device name of the block devices being passed through to the VM from the config, which is not true for a generic PV-capable kernel).
But I guess this will improve soon, now that openSUSE has dropped the kernel-xen package, they will face the same issues and hopefully someone will fix them ;)