Sunday, November 16, 2014

Is anyone still using PPP at all?

After talking to colleagues about how easy it is to contribute to the Linux Kernel simply by reporting a bug, I was actually wondering why I was the first and apparently the only one to hit this bug.
So there are two possible reasons:

  • nobody is testing -rc kernels
  • nobody is using PPP anymore
To be honest, I'm also only using PPP for some obscure VPN, but I would have expected it to be in wider usage due to UMTS/3G cards and such. So is nobody testing -rc kernels? This would indeed be bad...

Saturday, November 15, 2014

Switching from syslog-ng to rsyslog - it's easier than you might think

I had looked into rsyslog years ago, when it became default in openSUSE and for some reason I do not remember anymore, I did not really like it. So I stayed at syslog-ng.
Many people actually will not care who is taking care of their syslog messages, but since I had done a few customizations to my syslog-ng configuration, I needed to adapt those to rsyslog.

Now with Bug 899653 - "syslog-ng does not get all messages from journald, journal and syslog-ng not playing together nicely" which made it into openSUSE 13.2, I had to reconsider my choice of syslog daemon.

Basically, my customizations to syslog-ng config are pretty small:

  • log everything from VDR in a separate file "/var/log/vdr"
  • log everything from dnsmasq-dhcp in a separate file "/var/log/dnsmasq-dhcp"
  • log stuff from machines on my network (actually usually only a VOIP telephone, but sometimes some embedded boxes will send messages via syslog to my server) in "/var/log/netlog"
So I installed rsyslog -- which due to package conflicts removes syslog-ng -- and started configuring it to do the same as my old syslog-ng config had done. Important note: After changing the syslog service on your box, reboot it before doing anyting else. Otherwise you might be chasing strange problems and just rebooting is faster.

Now to the config: I did not really like the default time format of rsyslog:
2014-11-10T13:30:15.425354+01:00 susi rsyslogd: ...
Yes, I know that this is a "good" format. Easy to parse, unambiguosly, clear. But It is usually me reading the Logs and I still hate it, because I do not need microsecond precision, I do know in which timezone I'm in and it uses half of a standard terminal width if I don't scroll to the right.
So the first thing I changed was to create /etc/rsyslog.d/myformat.conf with the following content:
$template myFormat,"%timegenerated:1:4:date-rfc3339%%timegenerated:6:7:date-rfc3339%%timegenerated:9:10:date-rfc3339%-%timegenerated:12:21:date-rfc3339% %syslogtag%%msg%\n"
$ActionFileDefaultTemplate myFormat
This changes the log format to:
20141110-13:54:23.0 rsyslogd: ...
Which means the time is shorter, still can be parsed and has sub-second-precision, the hostname is gone (which might be bad for the netlog file, but I don't care) and it's 12 characters shorter.
It might be totally possible to do this in an easier fashion, I'm not a rsyslog wizard at all (yet) ;)

For /var/log/vdr and /var/log/dnsmasq-dhcp, I created the config file /etc/rsyslog.d/myprogs.conf, containing:
if $programname == 'dnsmasq-dhcp' then {
    -/var/log/dnsmasq-dhcp
    stop
}
if $programname == 'vdr' then {
    -/var/log/vdr
    stop
}
That's it! It's really straightforward, I really can't understand why I hated rsyslog years ago :)

The last thing missing was the netlog file, handled by /etc/rsyslog.d/mynet.conf:
$ModLoad imudp.so # provides UDP syslog reception
$UDPServerRun 514 # start syslog server, port 514
if $fromhost-ip startswith '192.168.' then {
    -/var/log/netlog
    stop
}
Again, pretty straightforward.
And that's it! Maybe I'll add an extra logformat for netlog to specify the hostname in there, but that would just be the icing on the cake.

What I liked especially on the rsyslog implementation in openSUSE (it might be default, but I don't know that) is, that the location of the "$IncludeConfig /etc/rsyslog.d/*.conf" is placed so that you really can do useful things without touching the distributions default config. With syslog-ng, the include of the conf.d directory was too late (for me), so you could not "split off" messages from the default definitions, e.g. the VDR messages would appear in /var/log/messages and /var/log/vdr. In order to change this, you had to change the syslog-ng.conf and this would need to be checked after a package update, and new distro-configs would need to be re-merged into my changed configuration.
Now it is totally possible that after an update of the distribution, I will need to fix my rsyslog configs because of changes in syntax or such, but at least it is possible that it might just work without that.

Monday, November 10, 2014

Home server updated to 13.2

Over the weekend, I updated my server at home from 13.1 to openSUSE 13.2.
The update was quite smooth, only a few bugs in apparently seldom used software that I needed to work around:
  • dnsmasq does not log to syslog anymore -- this is bug 904537 now
  • wwoffle did not want to start because the service file is broken, this can be fixed by adding "-d" to ExecStart and correcting the path of ExecReload to /usr/bin instead of /usr/sbin (no bugreport for that, the ExecStart is already fixed in the devel project and I submitrequested the ExecReload fix. Obviously nobody besides me is running wwwoffle, so I did not bother to bugreport)
  • The apache config needed a change from "Order allow,deny", "Allow from all" to "Require all granted", which I could find looking for changes in the default config files. Without that, I got lots of 403 "Permission denied" which are now fixed.
  • mysql (actually mariadb) needed a "touch /var/lib/mysql/.force_upgrade" before it wanted to start, but that's probably no news for people actually knowing anything about mysql (I don't, as you might have guessed already)
  • My old friend bug 899653 made it into 13.2 which means that logging from journal to syslog-ng is broken ("systemd-journal[23526]: Forwarding to syslog missed 2 messages."). Maybe it is finally time to start looking into rsyslog or plain old syslogd...
Because syslog-ng is broken for me, I needed to make the journal persistent and because journald sucks if its data is stored on rotating rust (aka HDDs), I added a separate mount point for /var/log/journal which is backed by bcache like other filesystems on that machine.

Everything seems to be running fine so far, apart from the fact that the system load was at a solid 4.0 all the time. Looking into this I found that each bcache-backed mount point had an associated kernel thread continuously in state "D". Even though this is rather cosmetic, I "fixed" it by upgrading to the latest kernel 3.17.2 from the Kernel:Stable OBS project (who wants old kernels anyway? ;)

Everything else looks good, stuff running fine:
  • owncloud
  • gallery2
  • vdr
  • NFS server
  • openvpn
Of course I have not tried everything (I eed to actually start up one of those KVM guests...), but the update has been rather painless until now.

Thursday, September 25, 2014

Building Yocto/Poky on openSUSE Factory

Since a few weeks, openSUSE Factory no longer is labeled as "openSUSE Project 13.2", but as:
seife@susi:~> lsb_release -ir
Distributor ID: openSUSE project
Release: 20140918
When trying to build the current Yocto poky release, you get the following Warning:
WARNING: Host distribution "openSUSE-project-20140918" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Now I know these warnings and have ignored those before. The list of tested distributions is hard coded in the build system configuration and in general it would be a bad idea to add not yet released versions (as 13.2) or rolling releases. And since the Factory release number changes every few days, it is clearly impossible to keep this up to date: once you have tested everything, the version has increased already. But apart from this, purely cosmetic warning, there is a really annoying consequence of the version change: the configuration cache of bitbake (the build tool used by Yocto poky/OpenEmbedded) is rebuilt on every change of the host distribution release. Updating the cache takes about 2 minutes on my machine, so doing a simple configuration check on your already built Yocto distribution once a week can get quite annoying. I looked for a solution and went for the "quick hack" route:
  • bitbake parses "lsb_release -ir"
  • I  replace "lsb_release" with a script that emits filtered output and is before the original lsb_release in $PATH
This is what I have put into ~/bin/lsb_release (the variable check is a bit of paranoia to let this only have an effect in a bitbake environment):

#!/bin/bash
if [ -z "$BB_ENV_EXTRAWHITE" -o "x$1" != "x-ir" ]; then
        exec lsb-release $@
fi
printf "Distributor ID:\topenSUSE project\nRelease:\t2014\n"

Then "chmod 755 ~/bin/lsb_release" and now the warning is
WARNING: Host distribution "openSUSE-project-2014" has not been validated...
And more important: it stays the same after updating Factory to the next release. Mission accomplished.

UPDATE: Koen Kooi noted that "Yocto" is only the umbrella project and what I'm fixing here is actually the "poky" build system that's part of the project, so I edited this post for clarity. Thanks for the hint!

Friday, September 19, 2014

Fix openSUSE's grub2 for Virtualization and Servers

After installing current openSUSE Factory in a VM, I found that the old GRUB option was removed from YaST2. I knew this from the mailing list, but now I actually realized that this happened. I still prefer GRUB over GRUB2, because for me it is easier to manage. But being lazy, I just went with the default.
Everything went well, until I added a customized kernel (I had installed the VM to do some kernel experiments after all). The boot menu suddenly was not very useful anymore. After selecting "advanced options", I got the following:

Well, which one of the four is now my hand-built, brand new kernel?
There is no such thing as in old GRUB where "Esc" got you out of gfxboot mode and into text mode. The command keys, like "e" for editing the current selection and "c" for a GRUB2 shell (something even more hellish than the old GRUB shell apparently) work, but you really need to know this, as there is no indication of that.

So I wanted to get rid of the gfxboot stuff. I don't need fancy, I need it usable.
Booted the VM, logged in. "zypper rm grub2-branding-openSUSE" followed by "grub2-mkconfig > /boot/grub2/grub.cfg". Much better:


But still it is in graphics mode, which I do not care about now, but once I have to deploy this stuff on something like an HP server where you can get a text console via SSH, but only if it is in plain VGA mode, I will not be amused. So boot that VM again, and look further. Finally, the solution is in /etc/default/grub: "GRUB_TERMINAL=console". The comment above says just uncommenting the original "gfxterm" setting would be enough, but it is not. After recreating the config file and rebooting, it looks quite useful:


And it is not even missing information, compared to the gfxterm version... no idea why this stuff is default.

Now that "Distribution" string in there looks completetly redundant, so getting rid of that will help, too.
Again, it is in /etc/default/grub, variable GRUB_DISTRIBUTOR. I see that in the grub2 rpm package, there is only "openSUSE" instead of  "openSUSE Factory Distribution", so it might be put into the config by the installer or something. I'll change it to just "Factory" (to distinguish between other openSUSE installations). After grub2-mkconfig, it looks almost good:


Now the important information (Kernel version) is completely visible. Much better than the original "bling bling" screen, which had no useful information at all...
Just fixing the Factory string would probably have helped also, but it still would fail the server test, so plain console will stay my favorite for now.


Friday, July 18, 2014

Restoring stock recovery on Moto G

Yesterday, my Moto G got an Official update to Android 4.4.4. Yeah!
Unfortunately, it did not work: After downloading the update, the phone went into a boot loop, because CWM 6.0.4.6 (which I had installed in order to root the device) cannot flash the update. The phone then boots up, just to shut reboot almost immediately into recovery to try again.
To get out of this boot loop, I manually entered recovery and wiped the "cache" partition.
I retried with the latest CWM 6.0.4.7, this also did not work.
So I had to get the original stock recovery image for the Moto G and flash that. I did not easily find it with a web search, so in the end I downloaded the matching stock SBF image for my installed firmware (in my case "Blur_Version.176.44.1.falcon_umts.Retail.en.DE") from the Moto G firmware page on droid-developers.org, looked into the zip file and found that there is a "recovery.img" in the archive.

Now everything was easy: boot the phone into fastboot mode (power on + volume down), then
fastboot erase recovery
fastboot flash recovery recovery.img
reboot, once the phone is booted select "install system update" from the notification, stock recovery boots and installs the update, done.
I did not even lose root access, so I kept the stock recovery for now.
Later on I checked the md5sum of the recovery.img I flashed and of the recovery partition on the phone and they are identical, so the 4.4.4 update did not flash a new recovery for me. I'll keep the old one around in case I need it again.

Sunday, June 22, 2014

Recovering my OwnCloud admin user password

Because of its endless awesomeness, I'm running an ownCloud instance on my server at home.
What I'm actually using of it is the WebDAV frontend to up/download stuff from Android (the native ownCloud App seems to have problems with really big files, such as several hundred MB) and the "instant upload" feature for the android camera, to automatically upload photos for easy reusing on the PC.

Today I wanted to configure some stuff and found out that I had totally forgotten the admin password, simply because I never needed it after the initial setup.

Modern applications no longer just store the passwords in the database, so it's not as simple as it could be. Additional problems arise from the fact that I have basically zero database knowledge.
Fortunately, I still knew that I'm using a mariadb on the server...

So that's what I did to restore admin access:

  • cat config/config.php, note values of "dbuser" and "dbpassword"
  • mysql -u <dbuser> -p;
    • paste <dbpassword>
  • show databases;
    • note that there is a database called "owncloud", which is probably the one I need...
  • use owncloud;
  • select * from oc_users;
    • Oh! My admin user is called "root", not "admin" as I would have guessed... Important information. So I try to use the password reset from the web form for user "root", however, it does not work...
  • select * from oc_preferences;
    • Oh! "root" has no email address configured, no wonder the password reset does not work.
    • After some searching, I found the way to the solution in the ownCloud forum:
  • INSERT INTO `oc_preferences` ( `userid` , `appid` , `configkey` , `configvalue` ) VALUES ( 'root','settings','email','root@localhost' );
And voilĂ : the password reset link is working, root get's an email which contains the link to set a new password. Life is good again :-)