Due to popular demand this weeks report contains another awesome ascii screenshot. I added some lines to /etc/hurd/runsystem.sysv to inspect the environment very early in the boot process:

start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
[...]
+ echo my pid is 1
my pid is 1
+ ps Ax
+ head
USER       PID TT STAT     TIME COMMAND
-            0  ? R<mo  0:00.01 /hurd/proc
root         1  - Sslow 0:00.02 /bin/sh /etc/hurd/runsystem
-            2  - Sp    0:00.00 /hurd/init root=device:hd0s1 console=com0
-            3  ? D<p   0:00.01 root=device:hd0s1 console=com0
-            4  - S<o   0:00.18 ext2fs --readonly --multiboot-command-line=root
-            5  - S<o   0:00.02 /hurd/exec
-            6  - S<o   0:00.00 /hurd/auth
root         7  - S<o   0:00.04 /hurd/term /dev/console device console
root         8  - S<o   0:00.00 /hurd/pflocal
+ pstree -p
sh(1)-+-init(2)-+-auth(6)
      |         `-ext2fs(4)-+-exec(5)
      |                     |-pflocal(8)
      |                     |-procfs(12)
      |                     `-term(7)
      |-pstree(11)
      `-root=device:hd0s1(3)

Those who are familiar with the Hurd might note that /etc/hurd/runsystem.sysv runs as PID 1. At the end of this script /sbin/init is exec(2)uted, so sysvinit gets run as PID 1. This is the result of a fairly small patch series. It still took me like two days to come up with it since bootstrapping a Hurd system (or any microkernel OS, or like bootstrapping in general) is complicated.

If you look at the screenshot above you will note the /hurd/pflocal process running with PID 8. This is the server providing pipe(2)s and Unix sockets. Before that server is started, one cannot use pipes to connect two processes with pipes like I did with ps and head (I actually lied a little here, because the way Debian/Hurd is set up, the pflocal server is started on demand courtesy of a Hurd feature called "passive translators", very similar to what the cool kids call "socket activation" today).

Process 0 is the /hurd/proc server. It maps Mach tasks to the concept of Unix processes. Before that server is started, the notion of a process does not even exist. Pid 4 is the root filesystem. Without that, no other server can be read from the disk. But if the root filesystem is to be started first, it cannot be registered as a Unix process at the proc server, because that has not been started yet.

So the first few processes are started in a weird way and order, and since they need to talk to each other, they need to register at each other. No Unix concepts exist yet, only Mach concepts like tasks and messages can be used. The boot sequence is roughly this (the numbers in parenthesis are the PIDs):

1. Grub loads GNU Mach(3), the statically linked root filesystem translator (4) and the /hurd/exec (5) server (the exec server loads executables into Mach tasks, implementing most parts of execve(2)).
2. The rootfs translator is started.
3. The rootfs initiates the Hurd server bootstrap by starting /hurd/exec and /hurd/init (2).
4. /hurd/init starts /hurd/proc (0) and /hurd/auth (6).
5. The translators perform several rendezvouses to introduce themselves to each other.
6. The proc servers state with respect to the already running processes is fixed.
7. /hurd/init starts /etc/hurd/runsystem (1) which starts the pager (that pages out memory to disk) and later execs /sbin/init.
8. From this point on the initialization of Debian/Hurd is no different from Debian/Linux.

Besides the pid one issue I spent my time on:

• A prototypical patch for the other sysvinit related issue, /sbin/killall5 wreaking havoc on a Hurd system: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00226.html
• I updated my patch series binding the mtab translator to /proc/mounts: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00216.html
• I updated my patch series for daemonizing support for console-client: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00246.html

Next week I hope to fully address the killall5 issue and to clean up, polish and submit my patch series for sysvinit. I will then revise my mtab translator prototype which is currently in its third iteration. As always, if there is time left, I will look at the network related issues.

I'm sure you all will be a bit disappointed (I know I am) that there are no ascii screenshots in this weeks report. But let me make it up to you by telling you a story.

I was at the FOSDEM in the year 2012 and I went to a nice workshop on sunday afternoon in the Virtualization Devroom. Renzo Davoli was the host of this workshop and he started with a little introductory talk to get everyone to agree on a common terminology first. He began by asking the question, what virtualization meant in the most general way. He defined the ability to virtualize a resource as the ability to freely and transparently control how someone (say a process) is interacting with said resource. So for example if you LD_PRELOAD a library to impersonate fopen(3) and friends, you have virtualized the filesystem (for some small values of virtualized). He then went on to discuss various methods of virtualization that are commonly available on Linux, not only full system virtualization solutions, but also all kinds of methods allowing a more fine-grained control over what resources are virtualized. Of course every method had its strengths and its weaknesses.

Having seen Samuels talk about all the awesome things Hurd can do for virtualization I approached him with a question in the free discussion part of his workshop. I asked whether he would agree that once a resource is moved from the kernel to the userspace, the problem of virtualizing that resource is almost trivially solved, and he agreed. So I said that this was awesome, because then the trivial and elegant solution for all his virtualization (and tracing) needs are micro kernel operating systems, and he also agreed to that but (as far as I can recall) he mentioned that there is none that would meet his needs (I believe he is a computer science professor and heavily relies on virtualization techniques for his lab and for teaching purposes).

And even though a microkernel operating systems has a cost (message passing instead of function calls for example) it also has its merits such as scaling better from a development point of view. Also, many cool features can emerge just from the design. For example think about the container support in Linux and how painfully long it took to make all the resources namespace aware (with one of the most critical, the user namespace being the most difficult one). On Hurd you get the same functionality for free. If you are curious, read Samuels awesome slides.

So what have I done this week?

• procfs should ignore --no{dev,exec,suid}: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00076.html
• libnetfs does not respond properly to file_get_translator requests on nodes with a passive translator record: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00148.html
• As the mtab translator is a translator of its own, there needs to be a way to bind it to /proc/mounts. I figured the easiest way would be to just serve a node with an appropriate passive translator record: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00149.html
• There was no clean way to start the console-client. I added daemonizing support to the console-client and created an appropriate initscript: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00165.html
• I also worked a lot on my mtab translator prototype: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00183.html

Next week I'm going to work on the two remaining sysvinit related issues. These are the only ones preventing me to come up with a clean patch series against the sysvinit package and I figured that it would be nice to propose such a series rather sooner than later so that I will have plenty of time to discuss any issues with the sysvinit maintainers.

See you next week :)

Everyone enjoys ASCII screenshots as much as I do, right? So here is the Hurd booting with my mtab-prototype bound to /run/mtab:

INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
/dev/hd0s1: clean, 44292/181056 files, 287759/723200 blocks
done.
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Cleaning up temporary files... /tmp.
Mounting local filesystems...done.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 4
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 42550 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.


GNU 0.3 (debian) (console)

login:

As you can see, the boot process looks quite clean. In particular, there are no issues left with Debians initscripts. The remaining noise comes from:

• The GNU mach kernel, but that's okay. The FreeBSD kernel writes lot's of stuff by default too, and so does Linux if one increases the kernel log level. Also, some of the stuff might be due to me using qemu.
• Network related issues.
• The default pager. This is the translator paging out memory to swap space.

This is accomplished by my mtab translator prototype. It's work in progress, but I have reworked and broken up my initial prototype. My initial prototype only patched libdiskfs, one of three libraries translator authors can use to write filesystem-like translators. I have made the necessary changes to the other two libraries and patched up any loose ends.

This is the translator in action:

% fsysopts /run/mtab
/hurd/mtab /
% cat /run/mtab
/dev/hd0s1 / ext2fs writable,no-inherit-dir-group,store-type=typed 0 0
none /dev/ttyp0 /hurd/term name,/dev/ptyp0,type,pty-master 0 0
none /run /hurd/tmpfs writable,no-suid,no-exec,no-inherit-dir-group,no-sync,size=80484K 0 0
none /run/lock /hurd/tmpfs writable,no-suid,no-exec,no-inherit-dir-group,no-sync,size=5M 0 0
none /run/shm /hurd/tmpfs writable,no-suid,no-exec,no-inherit-dir-group,no-sync,size=259480K 0 0

I also patched the sysvinit package. My version includes and switches to runsystem.sysv that uses /sbin/init to start the system. It also carries all the necessary workarounds for the remaining sysvinit related issues.

The resulting Debian/Hurd system is very nice, it is getting more and more similar to what a Debian user would expect:

• For the first time, df without arguments works:

  % df
  Filesystem     1K-blocks    Used Available Use% Mounted on
  /dev/hd0s1      10005504 4671612   4833620  50% /
  none               80484      44     80440   1% /run
  none                5120       4      5116   1% /run/lock
  none              259480       4    259476   1% /run/shm
  

• The mount utility is accompanied by umount.

• /etc/fstab is honored and auto works as an fstype.

• /run and friends are in fact ram-disks.

• /sbin/shutdown works.

• All init scripts are properly started and stopped.

  • Among other nice things this enables one to manage the network configuration using the Debian way via interfaces(5).

• /etc/inittab is honored.

I have rebuild the hurd and sysvinit package and uploaded them into an apt repository:

deb http://teythoon.cryptobitch.de/gsoc/heap/debian unstable main

Please use unstable for now. Also make sure that you have a recovery plan for your Debian/Hurd installation if anything goes wrong. For your convenience there's a seed tarball containing packages with the appropriate sources.list.d snippets and the repository key:

https://teythoon.cryptobitch.de/gsoc/heap/debian/seed.tar

Currently /hurd/console is not started (I'll add a init script for that later), but for now you have to start a getty on the console. Make sure that your /etc/inittab contains a line like this:

7:2345:respawn:/sbin/getty 38400 console

The patched initscripts package contains the runsystem.sysv file and uses the Debian alternatives system to replace the runsystem.gnu variant. It also switches to the appropriate halt and reboot utilities. If you install the package, you must use halt-hurd or reboot-hurd to halt or reboot the system. Running halt or reboot is not harmful though, it just doesn't work.

So here I am. I've been coding for the Hurd for four weeks now, and suddenly I'm the guy who wrote umount. My point is, if you ever wanted to work on an operating system, you might want to consider working on the Hurd. It the Debian/Hurd port is very decent and (according to #debian-hurds "Debian GNU/Hurd Doomsday-o-meter") 78.44% of all Debian packages are available, so you've got a familiar environment. Also, if your solution is right and your code is reasonably clean, it is not that hard to get your changes accepted. The list of open issues is long and might contain something that interests you. You could be the one implementing a read-ahead solution to massively improve Hurds overall performance.

Also since Hurd is just a Hird of Unix replacing daemons running on-top the GNU Mach microkernel, most of the functionality lies within userspace processes. This makes the whole system very extensible and quite easy to work with. For an overview over the Hurd, see this page.

Next week I'll focus on improving my mtab prototype, improving the sysvinit patches and upstreaming the ones that are ready. As always, in case I run out of stuff to do, I'll pick something from my list of issues, most likely the network related issues.

In completely unrelated news, my Debian/Linux workstation died with a kernel panic while I was writing this blog entry. It's a good thing Emacs honors "save early, save often". M-x insert-monolithic-kernel-rant-here.

This is it for this week, thanks for your attention :)

This week was all about getting mount into shape, addind umount and hacking up a prototype for the mtab issue. With all my patches to mount the boot log looks like this:

INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/hd0s1 is mounted.
/dev/hd0s1: clean, 44101/181056 files, 283413/723200 blocks
done.
Cleaning up temporary files... /tmp.
/etc/rcS.d/S06mtab.sh: 48: /etc/rcS.d/S06mtab.sh: cannot open /proc/mounts: No such file
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Mounting local filesystems...done.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 7
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 42107 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.


GNU 0.3 (debian) (console)

login:

As you can see, mountall.sh works fine now (it's the line about mounting local filesystems). This is a list of issues I tackled this week:

• Hurds mount lacks some command line flags:
  • -O, --test-opts
  • -t, --types: semantic difference to what Linux' and FreeBSDs mount does.
• Hurds mount -t auto does not work.
  • needed for fstab entries with fstype=auto
  • needed for d-i
• The hurd package does not provide an umount helper, needed for d-i.
  • also needed for umountfs, umountnfs.sh
  • required arguments -f, -v, -r, -d
• Patch series for all the issues above posted: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00031.html
• Hurds swapon/swapoff lacks some command line flags:
  • -v, --verbose: set if VERBOSE=yes is given in /etc/default/rcS.
  • Patch posted: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00055.html

Note that even though I've implemented umount, it doesn't actually work yet. That is because umount needs a mtab file or /proc/mounts to do anything, because you can either invoke umount with the mount point or the device and umount needs an mtab style file to figure out whether the argument is referring to the former or the latter.

So the next big issue is the mtab issue. There is some information about that in the Hurd wiki, it even is a gsoc project idea of its own: http://www.gnu.org/software/hurd/community/gsoc/project_ideas/mtab.html

But I need this now, this cannot wait for another gsoc. The implementation outlined by that page consists roughly of:

• In every translator, keep a list of translators (active and passive) being bound to nodes of the translator. Think of this as keeping a list of child-translators.
• Add a RPC procedure to query this list.
• Creating a mtab-translator that recursively queries translators starting from a given translator and collects all the information required for an mtab file.

If this hypothetical mtab translator is bound to say /proc/mounts this would solve all our problems:

% settrans -ca /proc/mounts /hurd/mtab /

As this involves adding a RPC procedure and keeping a list of passive and active translators, this involves some design decisions. As discussed in #hurd I hacked up a prototype and post it for review: http://lists.gnu.org/archive/html/bug-hurd/2013-07/msg00022.html

As soon as I get the necessary feedback on this issue, I will get the implementation up to speed. Until then, I will pick another item from my status list, most probably the ifupdown issue.

Speaking of feedback, all of my patches I sent last week were merged at the appropriate locations. That was really quick, many thanks for that :)

While working on the mount issues, I frequently looked at the implementation of Linux' and FreeBSD's mount to see how --types is handled there. I must admit that I was not to keen about the idea of writing c again, but I was positively surprised by the code I've seen from Hurd so far. As Hurd is the operating system of the GNU system, its code is free to use any GNU extension to libc and gcc including nice stuff like nested functions. If you look at the different mount implementations, you'll find the Hurd version to be the most modern and clean one imho:

Linux

http://sources.debian.net/src/util-linux/2.20.1-5.4/mount/mount.c

FreeBSD

http://sources.debian.net/src/freebsd-utils/9.0%2Bds1-11/sbin/mount/mount.c

Hurd

http://sources.debian.net/src/hurd/20130620-1/utils/mount.c

See you next week :)

tl; dr version: shutdown works, Debian GNU/Hurd booting using sysvinit:

Loading GNU Mach ...
Loading the Hurd ...
GNU Mach 1.3.99-486
AT386 boot: physical memory map from 0x0 to 0x9f400
AT386 boot: physical memory map from 0x100000 to 0x1fffe000
AT386 boot: physical memory from 0x0 to 0x1fffe000
Enabling FXSR
pcibios_init : BIOS32 Service Directory structure at 0xfcff0
pcibios_init : BIOS32 Service Directory entry at 0xfc7ba
pcibios_init : PCI BIOS revision 2.10 entry at 0xfc78c
Probing PCI hardware.
ide: Intel 82371 PIIX3 (dual FIFO) DMA Bus Mastering IDE
    Controller on PCI bus 0 function 9
ide: BM-DMA feature is not enabled (BIOS), enabling
    ide0: BM-DMA at 0xc100-0xc107
    ide1: BM-DMA at 0xc108-0xc10f
hd0: got CHS=762/128/63 CTL=c8 from BIOS
hd0: QEMU HARDDISK, 3001MB w/256kB Cache, CHS=762/128/63, DMA
hd2: QEMU DVD-ROM, ATAPI CDROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
Floppy drive(s): fd0 is 1.44M
FDC 0 is a S82078B
probing scsi 5/22: Adapteco1542FInvalidiaddresseforishpnt.with 1542.
Invalid address for shpnt with 1542.
probing scsi 18/22:AWesternxDigitalAWD-7000nFailed9initialization of WD-7000 SCSI card!
probing eata on/1f0EATA0:Daddress 0x1f04in3use, skipping probe.
probing eata on 170EATA0: address 0x170 in use, skipping probe.
probing scsi 21/22: Iomega7parport ZIP drive ppa: Version 1.42
ppa: Probing port 03bc
ppa: Probing port 0378
ppa:         SPP port present
ppa:         PS/2 bidirectional port present
ppa: Probing port 0278

done
scsi : 0 hosts.
scsi : detected total.
Partition check (DOS partitions):
 hd0: hd0s1 hd0s2 < hd0s5 >
lpr0: at atbus0, port = 378, spl = 6, pic = 7.
2omultibootxmodulesd/execl$(exec-task=task-create)ne=${kernel-command-line} --host-priv-port=${host-port} --device-master-port=${device-port} --exec-server-task=${exec-task} -T typed ${root} $(task-create) $(task-resume)               task loaded: ext2fs --readonly --multiboot-command-line=root=device:hd0s1 console=com0 --host-priv-port=1 --device-master-port=2 --exec-server-task=3 -T typed device:hd0s1
task loaded: exec /hurd/exec

start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/hd0s1 is mounted.
/dev/hd0s1: clean, 43930/181056 files, 281248/723200 blocks
done.
Cleaning up temporary files... /tmp.
/etc/rcS.d/S06mtab.sh: 48: /etc/rcS.d/S06mtab.sh: cannot open /proc/mounts: No such file
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Mounting local filesystems...mount: invalid option -- 'O'
Try `mount --help' or `mount --usage' for more information.
failed.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 6
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 38544 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.


GNU 0.3 (debian) (console)

login:

This is the result of:

• A patch by Pino Toscano that adds the -e and --ifexists flags to Hurds swapon: http://lists.gnu.org/archive/html/bug-hurd/2012-01/msg00060.html
• Pino Toscanos runsystem.sysv that starts sysvinits init (see below).
• A patch series to allow setting options of procfs translators at runtime (the hurdish equivalent of "remounting"): http://lists.gnu.org/archive/html/bug-hurd/2013-06/msg00119.html
• A trivial patch to tmpfs, also for updating the options at runtime: http://lists.gnu.org/archive/html/bug-hurd/2013-06/msg00126.html
• A patch series adding --fake and --no-mtab to our mount utility, fixing -oremount with one parameter (e. g. mount -oremount,ro /) and finally fixing a subtle bug in an internal look-up function: http://lists.gnu.org/archive/html/bug-hurd/2013-06/msg00135.html
• A minor tweak to checkroot.sh.
• A minor tweak to mountall.sh. Turns out that there are two init processes at this point, pidof -s takes care of that. Not sure why this does not happen on Linux, maybe it even does? Or it is related to us using inits undocumented -i flag to make it ignore the fact that it isn't pid 1. Needs further investigation.
• Disabling killprocs and sendsigs. See below.
• A minor tweak to mount-functions.sh. Same as last week though.
• Using the same inittab as Debian/Linux uses (we seem to ship our own, though I must admit I'm not sure it comes from, dpkg doesn't know about this file on my installations).
• Using the reboot and halt utilities from the sysvinit package instead of our own.

This yields a surprisingly functional Debian/Hurd system with sysvinit. There are still lot's of loose ends, I populated my status page with them.

I spent quite some time in #hurd on freenode and Pino Toscano offered to send me some old notes of his about the sysvinit issue. This turned out to be most helpful as his notes were about plugging sysvinits init into the boot process. This was the next logical thing to do for me, so first thing Monday morning I just put his runsystem.sysv file with some very minor tweaks (my version) into my overlay and had a working system in front of my eyes. That certainly was a nice way to start a week of work :), so many thanks again Pino!

For your convenience I've created an overlay with all necessary changes and recompiled binaries. Be careful though, you might want to backup /sbin/reboot first or you won't be able to reboot your system! Of course my tool does that for you, and I've managed to create a reasonably small (253 mb) image thanks to the zerofree tool, the process of compacting images is implemented in hurdtest as the compact subcommand. I also added sysvinit specific tests. So if you want to try hurdtest, download the image and unxz it, my tool expects the image to be in the current working directory and be named image.qcow2. Download and untar an overlay and execute:

% hurdtest -logfile log -sysvinit overlay path/to/second-week-overlay

If you specify -logfile you can execute tail -f path/to/log to see what the VM is printing to the console in real time. The sysvinit specific tests demonstrate that indeed sysvinit is managing the system and that e. g. init 0 or shutdown -r now works.

There are at least three major problems that I will need to tackle for this project:

• /bin/init is not running as pid 1.
• kill(-1, SIGTERM) kills essential processes.
• There is no /proc/mounts.

I think the last issue is what I'm going to look at next week. There's information about this in the Hurd wiki and I've spoken with Richard Braun about it and he drilled me to think about the hurdish way to solve this. I think I've got a pretty good picture of how to implement this in my head and am eager to give it a go. This will involve adding a function to Hurds filesystem interface that allows one to query an active translator about his mount options and "source" and about all active and passive translators that are attached to one of its nodes.

This is where the fun actually begins. This is changing an integral part of an operating system, but this is okay since it is just another program written in c, operating in userspace. But hey I know that, that's just c and if something goes wrong (and it always does) I fire up gdb and have a look. And mind you, I'm running all this as an unprivileged user, no need to jeopardize the entire system for that.

Besides implementing /proc/mounts I will also add -O, --test-opts to mount, and -v, --verbose to swapo{n,ff}.

PS: My blog is still not properly registered to planet.d.o. I asked planet@ about that, but got no response. Any help? I never imagined blogging being so stressful, I care about losing potential readers :/

Hi Planet Debian :)

my name is Justus Winter, I'm a computer science student at the University of Hamburg. I'm participating in the GSOC 2013, my task is to make Debian/Hurd systems boot using the same initialization scripts Debian/Linux uses.

I spent the community bonding period with some gsoc related administrative tasks, updating my development machine and getting reacquainted with the Hurd system. It was nice to see how far Debian/Hurd has come since my last "serious" encounter with it.

I've setup a continuous build solution on one of my Hurd VMs and prepared a package repository. I'll post details once I have some packages to share.

I built the hurd, sysvinit and ifupdown packages and dived into the interesting world of literate programming. I'm going to write about that when I start looking into the ifupdown related issues.

But most importantly I started writing a tool to automate all the tasks I would do during my work that could possibly be automated. As I'm working on the boot process, it makes sense to use some kind of virtualization software so I can work in the comfort of my usual development environment. My virtualization solution of choice is qemu and I actually wrote a qemu based black-box test solution for a former employer once, so I already was familiar with qemu and wrapping my own code around it. Without further ado, here it is:

https://bitbucket.org/teythoon/hurdtest

Provided that you have golang installed, you can compile it easily:

teythoon@europa /tmp % export GOPATH=`pwd`
teythoon@europa /tmp % go get bitbucket.org/teythoon/hurdtest
teythoon@europa /tmp % bin/hurdtest
Usage: hurdtest [options] <command> [<arguments...>]

Currently it has the following subcommands:

test

Starts a VM, logs in as root, tests the interaction with the VM, verifies that certain VFS locations are writable, tests the network connectivity, the DNS resolution and installs and runs the hello package.

update

Executes aptitude update && aptitude full-upgrade, reboots the system and runs the test subcommand. If successful, it commits the changes to the virtual disk and creates a snapshot.

overlay

Takes as argument the path to a directory that is copied to the VM, reboots the machine and runs the test subcommand.

The update command is nice because it allows me to keep the VM up to date while ensuring that the image remains usable, with respect to what the test command considers a usable state of a Debian system. I'd really like to get some suggestions on how to test the "health" of a Debian system. You can look at how the tests are implemented in test.go, it's quite easy to add more and more interesting tests, and you don't even have to clean up after them or worry about any side-effects as the changes are not committed to the virtual disk.

This is hurdtest doing an update:

teythoon@europa ~/repos/hurd/hurd-tests % bin/hurdtest update
2013/06/22 19:59:45 starting kvm [-drives file=image.qcow2,cache=writeback,snapshot=on -serial unix:/tmp/285813369/serial -monitor unix:/tmp/285813369/monitor -nographic -m 512]
2013/06/22 19:59:45 Waiting for grub...
2013/06/22 19:59:46 Waiting for login prompt...
2013/06/22 20:00:05 Updating system
2013/06/22 20:02:52 Get: 1 http://www.local unstable InRelease [8279 B]
Get: 2 http://http.debian.net sid InRelease [205 kB]
Get: 3 http://ftp.debian-ports.org unreleased InRelease [23.9 kB]
[...]
Setting up perl (5.14.2-21) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Reading task descriptions...
2013/06/22 20:02:52 Update successful. Snapshotting system...
2013/06/22 20:02:53 Rebooting system...
2013/06/22 20:02:54 Waiting for grub...
2013/06/22 20:02:59 Waiting for login prompt...
2013/06/22 20:03:16 Running main.testTrue...
2013/06/22 20:03:16 Running main.testFalse...
2013/06/22 20:03:17 Running main.testClean...
[...]
2013/06/22 20:03:18 Running main.testNetwork...
2013/06/22 20:03:18 Running main.testResolver...
2013/06/22 20:03:20 Running main.testAptitudeInstall...
2013/06/22 20:03:46 Test successful. Reverting system...
2013/06/22 20:03:48 Halting system...
2013/06/22 20:03:53 Committing changes...
2013/06/22 20:03:57 Creating snapshot...
2013/06/22 20:04:04 All done.
bin/hurdtest update  157.31s user 69.71s system 87% cpu 4:18.94 total

The overlay command implements my work flow. I can edit scripts or drop hacked up executables onto my VM and see the results without any manual intervention. There's the -interactive flag that does this in a loop, avoiding the VM startup time using qemu snapshots. With this I can apply an overlay and see the results of the test suite in about a minute on my system.

From early experiments I knew that simply running /etc/rcS.d/S* prevented the system from booting correctly, so I figured my first task should be to figure out what went wrong there. Using my tool (and developing it in that process) and many iterations later I came up with a surprisingly small diff of scripts in the initscripts and utilities shipped with the hurd package. For your convenience here is a tarball that you can extract to the root of your Debian/Hurd installation to reproduce my results (of course I invite you to use my tool for that purpose):

% bin/hurdtest overlay test
2013/06/22 21:35:31 starting kvm [-drive file=image.qcow2,cache=writeback,snapshot=on -serial unix:/tmp/345188134/serial -monitor unix:/tmp/345188134/monitor -nographic -m 512]
2013/06/22 21:35:31 Waiting for grub...
2013/06/22 21:35:31 Waiting for login prompt...
2013/06/22 21:35:48 Creating tarball...
./
./etc/
./etc/hurd/
./etc/hurd/rc
./etc/init.d/
./etc/init.d/checkroot.sh
./etc/init.d/mountall.sh
./runme
./lib/
./lib/init/
./lib/init/mount-functions.sh
./sbin/
./sbin/swapon
./bin/
./bin/mount
2013/06/22 21:35:48 Extracting overlay...
2013/06/22 21:35:49 Rebooting system...
2013/06/22 21:35:49 Waiting for grub...
2013/06/22 21:35:55 Writing log to /tmp/bootlog-802006344
2013/06/22 21:35:55 Waiting for login prompt...
2013/06/22 21:36:20 Booting successful.
2013/06/22 21:36:20 Running main.testTrue...
2013/06/22 21:36:21 Running main.testFalse...
2013/06/22 21:36:21 Running main.testClean...
2013/06/22 21:36:21 Running anonymous function...
2013/06/22 21:36:21 Running anonymous function...
2013/06/22 21:36:21 Running anonymous function...
2013/06/22 21:36:21 Running anonymous function...
2013/06/22 21:36:21 Running anonymous function...
2013/06/22 21:36:22 Running anonymous function...
2013/06/22 21:36:22 Running main.testUname...
2013/06/22 21:36:22 Running main.testNetwork...
2013/06/22 21:36:22 Running main.testResolver...
2013/06/22 21:36:23 Running main.testAptitudeInstall...
bin/hurdtest overlay ../hurd-tests-old/test  44.20s user 20.01s system 82% cpu 1:17.94 total

And here is the boot log:

Loading GNU Mach ...
Loading the Hurd ...
GNU Mach 1.3.99-486
AT386 boot: physical memory map from 0x0 to 0x9f400
AT386 boot: physical memory map from 0x100000 to 0x1fffe000
AT386 boot: physical memory from 0x0 to 0x1fffe000
Enabling FXSR
pcibios_init : BIOS32 Service Directory structure at 0xfcff0
pcibios_init : BIOS32 Service Directory entry at 0xfc7ba
pcibios_init : PCI BIOS revision 2.10 entry at 0xfc78c
Probing PCI hardware.
ide: Intel 82371 PIIX3 (dual FIFO) DMA Bus Mastering IDE
    Controller on PCI bus 0 function 9
ide: BM-DMA feature is not enabled (BIOS), enabling
    ide0: BM-DMA at 0xc100-0xc107
    ide1: BM-DMA at 0xc108-0xc10f
hd0: got CHS=762/128/63 CTL=c8 from BIOS
hd0: QEMU HARDDISK, 3001MB w/256kB Cache, CHS=762/128/63, DMA
hd2: QEMU DVD-ROM, ATAPI CDROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
Floppy drive(s): fd0 is 1.44M
FDC 0 is a S82078B
probing scsi 0/22: advansys probing scsi 1/22: BusLogic probing scsi 2/22: UltraStor 14F/34F rev. 4.33.00  probing scsi 3/22: UltraStor 14F/24F/34F probing scsi 4/22: Adaptec 152x SCSI driver; $Revision: 1.1 $ probing scsi 5/22: Adaptec 1542 Invalid address for shpnt with 1542.
Invalid address for shpnt with 1542.
probing scsi 6/22: Adaptec 174x (EISA) probing scsi 7/22:  probing scsi 8/22: Always IN2000 probing scsi 9/22:  probing scsi 10/22:  probing scsi 11/22:  probing scsi 12/22: Trantor T128/T128F/T228 probing scsi 13/22: DTC 3180/3280  probing scsi 14/22: Tekram DC390(T) V1.11 Feb-05-1997 probing scsi 15/22:  probing scsi 16/22: ncr53c8xx - revision 2.5f.1 probing scsi 17/22: EATA (Extended Attachment) PIO driver probing scsi 18/22: Western Digital WD-7000 Failed initialization of WD-7000 SCSI card!
probing scsi 19/22: EATA/DMA 2.0x rev. 4.33.00  probing eata on 1f0EATA0: address 0x1f0 in use, skipping probe.
probing eata on 1c88probing eata on 2c88probing eata on 3c88probing eata on 4c88probing eata on 5c88probing eata on 6c88probing eata on 7c88probing eata on 8c88probing eata on 9c88probing eata on ac88probing eata on bc88probing eata on cc88probing eata on dc88probing eata on ec88probing eata on fc88probing eata on 170EATA0: address 0x170 in use, skipping probe.
probing eata on 230probing eata on 330probing scsi 20/22: AM53C974 probing scsi 21/22: Iomega parport ZIP drive ppa: Version 1.42
ppa: Probing port 03bc
ppa: Probing port 0378
ppa:     SPP port present
ppa:     PS/2 bidirectional port present
ppa: Probing port 0278

done
scsi : 0 hosts.
scsi : detected total.
Partition check (DOS partitions):
 hd0: hd0s1 hd0s2 < hd0s5 >
lpr0: at atbus0, port = 378, spl = 6, pic = 7.
module 0: ext2fs --readonly --multiboot-command-line=${kernel-command-line} --host-priv-port=${host-port} --device-master-port=${device-port} --exec-server-task=${exec-task} -T typed ${root} $(task-create) $(task-resume) module 1: exec /hurd/exec $(exec-task=task-create)                                                                                                                                                                           2 multiboot modules                                                                                                                                                                                                                    task loaded: ext2fs --readonly --multiboot-command-line=root=device:hd0s1 console=com0 --host-priv-port=1 --device-master-port=2 --exec-server-task=3 -T typed device:hd0s1
task loaded: exec /hurd/exec

start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
Using makefile-style concurrent boot in runlevel S.
ERROR: could not open /proc/stat: No such file or directory
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/hd0s1 is mounted.
/dev/hd0s1: clean, 44328/181056 files, 288266/723200 blocks
done.
Cleaning up temporary files... /tmp.
mount: /run: Unknown device or filesystem
mount: /run/lock: Unknown device or filesystem
mount: /proc: Unknown device or filesystem
mount: /run/shm: Unknown device or filesystem
/etc/rcS.d/S06mtab.sh: 48: /etc/rcS.d/S06mtab.sh: cannot open /proc/mounts: No such file
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Mounting local filesystems...mount: invalid option -- 'O'
Try `mount --help' or `mount --usage' for more information.
failed.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 5
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 36884 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.


GNU 0.3 (debian) (console)

login:

I know it isn't that pretty, but it yields a working Debian/Hurd system that is more like a Debian/Linux system. For example /dev/shm and /run{,/*} are connected to a tmpfs translator. I think this is a rather encouraging first result. I added my findings to the list of open issues.

Another nice thing about hurdtest is that it is not even Hurd specific (only a tiny bit GNU Mach specific). I'm going to make it work nicely with Debian/Linux and Debian/kFreeBSD, since this is a very handy tool for anyone hacking on the Debian boot process (and with some changes for any OS). If anyone tries it out, I'd love to get feedback and patches.

The tool requires the guest to be configured to use the first serial console for grub, the kernel and the userspace. The password for the user root must be empty. The guest has to have a working network connection setup at boot time. I will provide a suitable image as soon as I devise a good way to zero-fill the free space on the virtual disk. Is there any better way than to fill the disk with a huge file containing zeros? Without doing that the image is too large even compressed with xz.

So this was my first week working for Debian. Next week I will focus on getting patches for the hurd and initscripts package in shape, I need to add some functionality to the mount utility shipped with the hurd package. It provides a mount-like experience on the Hurd, on which binding a translator (e. g. /hurd/ext2fs) to a filesystem node is an operation that does not involve the kernel (GNU Mach) at all, and is in fact a operation that any unprivileged user may perform. That is one of the nice features the Hurd has, it empowers its users.

Hi, my name is Justus Winter, I'm a computer science student at the University of Hamburg.

So it has come to this. I created a blog. I'm so 2.0. Well, since almost everyone else left this internet thing and uses this shiny new social network thing, that's probably okay.

The real reason is that I'm participating in the GSOC 2013, my task is to make Debian/Hurd systems boot using the same initialization scripts Debian/Linux uses.

Stay tuned for updates on my efforts. For the time being, you can look at my application and my wiki like status page.