My first week and some promising though hacky results
Posted: | More posts about gsoc debian hurd |
Hi Planet Debian :)
my name is Justus Winter, I'm a computer science student at the University of Hamburg. I'm participating in the GSOC 2013, my task is to make Debian/Hurd systems boot using the same initialization scripts Debian/Linux uses.
I spent the community bonding period with some gsoc related administrative tasks, updating my development machine and getting reacquainted with the Hurd system. It was nice to see how far Debian/Hurd has come since my last "serious" encounter with it.
I've setup a continuous build solution on one of my Hurd VMs and prepared a package repository. I'll post details once I have some packages to share.
I built the hurd, sysvinit and ifupdown packages and dived into the interesting world of literate programming. I'm going to write about that when I start looking into the ifupdown related issues.
But most importantly I started writing a tool to automate all the tasks I would do during my work that could possibly be automated. As I'm working on the boot process, it makes sense to use some kind of virtualization software so I can work in the comfort of my usual development environment. My virtualization solution of choice is qemu and I actually wrote a qemu based black-box test solution for a former employer once, so I already was familiar with qemu and wrapping my own code around it. Without further ado, here it is:
https://bitbucket.org/teythoon/hurdtest
Provided that you have golang installed, you can compile it easily:
teythoon@europa /tmp % export GOPATH=`pwd` teythoon@europa /tmp % go get bitbucket.org/teythoon/hurdtest teythoon@europa /tmp % bin/hurdtest Usage: hurdtest [options] <command> [<arguments...>]
Currently it has the following subcommands:
- test
- Starts a VM, logs in as root, tests the interaction with the VM, verifies that certain VFS locations are writable, tests the network connectivity, the DNS resolution and installs and runs the hello package.
- update
- Executes aptitude update && aptitude full-upgrade, reboots the system and runs the test subcommand. If successful, it commits the changes to the virtual disk and creates a snapshot.
- overlay
- Takes as argument the path to a directory that is copied to the VM, reboots the machine and runs the test subcommand.
The update command is nice because it allows me to keep the VM up to date while ensuring that the image remains usable, with respect to what the test command considers a usable state of a Debian system. I'd really like to get some suggestions on how to test the "health" of a Debian system. You can look at how the tests are implemented in test.go, it's quite easy to add more and more interesting tests, and you don't even have to clean up after them or worry about any side-effects as the changes are not committed to the virtual disk.
This is hurdtest doing an update:
teythoon@europa ~/repos/hurd/hurd-tests % bin/hurdtest update 2013/06/22 19:59:45 starting kvm [-drives file=image.qcow2,cache=writeback,snapshot=on -serial unix:/tmp/285813369/serial -monitor unix:/tmp/285813369/monitor -nographic -m 512] 2013/06/22 19:59:45 Waiting for grub... 2013/06/22 19:59:46 Waiting for login prompt... 2013/06/22 20:00:05 Updating system 2013/06/22 20:02:52 Get: 1 http://www.local unstable InRelease [8279 B] Get: 2 http://http.debian.net sid InRelease [205 kB] Get: 3 http://ftp.debian-ports.org unreleased InRelease [23.9 kB] [...] Setting up perl (5.14.2-21) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Reading task descriptions... 2013/06/22 20:02:52 Update successful. Snapshotting system... 2013/06/22 20:02:53 Rebooting system... 2013/06/22 20:02:54 Waiting for grub... 2013/06/22 20:02:59 Waiting for login prompt... 2013/06/22 20:03:16 Running main.testTrue... 2013/06/22 20:03:16 Running main.testFalse... 2013/06/22 20:03:17 Running main.testClean... [...] 2013/06/22 20:03:18 Running main.testNetwork... 2013/06/22 20:03:18 Running main.testResolver... 2013/06/22 20:03:20 Running main.testAptitudeInstall... 2013/06/22 20:03:46 Test successful. Reverting system... 2013/06/22 20:03:48 Halting system... 2013/06/22 20:03:53 Committing changes... 2013/06/22 20:03:57 Creating snapshot... 2013/06/22 20:04:04 All done. bin/hurdtest update 157.31s user 69.71s system 87% cpu 4:18.94 total
The overlay command implements my work flow. I can edit scripts or drop hacked up executables onto my VM and see the results without any manual intervention. There's the -interactive flag that does this in a loop, avoiding the VM startup time using qemu snapshots. With this I can apply an overlay and see the results of the test suite in about a minute on my system.
From early experiments I knew that simply running /etc/rcS.d/S* prevented the system from booting correctly, so I figured my first task should be to figure out what went wrong there. Using my tool (and developing it in that process) and many iterations later I came up with a surprisingly small diff of scripts in the initscripts and utilities shipped with the hurd package. For your convenience here is a tarball that you can extract to the root of your Debian/Hurd installation to reproduce my results (of course I invite you to use my tool for that purpose):
% bin/hurdtest overlay test 2013/06/22 21:35:31 starting kvm [-drive file=image.qcow2,cache=writeback,snapshot=on -serial unix:/tmp/345188134/serial -monitor unix:/tmp/345188134/monitor -nographic -m 512] 2013/06/22 21:35:31 Waiting for grub... 2013/06/22 21:35:31 Waiting for login prompt... 2013/06/22 21:35:48 Creating tarball... ./ ./etc/ ./etc/hurd/ ./etc/hurd/rc ./etc/init.d/ ./etc/init.d/checkroot.sh ./etc/init.d/mountall.sh ./runme ./lib/ ./lib/init/ ./lib/init/mount-functions.sh ./sbin/ ./sbin/swapon ./bin/ ./bin/mount 2013/06/22 21:35:48 Extracting overlay... 2013/06/22 21:35:49 Rebooting system... 2013/06/22 21:35:49 Waiting for grub... 2013/06/22 21:35:55 Writing log to /tmp/bootlog-802006344 2013/06/22 21:35:55 Waiting for login prompt... 2013/06/22 21:36:20 Booting successful. 2013/06/22 21:36:20 Running main.testTrue... 2013/06/22 21:36:21 Running main.testFalse... 2013/06/22 21:36:21 Running main.testClean... 2013/06/22 21:36:21 Running anonymous function... 2013/06/22 21:36:21 Running anonymous function... 2013/06/22 21:36:21 Running anonymous function... 2013/06/22 21:36:21 Running anonymous function... 2013/06/22 21:36:21 Running anonymous function... 2013/06/22 21:36:22 Running anonymous function... 2013/06/22 21:36:22 Running main.testUname... 2013/06/22 21:36:22 Running main.testNetwork... 2013/06/22 21:36:22 Running main.testResolver... 2013/06/22 21:36:23 Running main.testAptitudeInstall... bin/hurdtest overlay ../hurd-tests-old/test 44.20s user 20.01s system 82% cpu 1:17.94 total
And here is the boot log:
Loading GNU Mach ...
Loading the Hurd ...
GNU Mach 1.3.99-486
AT386 boot: physical memory map from 0x0 to 0x9f400
AT386 boot: physical memory map from 0x100000 to 0x1fffe000
AT386 boot: physical memory from 0x0 to 0x1fffe000
Enabling FXSR
pcibios_init : BIOS32 Service Directory structure at 0xfcff0
pcibios_init : BIOS32 Service Directory entry at 0xfc7ba
pcibios_init : PCI BIOS revision 2.10 entry at 0xfc78c
Probing PCI hardware.
ide: Intel 82371 PIIX3 (dual FIFO) DMA Bus Mastering IDE
Controller on PCI bus 0 function 9
ide: BM-DMA feature is not enabled (BIOS), enabling
ide0: BM-DMA at 0xc100-0xc107
ide1: BM-DMA at 0xc108-0xc10f
hd0: got CHS=762/128/63 CTL=c8 from BIOS
hd0: QEMU HARDDISK, 3001MB w/256kB Cache, CHS=762/128/63, DMA
hd2: QEMU DVD-ROM, ATAPI CDROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
Floppy drive(s): fd0 is 1.44M
FDC 0 is a S82078B
probing scsi 0/22: advansys probing scsi 1/22: BusLogic probing scsi 2/22: UltraStor 14F/34F rev. 4.33.00 probing scsi 3/22: UltraStor 14F/24F/34F probing scsi 4/22: Adaptec 152x SCSI driver; $Revision: 1.1 $ probing scsi 5/22: Adaptec 1542 Invalid address for shpnt with 1542.
Invalid address for shpnt with 1542.
probing scsi 6/22: Adaptec 174x (EISA) probing scsi 7/22: probing scsi 8/22: Always IN2000 probing scsi 9/22: probing scsi 10/22: probing scsi 11/22: probing scsi 12/22: Trantor T128/T128F/T228 probing scsi 13/22: DTC 3180/3280 probing scsi 14/22: Tekram DC390(T) V1.11 Feb-05-1997 probing scsi 15/22: probing scsi 16/22: ncr53c8xx - revision 2.5f.1 probing scsi 17/22: EATA (Extended Attachment) PIO driver probing scsi 18/22: Western Digital WD-7000 Failed initialization of WD-7000 SCSI card!
probing scsi 19/22: EATA/DMA 2.0x rev. 4.33.00 probing eata on 1f0EATA0: address 0x1f0 in use, skipping probe.
probing eata on 1c88probing eata on 2c88probing eata on 3c88probing eata on 4c88probing eata on 5c88probing eata on 6c88probing eata on 7c88probing eata on 8c88probing eata on 9c88probing eata on ac88probing eata on bc88probing eata on cc88probing eata on dc88probing eata on ec88probing eata on fc88probing eata on 170EATA0: address 0x170 in use, skipping probe.
probing eata on 230probing eata on 330probing scsi 20/22: AM53C974 probing scsi 21/22: Iomega parport ZIP drive ppa: Version 1.42
ppa: Probing port 03bc
ppa: Probing port 0378
ppa: SPP port present
ppa: PS/2 bidirectional port present
ppa: Probing port 0278
done
scsi : 0 hosts.
scsi : detected total.
Partition check (DOS partitions):
hd0: hd0s1 hd0s2 < hd0s5 >
lpr0: at atbus0, port = 378, spl = 6, pic = 7.
module 0: ext2fs --readonly --multiboot-command-line=${kernel-command-line} --host-priv-port=${host-port} --device-master-port=${device-port} --exec-server-task=${exec-task} -T typed ${root} $(task-create) $(task-resume) module 1: exec /hurd/exec $(exec-task=task-create) 2 multiboot modules task loaded: ext2fs --readonly --multiboot-command-line=root=device:hd0s1 console=com0 --host-priv-port=1 --device-master-port=2 --exec-server-task=3 -T typed device:hd0s1
task loaded: exec /hurd/exec
start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
Using makefile-style concurrent boot in runlevel S.
ERROR: could not open /proc/stat: No such file or directory
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/hd0s1 is mounted.
/dev/hd0s1: clean, 44328/181056 files, 288266/723200 blocks
done.
Cleaning up temporary files... /tmp.
mount: /run: Unknown device or filesystem
mount: /run/lock: Unknown device or filesystem
mount: /proc: Unknown device or filesystem
mount: /run/shm: Unknown device or filesystem
/etc/rcS.d/S06mtab.sh: 48: /etc/rcS.d/S06mtab.sh: cannot open /proc/mounts: No such file
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Mounting local filesystems...mount: invalid option -- 'O'
Try `mount --help' or `mount --usage' for more information.
failed.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 5
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 36884 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.
GNU 0.3 (debian) (console)
login:
I know it isn't that pretty, but it yields a working Debian/Hurd system that is more like a Debian/Linux system. For example /dev/shm and /run{,/*} are connected to a tmpfs translator. I think this is a rather encouraging first result. I added my findings to the list of open issues.
Another nice thing about hurdtest is that it is not even Hurd specific (only a tiny bit GNU Mach specific). I'm going to make it work nicely with Debian/Linux and Debian/kFreeBSD, since this is a very handy tool for anyone hacking on the Debian boot process (and with some changes for any OS). If anyone tries it out, I'd love to get feedback and patches.
The tool requires the guest to be configured to use the first serial console for grub, the kernel and the userspace. The password for the user root must be empty. The guest has to have a working network connection setup at boot time. I will provide a suitable image as soon as I devise a good way to zero-fill the free space on the virtual disk. Is there any better way than to fill the disk with a huge file containing zeros? Without doing that the image is too large even compressed with xz.
So this was my first week working for Debian. Next week I will focus on getting patches for the hurd and initscripts package in shape, I need to add some functionality to the mount utility shipped with the hurd package. It provides a mount-like experience on the Hurd, on which binding a translator (e. g. /hurd/ext2fs) to a filesystem node is an operation that does not involve the kernel (GNU Mach) at all, and is in fact a operation that any unprivileged user may perform. That is one of the nice features the Hurd has, it empowers its users.