Gentoo

So, what's going on that does not really fit into one of my diary-like posts?

• It has been getting cold and wet during the last weeks. And I missed some ultimate frisbee training sessions because of not having the right clothes. I should really go shopping. Our mixed team from ultimate frisbee is the german champion for the second time in a row. Congratulations to all of them. They are going to some tournament in London on one of the next weekends.


• After trying to learn a lot for my probability theory exam I finally started to get productive. Yay!


• I get two computer magazines sent to my house: IX and Linux Magazin. Usually I have enoughe time to read the most interesting articles before the next edition comes out, but I bought a CT some weeks ago and now I am completely out of schedule.


• My computer starts to do funny things. I get a message about some accessibility feature not being present at every boot. If I am at the computer at around half past 10 (or half past 9?) in the morning, it hangs and I have to reboot it. The clock applet in Gnome does not work anymore. The computer hangs from time to time, but only when doing lots of stuff (compiling, email, webbrowser and probably more, all at the same time). Since some time ago another RAM bar died, I definitely have not enough RAM. Less than a GB at the moment. So I'm thinking about getting rid of this complicated Gentoo stuff and installing Fedora. But my machine is some years old and I want a new one. I only need the core parts, motherboard, CPU, fan, RAM, graphics adapter (if it is not onboard). So I'm thinking about buying some 4-core CPU and 4 GB of RAM and some cheap board with integrated graphics adapter (I can upgrade later). But I'll do at least one exam before.


• I read this news item about a guy (or actually more than one) building a $150 balloon with camera and GPS and taking pictures of space (the area around earth). I want to do that, too.


• I' thinking of getting rid of categories in this blog. They only make sense when they are used consistently, and I don't do that.


• I started using F-Spot for organizing my pictures. I also started hating pictures without embedded timestamps. Scans from old photographes, pictures from mobile phone cameras, pictures edited with a tool that removes that stuff. New tools, new restrictions. That's life.

I just posted this to the gentoo-dev ML, feel free to read and post comments.


Good morning,

I want to talk about improved binary package support for Gentoo. About
1-2 months ago there already was a discussion about this on gentoo-soc@
and on bugzilla [1]. If I remember correctly, there were no devs
involved in the discussion, so I thought I'll post my thoughts here.

I know, that Gentoo is a source-based distribution or meta-distribution,
and I don't want to make Gentoo another Fedora or Ubuntu, but I think
there are some things we can learn from them.

The current situation:

Binary packages are (usually) stored
in /usr/portage/packages/$category/$package-$version.tbz2. The package
consists of the "real binary package" and the metadata (combined using
xpak or whatever).

Problems I see with this:

1) If a binary package is built because it needs to be linked against a
new library, because the USE-flags change or because the ebuild changes
without a revision bump, the "old" binary package is overwritten. This
also means that there is no support to store multiple packages with
different USE-flags without, well, using different directories.
2) To find out which USE-flags a package is built with, one needs to
download the package and look at the metadata. Today I discoveres a file
called "Packages" which looks like a metadata cache, but I did not find
more information about it (only tried "man portage").

So, how can we address this?

First we should do something about 2), I think: I want to propost the
following scheme:

Binary packages are stored in
$arch/$description/$category/$package/$package-$version-$ev-$use-$bv.tbz2.

$arch: This is x86, ppc or whatever you put into ACCEPT_KEYWORDS minus
the '~'. It does not make sense to make a distinction here.
$description: Something like pentium3, core2quad, G4, or whatever.
Pentium3-uclibc, Pentium3-solaris-prefix are also possible.
$category, $package and $version should be clear.
$ev: The "ebuild version". See below.
$bv: The "binary version". See below.
$use: The USE-flags. See below.

About ebuild version, USE-flags and binary version:

I would like to encode the USE-flags into the filename. This enables us
to have binary packages of the same version built with different
USE-flags in the same repository. Some wanted to have this in the
directory, some say it is ok to have it in the xpak only and some prefer
the "Packages"-like file.

I think, USE-flags can be set per package and therefore should be stored
per package, not per $description or whatever. Having it only in the
xpak allows no distinction between multiple binary packages, same
version, differen USE-flags and the same is true for the Packages file.
This would also be created, downloaded all the time and so on. Therefore
I think the cleanest solution is having USE-flags in the filename.

There are different methods to store it there.

a) A checksum (of the USE-flags, the USE-flag string, the ebuild and the
USE-flag string, whatever).
b) List the enabled USE-flags in the filename, use a) if the string gets
too long.
c) Use a packed binary vector.

I don't like a), because it is not easily reversible. You could always
download the Packages file or the binary package and look into the xpak
metadata, but that's too much effort. b) also has the problems i
mentioned for a). Also, you'd need some system to distinguish ebuilds
with the same version but different USE-flags. You also need that for
c), so b) has no advantages ofer c) in my eyes.

For c) I think of the following: Sort the USE-flags in some defined way
(ASCII code, whatever) and make a vector with a 1 for every enabled
USE-flag and a 0 for every disabled USE-flag. Compress that vector: If
you use HEX code, you need 1 character for every 4 bits, but it should
be possible to find 64 different characters, then you need 1 character
for every 6 bits. PHP has 106 USE.flags, that would make a USE-string
with about 18-27 characters. Packages with lots of USE-expand stuff like
languages would need more, but not too much, I think.

Problems: The string might get long, you get big problems with USE-flag
renames, USE-flag additions or removals. That's where the ebuild version
is needed. Or not. We have 3 possibilities:

a) Change policy: USE-flag changes in an ebuild need a version bump.
b) Use a checksum of the ebuild.
c) Use the version given by the version control system.

The problem with a) is, that is a change in policy and probably hard to
do. Increasing the revision for a (trivial) change leads to a lot of
unnecessary rebuilds for users. It also means, that USE-flag changes in
eclasses are difficult, the eclass should probably copied over to a new
name with version and only ebuilds with a new version (revision) are
allowed to use it.

The problem with b) is, that it is not ordered. You don't know, which is
the newest version. If you have an ebuild with a version where there is
no binary package for, it gets difficult/ugly.

c) also has problems: When using cvs, there are versions easily
available. The same is true for svn, but lots of distributed version
control systems like git use checksums as versions. Welcome back to b).
Another thing is, how do we get to the versions? Will they be in the
header forever, since they make signing ebuilds or the manifest much
more complicated (multiple commits necessary)? But, well, since metadata
is generated and provided by "the tree", it should be not too hard to ad
a unique ebuild version there (in the case of checksums, use an integer,
increase whenever the checksum changed or something). It just might make
using overlays a bit more difficult.

The last thing to be discribed is the binary version. Lots of people
talk about dependencies to other binary packages when they talk about
binary packages for Gentoo, but that gets quite difficult (and, in my
opinion, ugly). We mostly need to provide a "consistent set" of
packages, which means, if A depends on B, B changes and therefore breaks
A, we need to provide an updated version of A. And we can do that with
simply increasing the binary version, since the package manager knows
then, that this package needs updating, too.

How to create binary packages?

Create some build server (or build server infrastructure). The most
important thing is a script or something that provides the
functionality. One enters a make.conf, /etc/portage dir, path to the
profile, description and whatever else is needed and the system starty
building. Then you can create a second set of data and start building
and the system puts the binary packages in the same directory and
discovers what needs to be built and what not (because apache needs to
be built only once if its USE-flags are the same for the different
configuration sets).

But there are thousands of packages and millions of USE-flag
combinations!

Seriously, who cares? The goal of this project (as it exists in my head)
is not to provide everything. It is to provide the most used packages.
If you need parrot, compile it yourself. If you need netbeans, compile
it yourself. We have @system, gnome, kde and anothe hand full of
packages, which will change over time. I'm, really lookign forward to
the data collected by the statistics project (GSoC).

The same is true for USE-flags: We might provide gnome, kde, both, a
server profile and whatever we decide to provide, but not everthing.
Again, statistics will help.

Same with CFLAGS. Probably no -O3, no -ffast-math, no -break-my-code or
whatever. Probably x86 with 32 and 64 bit for the beginning, later maybe
more.

So, the really really cool thing is, that if you are some company,
university, institution or freak, with lots of (similar) Gentoo boxes,
you can set up a build server and even share the binary packages, if you
want. Same level of security as non-official overlays, but in the
university of FooBar in Jamaica uses it, there should not be too many
security problems.

Thanks for reading, please discuss, I probably forgot lots of stuff, but
I can tell it later in the discussion.

Philipp




[1] https://bugs.gentoo.org/show_bug.cgi?id=150031

I've been thinking about this for a long long time. It basically comes down to this:

Iv you're running gentoo, you usually have the portage tree synced to your system. It will ocupy around 300 MB on ext 3 with smallest block size (or a little less, I have some overlays there, too). That's not that much space, but there are lots and lots of small text files. Compressed, they are very small, around 50 MB I guess. They are synced using rsync, which uses up some traffic on the net, I guess.

Some years ago I found an init script (never used it, though), which does teh following: It takes the portage tree, creates a squashfs filesystem out of it and mounts that. If you do a sync, the changes get written to a union-fs mounted directory and on shutdown the new squashfs image gets created. The advantage of this is, that it uses much less space, you don't need an extra filesystem with small block size (or twice as much or more space if not) and it is said to be much faster. On todays hardware, reading less data from the disk and uncompressing them is faster than reading more data from the disk. Well, the disadvantage is that you need some init script that does funny things, and lots of things can go wrong with that procedure, I guess. Not sure, it was too much of a hack for me to use it.

Now, how could we do it right? How could we solve the problem? There are already portage snapshots on the gentoo mirrors, the format is either .tar.bz2 or .tar.lzma. One nice thing would be to be able to mount those. You download them, mount them, everything is fine. At the moment the snapshots get recreated once a day. rsync mirrors are synced twice an hour, I think. This difference is unimportant for "bleeding edge" arguments, but if something wrong gets packed into the tarball, one has to wait a day to get it fixed. That's rather long. Well, lots of those "wrong stuff in the tree" could be fixed with another source controll system like svn or git, but that's a completely different topic.

Another way to solve it would be for gentoo to provide squashfs images on the mirrors. It is possile to mount those, but apart from that one has the same problems as above.

What I thought of definitely needs the most development, be it design work or programming work, but I kind of like the idea. It consists of the following:

We use a squashfs like filesystem. The layout is as follows: A filesystem image has the form AB, where A is all the metadata and B is all the files. To create B all the files are sorted by path and then compressed individually and concatenated. So you get one big image with lots and lots of files in it. To extract one, you need some blocks of that image and uncompress them. How they are compressed is unimportant. Part A has to contain the following: All filenames, all metadata (we don't need permissions, I think, maybe mtime), all the information about where to find the files (offset, length) and maybe the compression method used, in case we want to allow different compression methods. Probably there is more information needed for A, but I don't really know since I'm no filesystem expert. Strings like all the filenames should be compressed, of course.

Now, what happens? Well, we can obviously write some driver to mount that image. That would be nice. To sync a newer image from gentoo we can create bindiffs. They should be very efficient, since lots and lots of files stay the same and therefore lots and lots of blocks in our image don't change. A server that created bindiffs on demand could be created, or some differntly grained diffs could be uploaded. Half-hourly diffs, 6-hourly diffs, daily diffs, weekly diffs, monthly diffs. Then a client has to download some small diffs and can create the most recent image our of the image it has.

How does that sound? Did I miss anything?

I just had something like the most stupid problem for a long long while. My mothers new PC did not work, which means she could not login. The logfile said:

Jan 9 17:08:10 kurt gdm[3149]: pam_unix(gdm:auth): authentication failure; logname= uid=0 euid=0 tty=:0 ruser= rhost= user=lotte
Jan 9 17:08:13 kurt gdm[3149]: WARNING: Couldn't authenticate user


Well, googling did not give me anything, so I tried rebuilding stuff and upgrading some packages. Nothing worked. In the end I decided to have a look at the Gentoo Bugzilla, if there's anything about it and I found a bug aboug gdm and wring locales. Well... before, the gdm screen was in german, then it somehow switched to english. I don't know, how and why. But now every y is a z, and so on... so I simply changed my mothers password that it contains all the "wrong" characters and now she can login again. Stupid hack, but since I don't have too much time at the moment...

I already mentioned this article.

While trying to fix my Routerboard, I somehow managed to make my computer unusable. I have two network interfaces, and they were not sorted in the right way but in a random way. Not good, if you use static network configuration. Therefore I could not get dhcp and everything I needed to reinstall my routerboard running. My boot partition got destroyed in the progress of fixing the network problem. The fix was not to compile the driver for the second network card. Not good.

BTW: What is the right way of installing grob into the MBR of 2 harddrives and have the grub files on an md device?

Well, next thing was, that my mothers computer broke. It has problems from time to time (beeing six years old or something like that) and they can usually be "fixed" by opening the case and wiggling on every card, RAM module and cable. Well, this time it did not work. And I don't want to find the problem, since that's the most ugly job one can have. Well, the "break" is like this: If you switch on the computer, it beeps long, one time, it cannot be reached on the network and the monitor stays in standbye. I tried booting without harddrives, graphics card,... and since then I get the long beep every few seconds when the computer is switched on. Nce, isn't it?

Well, I was thinking about getting an EEE Box B202. It would be great for my mother, since she should have no problem with the slower 2,5" harddrive and the slower CPU. All she needs is Email and a Webbrowser. She uses Thunderbird and Firefox (although I want to switch her to Evolution one day, because I use it) which both use a lot of RAM, but 2 GB are really enough. She also wants to save the environment and stuff, so a computer that uses a maximum of 24 Watts should be ok. If Amazon had it on stock, I would already have ordered it.

My sister has an Acer laptop and problems from time to time. If it's software, I have to fix it. If it's hardware, she gives it back to the shop. That happened about 10 days ago, and since then my sister does not have a computer. First she used my mothers, but now that's broken. Quite bad, actually. I hope, she will have it back quite soon (and listen to me... you don't buy a cheap Acer if you want quality).

I found a nice blog post about removing stuff from xorg.conf. I could remove almost everything, I only have my graphics adapter section and the screen section left. Really nice. First I was looking into removing those two, too, but it's not worth the effort (if it's possible at all). Good thing: Less stuff in a strange config file. Bad thing: Where will I have to search for the reasons, when my xorg configuration breaks? But to be honest, that did not happen for a long long time. I hope, hal will not make it worse.

Some weeks ago I finally decided to order parts for a router for my home network and some days ago they arrived. Now I own a nice little Routerboard 433 (by Mikrotik, a MiniPCI WLAN adapter called Wistron Weben CM9, some antennas, a case, a power supply and what else I need for it. I already fould the time to play around with it for one day: I got it assembled, it boots the internal RouterOS without a problem, I set it to boot via bootp (and dhcp), set up bootp (and dhcp) (the servers) on my workstation and had a small success: It gets an IP address and loads the kernel image I tell it to load. Then I see the line "jumping to kernel code" and nothing from there. Nothing on the serial console, no beeps, no ping returns, nothing. Really really sad. But I knew it might get hard.

So today I tried looking into it some more. I went away from OpenWRT and tried to build a cross toolchain using Gentoo's crossdev. Well, I failed.

I hope I will get this to work some time soon. And I hope that one day I will not be too lazy to post pictures.

Another thing I'm kind of into at the moment is beeging a second mentor for a google summer of code project. The project is part of gentoo, and I have done some work there when I was a summer of code student last year.

GNAP is short for Gentoo Network Appliance and the project aims at enabling people to build routers or access points or whatever out of their old i386 compatible hardware in an easy way. The last release is quite old and does not build without a lot of modifications and hacks and a lot of help. Last year I wanted to enable cross compiling with it, but I did not succeed. I had to fight too many problems on my way to reaching the goal and then I found out how it could be done, but I did not have enough time left. What I did were some small improvements on the gnap code (3 bash scripts, some perl helper scripts for doing gnap releases), writing a Makefile for building stages using catalyst and then building i386-uclibc, i486-glibc (i386 did not work with recent glibc), ppc-uclibc, ppc-glibc, ppc64-uclibc and ppc64-glibc. But nobody really took my work anywhere, since then. There was another student last year, his provject was called mikroconf. It aimed at providing a Cisco IOS like command line interface for routers built with gnap. It's not included anywhere, yet, I think. But it all looked very promising.

So this years gnap project is called GNAP Love. It does not aim at providing one big feature (which nobody will use) but at doing all the small things nobody else does. Create a new working gnap release, include the work from last year, more script cleanups, maybe some small improvements. We also want to provide support for at least one embedded board. We decided to go the Routerboard way for now and our student ordered one ppc based board. I ordered the smaller version of it. Or I'm about to order it. I think I will blog about it.

At the moment we are stuck with some problems: Uclibc stages do not build with the hardened profile, and stuff like that. But I hope we will get that fixed, soon and I will be able to post some more interesting stuff here.

You can find more information on our development page. We are using google code for the moment because the gentoo infrastructure cannot provide us with a svn repository.