lostwebsite.net blog

Adobe Air is a new software platform from Adobe which mixes JavaScript and Flash technologies to enable developers to make rich Internet applications that can run on desktop computers. It is remarkable in the world of proprietary applications in the sense that it has included Linux support early on.

See the rest at www.lostwebsite.net…

In my last post I’ve shown that some people are annoyed at the fact that they can no longer erase their root directory simply by typing rm -rf /. I’m happy that this possibility is removed. I’m not scared that means Linux is being dumbed down. There are already some safeguards in Linux and nobody is complaining about them because they guard even seasoned users to do things that are dangerous or silly.

See the rest at www.lostwebsite.net…

Original post

Did you know that rm -rf / no longer works on recent Ubuntu version? I bet you did not because this command tends to be a bit destructive.

I did not know that. I’ve never been bitten by an accidental rm -rf / but the possibility scares the hell out of me whenever I do a command in my root directory or whenever I write scripts thats erase files.

See the rest at www.lostwebsite.net…

This little Bash snippets scans the Debian dpkg database for packages whose configurations files have been modified.

This only touches configurations files which are packaged and made known to dpkg. Some package manually handle their configuration file in maintainer scripts. A good example of that is /etc/network/interfaces which is an important configuration file but not automatically handled by dpkg.

This is as useful as you want to make it. I use it to track changes I’ve done which could end up with a conflict during a package upgrade. If you want a more thorough tracking of configuration changes, use something like etckeeper.

Notice I took some care to use more Bash’isms, to optimize the script. I was able to optimize by just a few percents. The bulk of the time must spent calculating MD5 sums.

#!/bin/bash

for pkg in /var/lib/dpkg/info/*.conffiles; do    
    p1=${pkg#/var/lib/dpkg/info/}
    p=${p1%%.conffiles}
    IFS=$'n'
    for cfsum in $(dpkg-query -W -f='${Conffiles}' $p); do
        IFS=' '
        c=($(eval "echo $cfsum"))
        if [ -r ${c[0]} ]; then
            sum=${c[1]}
            cur_sum=($(eval "md5sum ${c[0]}"))
            if [ $sum != ${cur_sum[0]} ]; then
                echo "$p: ${c[0]}"
            fi
        else
            echo "Can't access file: ${c[0]}." > /dev/stderr
        fi
    done
done

This will output something like this:

$ ./check_changed
bash: /etc/bash.bashrc
console-tools: /etc/console-tools/config
libldap2: /etc/ldap/ldap.conf
ntp: /etc/ntp.conf
...

I will not go over the story of the OpenSSL fiasco again. Let’s just say that a developer inserted a particular patch in Debian OpenSSL package that he shouldn’t have. The patch was flawed in ways subtile enough that it escaped review 2 years. Long story short, Debian was left in a rather immense accumulation of fecal matter.

This was bad and made a lot of people object to the fact that software are patched in Linux distributions. It is true that software developers usually are the best person to fix the bugs in their software. What is still true but perhaps less obvious, is that most software developers are not ready or able to work in close cooperation with distribution developers so that their software are perfectly packaged without bugs and in time for a distribution release.

Distribution developers need to patch software for various, good, reasons, to ensure the best quality software distribution to their users, in a timely fashion.

Today I will examine three different kind of patch and try to justify why Debian developers, or distribution developers in general, will develope them. What is very important to note is that Debian, and probably most other distributions, wants that the patch the developers produce are sent to the original developers in the software they are packaging. In the case of the Debian distribution, this is mandated by the Debian Policy Manual, in section 4.3. That way, the patch can be eventually integrated in a future release, and the patch removed for the package for that new release.

The lesser divergence from upstream, the better. The OpenSSL fiasco is certainly a proof of that.

Plain bugfixes

It is a frequent occurance that packages are patched to fix small software bug. It is often inconvenient for a software packager to wait for a new release each time a bug is reported in the software they are responsible of. In some case, that release may come after several months (years), during which the bug will stay unfixed, which may in some case make the package unusable.

Patches are often pulled from the development repository of the software, if they can be backported to the version currently in the distribution.

Debian developers will then patch software when they can, to the best of their knowledge. It is recommended that they consult with the maintainer of the software before uploading their change to the distribution.

Patch example

http://patch-tracking.debian.net/patch/series/view/nano/2.0.7-4/ja_help_freeze.patch

In the case of the nano package, the existance of the patch is documented in the changelog of the package.

Compiler fixes

Like all other package in the distribution, compilers evolve and are eventually upgraded. Debian usually test new compilers ahead of time so they can stamp out compilation bugs early.

In case compilation fails for a package with a new version of a compiler, Debian developers will sometimes prefer to patch the program instead of asking the upstream developer to build his program with a recent or possibly unreleased version of the a compiler. They will patch the program to make it work with the new compiler and transmit the patch to the original developers for their future use.

Patch example

http://patch-tracking.debian.net/patch/series/view/qtodo/0.1.2-5/20-gcc.dpatch

This patch adapts the program to a newer than the one that was probably used to develop the package.

FHS bugfixes

The Debian distribution try to closely follow the Filesystem Hierarchy Standard (FHS). This is actually mandated in the Debian Policy Manual, section 9.1 so, if a program puts a file in a wrong place, it is viewed as a important bug.

Fortunately, most developers will follow the FHS, by habit or consciously. This simplify the packaging process, most program won’t need to be changed to be compliant. Sometimes though, some developers consider that it is not improper that a global configuration file is put outside /etc, or that image files can be put in /var.

Patch example

http://patch-tracking.debian.net/patch/series/view/trac/0.10.3-1etch4/01_use_global_config

This patch tells the Trac web application to look for its configuration file in /etc instead of /usr/share.

That will be enough for today. I think I’m already beyond the “tl;dr” level for most people. I will try to find some more example of types of patch done by packagers in another blog post.

This is an annoyance I found got around to solve.  In Konsole, with a basic X configuration, when you type AltGr Space, you get a non-breakable space. It might happen in other things X terminal/shell combo, or the both of them might not matter at all.

If you are an heavy shell user, you might hit the AltGr Space combo by accident. This wouldn’t be bad if the shell made it somehow obvious that what you entered in a non-breakable space. The fact is that it doesn’t and it’s then totally impossible to distinguish a regular space with a non-breakable space after it was typed in the shell.

cd $HOME
zsh: no such file or directory: cd /home/fdgonthier

After typing such a command quickly, I was often left wondering what the hell was wrong with my command, since, as far as I could see, it was syntaxically correct.

The problem is that since there is a non-breakable space between cd and $HOME, Zsh wants to execute the whole command as a single command. Since I didn’t know I could even type non-breakable space in X, I checked what the hell the shell was doing by using strace. strace is my favorite diagnosis tool for any kind of problems in Linux. Programs don’t lie when they are spied with strace. The problem is obvious:

execve("cd\302\240/home/fdgonthier", ["cd\302\240/home/fdgonthier"], [/* 30 vars */]) = -1 ENOENT (No such file or directory)

It took me a while but I’ve found that \302\240 is the Unicode sequence for non-breakable space. Once I have found that, I Google search lead me to Launchpad bug 218637.

On this page, you see that you can disable non-breakable space using a xkb option in xorg.conf. Simply add the following in the InputDevice section related to your keyboard.

...
Option "XkbOptions"    "nbsp:none"
...

or use

setxkbmap -option "nbsp:none"

in a console, your favorite initialization file, or elsewhere.

The following is a description of the technique I’ve used to create partial Debian mirrors. I have damn near mastered that technique that is not immediately obvious if you don’t know the tool. Nothing there is black magic though, as you will see.

The following example will use the programs called reprepro and
germinate. Both are available on several versions of Ubuntu, Debian Lenny and Debian Sid. I’m not going to explain either in great details. They are well documented in the case of reprepro, and there is enough documentation in germinate.

Required steps

This configuration is necessary in both case.

reprepro is by far my favorite Debian repository manager. It has a steep learning curve at first but once you get past the initial configuration and the first few commands you’ll see it cannot really be made easier.

The first step is to create the directory that will contain the mirror. In that directory create a directory called conf.

In that directory, edit a file named distribution in your favorite editor At this point I usually pop-open the man page for reprepro. Write the following in that file:

Codename: etch
Architectures: amd64 source
Description: Debian Etch (required package only)
Components: main
Update: debian-etch-update

That’s all we need in that file right now. This describes what distribution this mirror will include. You can have several distributions in a mirror. Most field are self explanatory except perhaps the Updates field which will explain below.

Partial mirror, using reprepro only

The next step explain how to create a partial mirror using just reprepro. reprepro is powerful enough to select a subset of a whole package archive.

Name: debian-etch-update
Method: http://gulus.usherbrooke.ca/debian
Components: main
Architectures: amd64 source
FilterFormula: Priority (==required)

The content above is the content of your conf/updates file. It includes all the rules needed to update the distributions configured in the distributions file. The distribution etch we created in is linked to its respective update rule by the Update configuration field.

The key to partial mirroring in this case is the FilterFormula field, which selects just the Debian package in required in a very basic installation. There is also a FilterList field which can select just the set of package you list. This is the basis of the next section.

To start mirroring, go in the mirror directory and do:

reprepro -V update etch

and reprepro will start downloading packages. The -V argument activates verbose mode. After successfully creating the archive, you can reuse the same command to update your mirror.

Germinate and reprepro

germinate is an oddball program which, given a set of package, will recursively find all packages a set of package depend on.

As a simple example, if you tell germinate to germinate the dependencies from the package python, it will list libssl, libreadline and will eventually find libc6. It can also fetch build dependencies if you want everything needed to build the package you want.

We will use germinate to create a Debian repository that mirrors just the set of package the python2.5 package depends on.

germinate is a very specialised tool which sees little use outside making new Debian and Ubuntu derivative distributions. It’s very powerful but a bit confusing. I’ll give the basic recipe you need to know to accomplish our current goal. If there is enough or any interest, I give more use cases for germinate.

1. Create directories called seeds and germinate. The former will include the files we give as input to germinate. The later will include germinate‘s output.

2. In the seeds directory, create files called blacklist, required, STRUCTURE and supported. The blacklist and required will stay empty.

3. In the STRUCTURE file, put the following:

   required:
   supported:
   

4. In the required file, put the following:

    * python2.5
   

   Don’t forget the space at the start of the line.

5. You can then run germinate.

   (MIRROR=http://gulus.usherbrooke.ca/debian
     cd germinate && \
     germinate -v \
               -m $MIRROR \
               -d etch \
               -c main \
               -s seeds \
               -S file:///tmp/temp-distro \
               --no-rdepends)
   

   where you can replace /tmp/temp-distro with the base directory of your mirror.

You should see a bunch of files created in the germinate directory. The only file we are interested in is germinate/required.

germinate output is meant to be human-readable so we need a bit of parsing to extract the required information.

The following Bash magic does the trick:

for pkg in $(cat germinate/required \
  | tail -n +3 \
  | head -n -2 \
  | cut -d '|' -f 1); do\
 echo $pkg install; \
done > mirror.packages

The resulting mirror.package is now suitable to act as a filter for reprepro.

Replace the FilterFormula line in conf/updates by:

FilterList: purge ../mirror.packages

purge tells reprepro not to do anything for packages not in the list.

Then run:

reprepro -V update etch

to create your mirror.

Did you know you could pretty much install Ubuntu on Debian Sid? Now you do!

Well, yes and no.

Yes because I installed a set of Ubuntu packages on a (de)bootstrapped Sid Debian distribution.

No because the debootstrap actually installed Ubuntu packages, and the set of packages Ubuntu and Debian install at their base are actually quite similar.

If you diff the 2 debootstraps scripts (Ubuntu Gutsy vs. Debian Sid), you’ll see very little difference between the 2. To make that work you have to alias the Debian Sid distribution to Gutsy otherwise deboostrap will fail saying it can’t find the source distribution on the mirror.

On the CD I’m making, I’m using a custom distribution name, made off Ubuntu packages. With a quick hack to deboostrap, you can make debootstrap install Sid using Gutsy packages!