Partial Debian Mirrors

The following is a description of the technique I’ve used to create partial Debian mirrors. I have damn near mastered that technique that is not immediately obvious if you don’t know the tool. Nothing there is black magic though, as you will see.

The following example will use the programs called reprepro and
germinate. Both are available on several versions of Ubuntu, Debian Lenny and Debian Sid. I’m not going to explain either in great details. They are well documented in the case of reprepro, and there is enough documentation in germinate.

Required steps

This configuration is necessary in both case.

reprepro is by far my favorite Debian repository manager. It has a steep learning curve at first but once you get past the initial configuration and the first few commands you’ll see it cannot really be made easier.

The first step is to create the directory that will contain the mirror. In that directory create a directory called conf.

In that directory, edit a file named distribution in your favorite editor At this point I usually pop-open the man page for reprepro. Write the following in that file:

Codename: etch
Architectures: amd64 source
Description: Debian Etch (required package only)
Components: main
Update: debian-etch-update

That’s all we need in that file right now. This describes what distribution this mirror will include. You can have several distributions in a mirror. Most field are self explanatory except perhaps the Updates field which will explain below.

Partial mirror, using reprepro only

The next step explain how to create a partial mirror using just reprepro. reprepro is powerful enough to select a subset of a whole package archive.

Name: debian-etch-update
Method: http://gulus.usherbrooke.ca/debian
Components: main
Architectures: amd64 source
FilterFormula: Priority (==required)

The content above is the content of your conf/updates file. It includes all the rules needed to update the distributions configured in the distributions file. The distribution etch we created in is linked to its respective update rule by the Update configuration field.

The key to partial mirroring in this case is the FilterFormula field, which selects just the Debian package in required in a very basic installation. There is also a FilterList field which can select just the set of package you list. This is the basis of the next section.

To start mirroring, go in the mirror directory and do:

reprepro -V update etch

and reprepro will start downloading packages. The -V argument activates verbose mode. After successfully creating the archive, you can reuse the same command to update your mirror.

Germinate and reprepro

germinate is an oddball program which, given a set of package, will recursively find all packages a set of package depend on.

As a simple example, if you tell germinate to germinate the dependencies from the package python, it will list libssl, libreadline and will eventually find libc6. It can also fetch build dependencies if you want everything needed to build the package you want.

We will use germinate to create a Debian repository that mirrors just the set of package the python2.5 package depends on.

germinate is a very specialised tool which sees little use outside making new Debian and Ubuntu derivative distributions. It’s very powerful but a bit confusing. I’ll give the basic recipe you need to know to accomplish our current goal. If there is enough or any interest, I give more use cases for germinate.

1. Create directories called seeds and germinate. The former will include the files we give as input to germinate. The later will include germinate‘s output.

2. In the seeds directory, create files called blacklist, required, STRUCTURE and supported. The blacklist and required will stay empty.

3. In the STRUCTURE file, put the following:

   required:
   supported:
   

4. In the required file, put the following:

    * python2.5
   

   Don’t forget the space at the start of the line.

5. You can then run germinate.

   (MIRROR=http://gulus.usherbrooke.ca/debian
     cd germinate && \
     germinate -v \
               -m $MIRROR \
               -d etch \
               -c main \
               -s seeds \
               -S file:///tmp/temp-distro \
               --no-rdepends)
   

   where you can replace /tmp/temp-distro with the base directory of your mirror.

You should see a bunch of files created in the germinate directory. The only file we are interested in is germinate/required.

germinate output is meant to be human-readable so we need a bit of parsing to extract the required information.

The following Bash magic does the trick:

for pkg in $(cat germinate/required \
  | tail -n +3 \
  | head -n -2 \
  | cut -d '|' -f 1); do\
 echo $pkg install; \
done > mirror.packages

The resulting mirror.package is now suitable to act as a filter for reprepro.

Replace the FilterFormula line in conf/updates by:

FilterList: purge ../mirror.packages

purge tells reprepro not to do anything for packages not in the list.

Then run:

reprepro -V update etch

to create your mirror.