lostwebsite.net blog

Annex to the Lost Website

Partial Debian Mirrors

with 6 comments

The following is a description of the technique I’ve used to create partial Debian mirrors. I have damn near mastered that technique that is not immediately obvious if you don’t know the tool. Nothing there is black magic though, as you will see.

The following example will use the programs called reprepro and
germinate. Both are available on several versions of Ubuntu, Debian Lenny and Debian Sid. I’m not going to explain either in great details. They are well documented in the case of reprepro, and there is enough documentation in germinate.

Required steps

This configuration is necessary in both case.

reprepro is by far my favorite Debian repository manager. It has a steep learning curve at first but once you get past the initial configuration and the first few commands you’ll see it cannot really be made easier.

The first step is to create the directory that will contain the mirror. In that directory create a directory called conf.

In that directory, edit a file named distribution in your favorite editor At this point I usually pop-open the man page for reprepro. Write the following in that file:

Codename: etch
Architectures: amd64 source
Description: Debian Etch (required package only)
Components: main
Update: debian-etch-update

That’s all we need in that file right now. This describes what distribution this mirror will include. You can have several distributions in a mirror. Most field are self explanatory except perhaps the Updates field which will explain below.

Partial mirror, using reprepro only

The next step explain how to create a partial mirror using just reprepro. reprepro is powerful enough to select a subset of a whole package archive.

Name: debian-etch-update
Method: http://gulus.usherbrooke.ca/debian
Components: main
Architectures: amd64 source
FilterFormula: Priority (==required)

The content above is the content of your conf/updates file. It includes all the rules needed to update the distributions configured in the distributions file. The distribution etch we created in is linked to its respective update rule by the Update configuration field.

The key to partial mirroring in this case is the FilterFormula field, which selects just the Debian package in required in a very basic installation. There is also a FilterList field which can select just the set of package you list. This is the basis of the next section.

To start mirroring, go in the mirror directory and do:

reprepro -V update etch

and reprepro will start downloading packages. The -V argument activates verbose mode. After successfully creating the archive, you can reuse the same command to update your mirror.

Germinate and reprepro

germinate is an oddball program which, given a set of package, will recursively find all packages a set of package depend on.

As a simple example, if you tell germinate to germinate the dependencies from the package python, it will list libssl, libreadline and will eventually find libc6. It can also fetch build dependencies if you want everything needed to build the package you want.

We will use germinate to create a Debian repository that mirrors just the set of package the python2.5 package depends on.

germinate is a very specialised tool which sees little use outside making new Debian and Ubuntu derivative distributions. It’s very powerful but a bit confusing. I’ll give the basic recipe you need to know to accomplish our current goal. If there is enough or any interest, I give more use cases for germinate.

  1. Create directories called seeds and germinate. The former will include the files we give as input to germinate. The later will include germinate‘s output.

  2. In the seeds directory, create files called blacklist, required, STRUCTURE and supported. The blacklist and required will stay empty.

  3. In the STRUCTURE file, put the following:

    required:
    supported:
    

  4. In the required file, put the following:

     * python2.5
    

    Don’t forget the space at the start of the line.

  5. You can then run germinate.

    (MIRROR=http://gulus.usherbrooke.ca/debian
      cd germinate && \
      germinate -v \
                -m $MIRROR \
                -d etch \
                -c main \
                -s seeds \
                -S file:///tmp/temp-distro \
                --no-rdepends)
    

    where you can replace /tmp/temp-distro with the base directory of your mirror.

You should see a bunch of files created in the germinate directory. The only file we are interested in is germinate/required.

germinate output is meant to be human-readable so we need a bit of parsing to extract the required information.

The following Bash magic does the trick:

for pkg in $(cat germinate/required \
  | tail -n +3 \
  | head -n -2 \
  | cut -d '|' -f 1); do\
 echo $pkg install; \
done > mirror.packages

The resulting mirror.package is now suitable to act as a filter for reprepro.

Replace the FilterFormula line in conf/updates by:

FilterList: purge ../mirror.packages

purge tells reprepro not to do anything for packages not in the list.

Then run:

reprepro -V update etch

to create your mirror.

Written by fdgonthier

October 21, 2008 at 10:40 pm

6 Responses

Subscribe to comments with RSS.

  1. Very interesting. I’ve tried to follow the instructions but got stuck with the following error:

    ? Unknown dependency libnewt0.52 by cdebconf-newt-udeb

    Which is where it stops. Do I need to blacklist it or something?

    Alex

    January 19, 2009 at 12:28 pm

  2. Please give me more informations. I don’t know after what command you obtain that error.

    If its a Germinate error, it can probably be ignored. Germinate spews a whole lot of warnings about missing dependencies because the archive is not perfect.

    fdgonthier

    January 19, 2009 at 12:40 pm

  3. [recommented as a direct reply to you since I’m not exactly sure about WordPress]

    Please give me more informations. I don’t know after what command you obtain that error.

    If its a Germinate error, it can probably be ignored. Germinate spews a whole lot of warnings about missing dependencies because the archive is not perfect.

    fdgonthier

    February 13, 2009 at 6:44 pm

  4. Hi. Sorry it was my mistake. The germinate error was around udebs and looking into it further it appears that germinate doesn’t handle udebs very well – nothing seems to. Your instructions worked very well for me. Thanks!

    Alex

    February 17, 2009 at 11:38 am

  5. […] go in the mirror directory and do: reprepro -V update etch The complete procedure is found here. ciao, […]

    Reprepro errors and setting up

    September 24, 2011 at 3:46 am

  6. To use FilterList to include only specific packages, add a line ‘FilterList: hold ‘ to the updates file. Now in the file specified by add a list of packages, one per line, in the same format as output by ‘dpkg –get-selections’ – in other words e.g:
    python-dnspython install
    python-xmpp install
    python-jabberbot install

    Now you can use ‘reprepro update’

    I also found that in some installations you need to add a line ‘VerifyRelease: blindtrust’ to the updates file.

    (Haven’t tried germinate yet…I’m hoping I won’t need to go that far)

    Shalom Crown

    October 25, 2012 at 6:22 am


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: