Torpage ======= Torpage was written to help me prevent downloading packages more than once. The problem I expierenced was that maintaining 10 Gentoo servers and just over 400 workstations caused stuff to be downloaded multiple times, whilst we have a local mirror of the official portage mirror, it is not always up to date, nor could we rely on all the packages being on the official mirror. Servers then fetched these packages from off-campus (I work at the University of Pretoria, department computer science) for which the department then pays per megabyte downloaded. I have a similar problem at home, although, not nearly as big. I could not work with the existing alternative solutions as these all allows for _arbitrary_ people to download _arbitrary_ files, causing me to pay for them. How it works ============ Torpage works by wrapping around some other program that performs fetching. It basically relies on being called from within portage, then reads CATEGORY, PN and PF environment variables in order to determine which package is being fetched. The script takes two parameters, the torpage:// uri and the DISTDIR variables. It then executes tcpclient with the configured torpage server and the client chat script as well as the package information. This contacts the torpage server which reads the package information from the network, and then causes the server to fetch the requested file for that package. If you configure torpage to wrap around something like wget, it will then call that program in a similar fashion that portage would. This is eseful for having the torpage server first fetch the package and then fetching the package from the server. In our case we simply mount the distfiles directory read-only nfs. How to configure it (client) ============================ You need to add the following two lines to your /etc/make.conf: FETCHCOMMAND_TORPAGE="/usr/sbin/torpage_fetch \${URI} \${DISTDIR}" RESUMECOMMAND_TORPAGE="${FETCHCOMMAND_TORPAGE}" Then you can add torpage mirrors to your /etc/portage/mirrors file, I have the following for example: local torpage:// ftp://ftp.up.ac.za/mirrors/gentoo.org/gentoo You can also use: local torpage://some.server.name If using the latter the default configuration file should be ok, I'd suggest browsing through it to double check. How to configure it (server) ============================ Edit /etc/conf.d/torpage if the defaults is not ok. By default it binds to all interfaces on port 16384. For gentoo systems no further configuration should be required. Torpage will use your portage configuration files as far as possible. You may want to browse through the configuration file though, and if you would like to have any settings different for torpage than for portage you may simply override them in /etc/torpage.conf To make torpage start up by default, do "rc-update add torpage default". On other systems (Like Debian for example) you will need to edit /etc/torpage.conf quite extensively and if you are not familiar with Gentoo - probably call in the help of someone that is. You will likely need to write your own init script based on the one provided in init.d/gentoo (check first whether one doesn't exist in the init.d directory yet). Mounting /usr/portage read-only from the clients ================================================ As alternative configuration one can mount /usr/portage read-only from the server on the clients. I do this with a reasonable amount of success, it does however require one to patch portage since portage refuses to fetch onto a read only filesystem. The patch can be found in the patch directory and was created against portage 2.0.51-r15, and can be applied by executing # patch < /path/to/portage.py-2.0.51-r15.patch from /usr/lib/portage/pym. Yes, I don't like custom patching either. You have to redo this every time after upgrading portage... After the patch has been applied you need to add "skiprocheck" to FEATURES in /etc/make.conf, as well as remove "distlocks". I'm using read-only NFS at home quite successfully, I've got the following 5 lines in my /etc/make.conf (amongst others) on the client machine: FETCHCOMMAND_TORPAGE="/usr/sbin/torpage_fetch \${URI} \${DISTDIR}" RESUMECOMMAND_TORPAGE="/usr/sbin/torpage_fetch \${URI} \${DISTDIR}" FETCHCOMMAND="/bin/false" RESUMECOMMAND="/bin/false" FEATURES="skiprocheck -distlocks" And then in /etc/portage/mirrors I've got: local torpage://pug.lan There is a small problem, namely that the kernel remembers that the file does not exist after portage tried to check for it's existence, thus you need to set up TORPAGE_{FETCH,RESUME}="ls \${DISTDIR} &>/dev/null" to work around this in /etc/torpage.conf. Cleaning out old distfiles ========================== I've seen some hacks that cleanes out distfiles (I'm thinking specifically of ogentools which a few of my friends maintain - http://tlug.up.ac.za/projects/gentoo-tools.html). The problem is that it'll nuke everything but the newest version of a file. And even on a single user system this is not desirable, never mind on a distfiles server. So I came up with this solution - perfect for a distfiles mirror, not quite so perfect for a desktop system: 1. Construct a list of all distfiles that occur in the files/digest-* files 2. For every *file* in ${DISTDIR}, if it is not in the list from (1), rm it. The distfiles_cleanup script implements this, and took about 4 minutes to run on a 24GB mirror (including being verbose accross a ssh link), and about a minute and a half on my desktop system (About 700MB of distfiles, not verbose). The script have two options: --pretend - switches into pretend mode - will print the rm commands instead of executing them. --verbose - print a message for each and every file in ${DISTDIR}, along with whether the script is keeping or removing the file. A note on the file locking implemented by portage ================================================= Portage implements a method of locking files which I cannot emulate (either a lack of know how or it cannot be done). Portage creates a file, by opening it with O_CREAT|O_RW which means it opens it for reading and writing (writing being the important one) and creates it if it does not exist. It then uses fcntl to take out a lock on this file. Needless to say, I don't like this mechanism since I cannot take out the fcntl lock from bash (at least, not afaik know). I made a compromise though that works most of the time. There are 4 possible conflicts: 1. portage has the lock, portage wants the lock 2. portage has the lock, torpage wants the lock 3. torpage has the lock, torpage wants the lock 4. torpage has the lock, portage wants the lock Number 1 doesn't concern me. The rest I need to take care off, and succeeds in all but number 4. What I do is use the dotlockfile utility to take out a lockfile of the same name as the file used by portage. This works for the cases where torpage wants to obtain the lock since torpage will only lock if the lockfile doesn't exist. In the case of number 4 though portage will see the file is there, and then take out a fcntl lock on my file, thus proceeding to download the same file torpage is already downloading. For number 4 there is possibly a fix, but I haven't tested this yet. It boils down to using the same mechanism as for read-only NFS, except don't use skiprocheck. Then add torpage://127.0.0.1 as a mirror, the first mirror. This will force portage to rely fully on torpage for fetching files without taking out any locks (allowing torpage to grab the lock). Additionally you will lose all progress information on downloads. Queries ======= If you have any queries, please direct them jaco@kroon.co.za