Updating packages in TeX Live

This page describes the current process of updating packages in TeX Live.

The basic premise is to take TL updates from CTAN. At present, on the machine where updates will be done, you need to have:

You will most likely also want to set up a nightly cron job to do rm -rf /your/tex/live/root/Build/tmp.*, and also svn -q update, to keep things in sync.

Usual updates

Given the above, running the script tlpkg-ctan-check --check compares a number of TL packages against a local CTAN tree and reports any that are out of date; each line starting with # in the output is one such package. If all packages are up to date, there will be no output.

Let's use poemscol as an example - it is a standard LaTeX package with no funny tricks. The basic command is ctan2tl.

  1. Run the command ctan2tl poemscol and inspect the output:
  2. Check the block of output labeled cooked. This is the package as it will be installed; it's available on the filesystem in Build/tmp.cooked/poemscol. In particular, there should be some newly-created files.
  3. Check the block of (comm) output labeled new vs. present tpm. Any changes here should make sense, e.g., a new file being added, old ones being removed. Huge masses of output probably indicate that ctan2tds (see next item) isn't set up to deal with this package. The original source as copied from CTAN is on the filesystem in Build/tmp.raw/poemscol.
  4. If there are any irregularities or questionable changes, they should be resolved before committing. The first block of output is the rather cryptic logging from ctan2tds, which does the actual CTAN-to-TDS conversion. It is not unusual for changes in a package to imply changes in ctan2tds.
  5. The last block of output is the equally cryptic logging from place, which takes the cooked tree and "places" it in the TL source hierarchy, eventually including updating the package database and doing repository adds and deletes as necessary. However, all that is a no-op in this run; so far, nothing we have done alters anything outside of the temp directories Build/tmp.*.
  6. Usually, all will look fine. Once it does, the next command to run is: ctan2tl --place (-p for short). This performs the repository commands, but does not commit anything. The most likely thing to fail here is the Subversion binary/text heuristic; those files need to be added by hand with svn add --no-auto-props, or whatever is needed.
  7. For a few packages, the first block of output will contain special instructions labeled YOU-MUST-DO. You must do them :). These involve copying executable scripts to the right place in the Build/source subtree. (Maybe one day ctan2tl will be smart enough to handle this automatically.)
  8. The last command commits the changes to the repository: svn commit -m'poemscol newversion' `cat /tmp/\`id -u\`.tlplace.dirs` where the id -u returns your uid. The reason for this complicated command is that with subversion, running a commit from a high-level directory, such as texmf-dist, is quite slow. Therefore, place writes the modified directories to the temporary file /tmp/tlplace.dirs (the environment variable TMPDIR will be used instead of /tmp if it is set). So we pass the contents of that file to svn. Naturally, if any other files were modified in the process, e.g., ctan2tds.pl, they should be included in the commit also. The version of the updated package is not always readily available; feel free to omit it from the log message.
  9. After this is done, tlpkg-ctan-check poemscol should yield no output.
  10. \relax. Then \repeat.

Abbreviations for updates

Typing ctan2tl is too long. Therefore there is a script tlpkg/bin/c2l to do the same thing. Furthermore, it also accepts platform names (and various abbreviations thereof), and calls tl-update-bindir on them. More importantly, it rearranges the output so that important diagnostics and added/removed files are visible at the beginning. This makes it possible to know whether the commit should be done without plowing through the painfully verbose full output. (E.g., if run in an Emacs compilation buffer, you won't even have to see it unless investigation is needed.) It is ugly and would need work for anyone but karl to use it, but it is there.

For commits, c2l accepts a first argument p (for place) which runs ctan2tl -p. There is no dash since then Emacs word commands will delete it without leaving the dash behind.

Furthermore, a script ~karl/bin/tlcom (sorry, but it has paths hardwired to avoid surprises) does the commit with all the relevant files and directories mentioned above.

So, in practice, the process for updating packages is: 

c2l somepkg   # and consider the output
c2l p somepkg # and svn add text/plain per item 6 above
tlcom 'somepkg (DDmonYY)'

(Anything that gets run several times a day has to be optimized to within an inch of its life, so that's what I (karl) have done. I run the first two as Emacs compile commands and then tlcom in an Emacs shell buffer. A typical update takes less than a minute.)

Another script: tlpkg/bin/c2lx. This is for importing the various LaTeX releases: with first argument 3, LaTeX3 (l3kernel et al.); with first argument e LaTeX2e (latex et al.); with first argument dev latex-dev (latex-base-dev et al.). The exact list of packages for each is at the beginning of the script. It runs ctan2l on each package and shows a summary at the end of the output.

c2lx also accepts an additional argument p to do the --place.

The reason to use this is that it is best to update the core LaTeX packages as a set; if there's an error of some kind (e.g., missing files) with one of its packages, best not to update anything.

Adding new packages to the automated checks

The tlpkg-ctan-check script used above should be updated whenever a new package is added to TL, that is, upload to CTAN. Here's the story:

  1. You can find the package on CTAN by running CTAN=/path/to/CTAN tlpkginfo --ctan-dir package, or generally searching CTAN. If the tlpkginfo script does not return the right directory, something needs to be fixed. Please look into it (or at least report it).
  2. Check the licensing. TeX Live is free software. That's free as in freedom, in case you didn't know. If it's nonfree, it should be removed from TL and a note left in ctan2tds.
  3. In particular, check that the source to the documentation (typically a pdf) is available. Often the documentation is generated from a dtx, and that is fine. But not infrequently it is written separately, and we must have sources (available under a free license) to include it in TL. If it's not there, please politely write the author - usually it is simply an oversight, not a conscious exclusion. Without sources, the processed documentation must be excluded.
  4. Optional (but much appreciated): check the TeX Catalogue entry for the package, and if there are any updates, send email with:
  5. Run ctan2tl and inspect the output, following the procedure and information above.
    1. If everything looks good, including files being installed in the correct directories:
      • add the package name to tlpkg-ctan-check.
      • call ctan2tl --place package, as above.
      • commit, as above.
    2. If anything looks wrong: hack ctan2tds.pl as necessary, which can mean anything from merely adding the package to the directory hashes at the top of the program, to a simple use of MAKEflatten, to a completely new routine that does arbitrary things. Or email the list so others can help.
  6. \relax. Then continue to \repeat.
  7. We run tlpkg-ctan-check --list-not-treated from cron. It should continue to report 0 packages not checked.

Removing a package from TeX Live

To remove a package from the TeX Live sources, use the tlprm script. It doesn't actually remove anything (or commit any changes), just tells you the actions to take.

Adding a new package on CTAN to TeX Live

If you want to add a new package that is not presently in TL, do more or less the same as above, only at the "if everything looks good" stage, also add it to one of the collections (Master/tlpkg/tlpsrc/collection-*) and the tlpkg-ctan-check script. If you don't know which collection it belongs in, ask.

Also, some packages need a special execute directive; the most common case is packages including Type1 fonts, which should have an addMap or (unlikely) addMixedMap line. See the main infrastructure documentation for all the possibilities, and/or the existing *.tlpsrc files.

Incidentally, if you come across a TeX package in TL that is not on CTAN, please encourage the authors to upload it to CTAN. Our ultimate goal is for everything in TL to come from CTAN.

Updating programs

The description above is only for the usual TeX-world packages - LaTeX styles, fonts, etc. Packages which contain executables, especially compiled executables, are updated differently. Full documentation on building TeX Live.

Mirroring CTAN

The best way to mirror CTAN is to use rsync. Here is a skeleton of the command used on tug.org (of course you'll probably want to redirect the output, etc.): 

cd /my/ctan/mirror/directory
nice /usr/local/bin/rsync -avz --delete --delete-excluded \
  --exclude='.in.*' --exclude=.htaccess \
  rsync://$CTAN_MIRROR/ \
  $CTAN_LOCAL_DIR

If the CTAN mirror is only for TL update purposes, more can be excluded: 

nice rsync -avz --delete --delete-excluded \
  --exclude='.in.*' --exclude=.htaccess \
  --exclude=/nonfree \
  --include='/obsolete/{biblio,macros/latex,fonts/wasy}' --exclude=/obsolete \
  --exclude=/systems --exclude=/tools --exclude=/usergrps --exclude=/web \
  rsync://$CTAN_MIRROR/ \
  $CTAN_LOCAL_DIR

Caveat on discerning which packages to update: although the ctan-ann list is of course a good source for discovering new updates to be propagated, it is not complete. Uploads to CTAN are commonly made without an announcement. This is one reason that tlpkg-ctan-check exists.

$Date: 2020/05/19 15:38:13 $; TeX Live;