af613a331e6839c6513c3e366abcb67af0fe8386 max Wed Sep 9 06:47:14 2026 -0700 UniProt otto: get the monthly update running again and make a stalled run visible The monthly UniProt job had produced nothing since January 2025. The tracks served release 2024_06 while the download sitting on disk was at 2026_02, on every assembly the job builds. Cause: uniprotToTab appended a personal conda site-packages directory to sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A venv's python is only a symlink to the system one, so when hgwdev moved to python 3.9 the compiled lxml in there stopped loading and every run died at the parse step. Removed both. The system python3 has lxml from python3-lxml and the two are upgraded together, so there is nothing left here to go stale. Verified by parsing real 2026_02 records under python 3.9 with lxml 5.4. Why nobody noticed for nineteen months: - doUpdate.sh read $? after an intervening echo, so it captured the echo's exit code and mailed "Big Uniprot update OK" every month while the job was dying. It now reads the real exit code, says FAILED, prints the tail of the log and exits nonzero. A month with no new UniProt release stays silent, which is the normal otto behaviour, so silence again means "nothing to do". - The logs were overwritten on every run, so a failure left no trace on disk. doUpdate.sh now appends one line per run to runLog.txt, which is never truncated, and keeps a failing log as lastFail.log. - version.txt in each bigBed directory was rewritten on every run even when the release string was identical. That is the file the trackDb dataVersion setting shows, and its date is what people check to decide whether a pipeline is still alive, so a stalled track could look freshly updated. It is now written only when the release actually changes. Also, so this cannot come back: - doUniprot checks that uniprotToTab can start before the download, instead of finding out 35 minutes later. - pylint on hgwdev is itself pinned to pythons that no longer exist, so "make install" aborted on its first line and could not be used. Replaced with a syntax check that needs nothing but python3; pylint stays best-effort. - uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from /hive/data/outside/otto/uniprot without being in the makefile's copy list. The tree copy of uniprotToTab was still python 2 from 2021. All are now listed and in sync, and "make diff" reports drift. - Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh and the pslMap -inType/-mapType flags. refs #38300 diff --git src/hg/utils/otto/uniprot/doUpdate.sh src/hg/utils/otto/uniprot/doUpdate.sh index 04cab75876e..8b9f14220c8 100755 --- src/hg/utils/otto/uniprot/doUpdate.sh +++ src/hg/utils/otto/uniprot/doUpdate.sh @@ -1,18 +1,69 @@ #!/bin/sh -# configuration setup for the doUniprot script -cd /hive/data/outside/otto/uniprot +# configuration setup and cron wrapper for the doUniprot script + +cd /hive/data/outside/otto/uniprot || exit 1 +umask 002 + #echo WARNING: NOT DOWNLOADING #./doUniprot run --skipDownload -# activate a virtual python environment with the lxml XML parser -source venv/bin/activate -umask 002 -echo uniprot start at `date` + +# There is deliberately no virtualenv here anymore. uniprotToTab needs the lxml XML +# parser, which on hgwdev comes from the system package python3-lxml and is upgraded +# together with /usr/bin/python3. A virtualenv used to sit in venv/ instead, but its +# python was only a symlink to /usr/bin/python3: when the system python moved from 3.6 +# to 3.9 the compiled lxml in the venv stopped loading, every monthly run died at the +# parse step, and the tracks stayed on release 2024_06 for 19 months (redmine #38300). + +runLog=runLog.txt + +logRun() { + echo "`date '+%Y-%m-%d %H:%M:%S'` $*" >> $runLog +} + +# Do not spend 35 minutes downloading UniProt only to find out that the parser cannot +# start. Run it with --help, which imports lxml and then exits, and stop here if that +# fails. Invoked exactly the way doUniprot invokes it, so this tests the same python. +if ! ./uniprotToTab --help > /dev/null 2>&1; then + logRun "PREFLIGHT-FAIL uniprotToTab cannot start" + echo "UniProt update did not start: ./uniprotToTab cannot be run." + echo + echo "Almost certainly the lxml python module is missing. Check with:" + echo " python3 -c 'import lxml.etree'" + echo "and see /hive/data/outside/otto/uniprot/README.txt for how to repair it." + echo + ./uniprotToTab --help 2>&1 | tail -20 + exit 1 +fi + +logRun "START" ./doUniprot run > lastRun.log 2>&1 -echo uniprot end at `date` exitCode=$? -if [ $exitCode -eq 0 ] ; then - echo Big Uniprot update OK, exit code $exitCode -else - echo Big Uniprot update failed. Look at /hive/data/outside/otto/uniprot/lastRun.log and restart manually with: cd /hive/data/outside/otto/uniprot followed by ./doUniprot run, usually with the -p option to skip download and parsing of the gigantic XML. - echo Exit code was: $exitCode +logRun "END exit=$exitCode" + +if [ $exitCode -ne 0 ] ; then + # lastRun.log is overwritten by the next run, so keep a copy. Without one, a + # failure that nobody reads leaves no trace on disk at all. + cp -f lastRun.log lastFail.log + logRun "FAIL exit=$exitCode log=lastFail.log" + echo "Big UniProt update FAILED, exit code $exitCode" + echo + echo "Full log: /hive/data/outside/otto/uniprot/lastFail.log" + echo "Restart manually with:" + echo " cd /hive/data/outside/otto/uniprot && ./doUniprot run" + echo "usually with the -p option to skip download and parsing of the gigantic XML." + echo + echo "Last 25 lines of the log:" + tail -25 lastFail.log + exit $exitCode +fi + +if grep -q "are not newer than file in" lastRun.log ; then + # UniProt had no new release this month. This is the normal case for most months, + # so stay silent: otto crons only mail when something changed or something broke. + logRun "NOCHANGE no new UniProt release on the server" + exit 0 fi + +logRun "OK updated to `cat tab/version.txt`" +echo "Big UniProt update OK" +echo "Now serving: `cat tab/version.txt`"