af613a331e6839c6513c3e366abcb67af0fe8386
max
  Wed Sep 9 06:47:14 2026 -0700
UniProt otto: get the monthly update running again and make a stalled run visible

The monthly UniProt job had produced nothing since January 2025. The tracks
served release 2024_06 while the download sitting on disk was at 2026_02, on
every assembly the job builds.

Cause: uniprotToTab appended a personal conda site-packages directory to
sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A
venv's python is only a symlink to the system one, so when hgwdev moved to
python 3.9 the compiled lxml in there stopped loading and every run died at the
parse step. Removed both. The system python3 has lxml from python3-lxml and the
two are upgraded together, so there is nothing left here to go stale. Verified
by parsing real 2026_02 records under python 3.9 with lxml 5.4.

Why nobody noticed for nineteen months:

- doUpdate.sh read $? after an intervening echo, so it captured the echo's exit
code and mailed "Big Uniprot update OK" every month while the job was dying.
It now reads the real exit code, says FAILED, prints the tail of the log and
exits nonzero. A month with no new UniProt release stays silent, which is the
normal otto behaviour, so silence again means "nothing to do".
- The logs were overwritten on every run, so a failure left no trace on disk.
doUpdate.sh now appends one line per run to runLog.txt, which is never
truncated, and keeps a failing log as lastFail.log.
- version.txt in each bigBed directory was rewritten on every run even when the
release string was identical. That is the file the trackDb dataVersion setting
shows, and its date is what people check to decide whether a pipeline is still
alive, so a stalled track could look freshly updated. It is now written only
when the release actually changes.

Also, so this cannot come back:

- doUniprot checks that uniprotToTab can start before the download, instead of
finding out 35 minutes later.
- pylint on hgwdev is itself pinned to pythons that no longer exist, so
"make install" aborted on its first line and could not be used. Replaced with
a syntax check that needs nothing but python3; pylint stays best-effort.
- uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from
/hive/data/outside/otto/uniprot without being in the makefile's copy list.
The tree copy of uniprotToTab was still python 2 from 2021. All are now
listed and in sync, and "make diff" reports drift.
- Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh
and the pslMap -inType/-mapType flags.

refs #38300

diff --git src/hg/utils/otto/uniprot/makefile src/hg/utils/otto/uniprot/makefile
index 14c5bfa998a..6f3cfbba44f 100644
--- src/hg/utils/otto/uniprot/makefile
+++ src/hg/utils/otto/uniprot/makefile
@@ -1,4 +1,29 @@
 PREFIX=/hive/data/outside/otto/uniprot/
+PARSER=../../../../utils/uniprotToTab
+
+# Everything doUniprot calls at run time has to be listed here, otherwise the copy that
+# cron runs is not under version control. uniprotToTab, pslProtCnv and
+# trackDb.template.txt were all missing from this list for years, and the copy of
+# uniprotToTab in ${PREFIX} drifted away from the tree until it stopped working (#38300).
+FILES=*.sh *.as doUniprot mapUniprot_doBlast pslProtCnv trackDb.template.txt README.txt ${PARSER}
+
+# A plain syntax check, using nothing but the python that will run the scripts. This used
+# to be "pylint -E", but every pylint on hgwdev is itself pinned to a python that no longer
+# exists (/cluster/software/bin/pylint wants a gone python3, ~/.local/bin/pylint wants
+# python2), so "make install" aborted on its first line and nobody could install anything.
+SYNTAXCHECK=python3 -c 'import ast,sys; [ast.parse(open(f).read(), f) for f in sys.argv[1:]]'
+
 install:
-	pylint -E doUniprot
-	rsync -avpu *.sh *.as doUniprot makeUniProtPsl.sh mapUniprot_doBlast ${PREFIX} # never overwrite newer files
+	${SYNTAXCHECK} doUniprot ${PARSER}
+	-pylint -E doUniprot ${PARSER} # best effort, do not stop the install if pylint is broken
+	rsync -avpu ${FILES} ${PREFIX} # never overwrite newer files
+
+# rsync -u above refuses to overwrite a file that is newer in ${PREFIX} than in the tree,
+# so a hand edit of the live copy wins silently and forever. Run this to see what has
+# drifted apart.
+diff:
+	@for f in ${FILES}; do \
+	    for g in $$f; do \
+	        cmp -s $$g ${PREFIX}`basename $$g` || echo "DIFFERS: $$g"; \
+	    done; \
+	done