bee41315e53da6fb5671f96302233c655906dfd5
max
  Wed Sep 9 06:56:49 2026 -0700
UniProt otto: rebuild the venv rather than relying on a system lxml

Correction to the previous commit: hgwdev has no system-wide lxml at all. The
import I tested was resolving to my own ~/.local/lib/python3.9/site-packages,
which cron never sees, because it runs the pipeline as otto.

So the environment is a virtualenv again, but a reproducible one. makeVenv.sh
deletes venv/ and rebuilds it from /usr/bin/python3, installs lxml, opens up the
permissions for otto, and then checks that lxml imports with an empty environment
so we know the venv stands on its own instead of borrowing from whoever ran it.
Built with --copies, so venv/bin/python is a real copy rather than a symlink that
would silently follow a system python upgrade while its compiled modules stayed
behind.

doUpdate.sh activates venv/ again and says to run makeVenv.sh if it is missing or
if the parser will not start. Verified: /usr/bin/python3 without per-user packages
cannot import lxml, the venv can, and after activation the parser runs and
converts real 2026_02 records.

Also shortened the README to how the pipeline is started and how it works, and
trimmed the history out of the code comments, leaving the ticket as the pointer.

refs #38300

diff --git src/hg/utils/otto/uniprot/makefile src/hg/utils/otto/uniprot/makefile
index 6f3cfbba44f..48c58aefac1 100644
--- src/hg/utils/otto/uniprot/makefile
+++ src/hg/utils/otto/uniprot/makefile
@@ -1,28 +1,24 @@
 PREFIX=/hive/data/outside/otto/uniprot/
 PARSER=../../../../utils/uniprotToTab
 
-# Everything doUniprot calls at run time has to be listed here, otherwise the copy that
-# cron runs is not under version control. uniprotToTab, pslProtCnv and
-# trackDb.template.txt were all missing from this list for years, and the copy of
-# uniprotToTab in ${PREFIX} drifted away from the tree until it stopped working (#38300).
+# Everything doUniprot calls at run time must be listed here, or the copy that cron runs
+# is not under version control. refs #38300
 FILES=*.sh *.as doUniprot mapUniprot_doBlast pslProtCnv trackDb.template.txt README.txt ${PARSER}
 
-# A plain syntax check, using nothing but the python that will run the scripts. This used
-# to be "pylint -E", but every pylint on hgwdev is itself pinned to a python that no longer
-# exists (/cluster/software/bin/pylint wants a gone python3, ~/.local/bin/pylint wants
-# python2), so "make install" aborted on its first line and nobody could install anything.
+# Not "pylint -E": every pylint on hgwdev is itself pinned to a python that no longer
+# exists, so "make install" used to abort on its first line.
 SYNTAXCHECK=python3 -c 'import ast,sys; [ast.parse(open(f).read(), f) for f in sys.argv[1:]]'
 
 install:
 	${SYNTAXCHECK} doUniprot ${PARSER}
 	-pylint -E doUniprot ${PARSER} # best effort, do not stop the install if pylint is broken
 	rsync -avpu ${FILES} ${PREFIX} # never overwrite newer files
 
 # rsync -u above refuses to overwrite a file that is newer in ${PREFIX} than in the tree,
 # so a hand edit of the live copy wins silently and forever. Run this to see what has
 # drifted apart.
 diff:
 	@for f in ${FILES}; do \
 	    for g in $$f; do \
 	        cmp -s $$g ${PREFIX}`basename $$g` || echo "DIFFERS: $$g"; \
 	    done; \