bee41315e53da6fb5671f96302233c655906dfd5
max
  Wed Sep 9 06:56:49 2026 -0700
UniProt otto: rebuild the venv rather than relying on a system lxml

Correction to the previous commit: hgwdev has no system-wide lxml at all. The
import I tested was resolving to my own ~/.local/lib/python3.9/site-packages,
which cron never sees, because it runs the pipeline as otto.

So the environment is a virtualenv again, but a reproducible one. makeVenv.sh
deletes venv/ and rebuilds it from /usr/bin/python3, installs lxml, opens up the
permissions for otto, and then checks that lxml imports with an empty environment
so we know the venv stands on its own instead of borrowing from whoever ran it.
Built with --copies, so venv/bin/python is a real copy rather than a symlink that
would silently follow a system python upgrade while its compiled modules stayed
behind.

doUpdate.sh activates venv/ again and says to run makeVenv.sh if it is missing or
if the parser will not start. Verified: /usr/bin/python3 without per-user packages
cannot import lxml, the venv can, and after activation the parser runs and
converts real 2026_02 records.

Also shortened the README to how the pipeline is started and how it works, and
trimmed the history out of the code comments, leaving the ticket as the pointer.

refs #38300

diff --git src/hg/utils/otto/uniprot/makeVenv.sh src/hg/utils/otto/uniprot/makeVenv.sh
new file mode 100755
index 00000000000..96848c8e84b
--- /dev/null
+++ src/hg/utils/otto/uniprot/makeVenv.sh
@@ -0,0 +1,47 @@
+#!/bin/sh
+# Build the python environment that uniprotToTab needs, in venv/ next to this script.
+# Run this whenever the preflight check in doUpdate.sh says the parser cannot start,
+# e.g. after hgwdev gets a new python. Takes about a minute.
+#
+# Run it as a user who can write to the otto directory, not as otto.
+#   cd /hive/data/outside/otto/uniprot && ./makeVenv.sh
+
+set -e
+
+cd `dirname $0`
+venvDir=venv
+
+# uniprotToTab needs lxml, which is not in the python standard library and is not
+# installed system-wide on hgwdev. A per-user "pip install --user" is not enough either:
+# cron runs this pipeline as otto, which does not see anyone else's ~/.local.
+modules="lxml"
+
+# --copies gives the venv its own copy of the python binary instead of a symlink to
+# /usr/bin/python3. A symlink silently follows a system python upgrade while the
+# compiled modules in the venv stay behind, which is how this environment broke before.
+if [ -d $venvDir ] ; then
+    echo "Removing the old $venvDir"
+    rm -rf $venvDir
+fi
+
+echo "Building $venvDir with `/usr/bin/python3 -V 2>&1`"
+/usr/bin/python3 -m venv --copies $venvDir
+
+$venvDir/bin/pip install --quiet --upgrade pip
+$venvDir/bin/pip install --quiet $modules
+
+# cron runs as otto, so everything has to be readable and executable by everyone
+chmod -R a+rX $venvDir
+
+echo
+echo "Installed:"
+$venvDir/bin/pip list 2>/dev/null | grep -i -E "lxml|^Package|^---"
+echo
+# Check with an empty environment, so we know the venv stands on its own and is not
+# quietly borrowing a module from whoever happens to run it.
+if env -i $venvDir/bin/python -c "import lxml.etree; print('lxml', lxml.etree.__version__, 'from', lxml.__file__)" ; then
+    echo "$venvDir is ready"
+else
+    echo "$venvDir is broken, lxml does not import" >&2
+    exit 1
+fi