bee41315e53da6fb5671f96302233c655906dfd5
max
  Wed Sep 9 06:56:49 2026 -0700
UniProt otto: rebuild the venv rather than relying on a system lxml

Correction to the previous commit: hgwdev has no system-wide lxml at all. The
import I tested was resolving to my own ~/.local/lib/python3.9/site-packages,
which cron never sees, because it runs the pipeline as otto.

So the environment is a virtualenv again, but a reproducible one. makeVenv.sh
deletes venv/ and rebuilds it from /usr/bin/python3, installs lxml, opens up the
permissions for otto, and then checks that lxml imports with an empty environment
so we know the venv stands on its own instead of borrowing from whoever ran it.
Built with --copies, so venv/bin/python is a real copy rather than a symlink that
would silently follow a system python upgrade while its compiled modules stayed
behind.

doUpdate.sh activates venv/ again and says to run makeVenv.sh if it is missing or
if the parser will not start. Verified: /usr/bin/python3 without per-user packages
cannot import lxml, the venv can, and after activation the parser runs and
converts real 2026_02 records.

Also shortened the README to how the pipeline is started and how it works, and
trimmed the history out of the code comments, leaving the ticket as the pointer.

refs #38300

diff --git src/utils/uniprotToTab src/utils/uniprotToTab
index 88c20757797..277acef47fb 100755
--- src/utils/uniprotToTab
+++ src/utils/uniprotToTab
@@ -1,37 +1,33 @@
 #!/usr/bin/env python3
 
 # load default python packages
 import logging, optparse, sys, glob, gzip, collections, copy, gzip, os, doctest, re
 from os.path import *
 from collections import defaultdict
 
-# lxml is not part of the python standard library. On hgwdev it comes from the
-# system package python3-lxml, which is upgraded together with /usr/bin/python3.
-# Do not put a private site-packages directory on sys.path here: this file used to
-# append one from a personal conda environment built for python 3.6, and when the
-# system python moved to 3.9 the compiled lxml in it stopped loading. Every monthly
-# UniProt update then died at this line and the tracks sat unchanged for 19 months
-# before anyone noticed (see redmine #38300).
+# lxml is not part of the python standard library and hgwdev has no system-wide copy.
+# The otto pipeline gets it from a virtualenv, see makeVenv.sh in the otto/uniprot
+# directory. Do not append a site-packages directory to sys.path here: pointing at one
+# person's conda environment is what silently broke the monthly update. refs #38300
 try:
     from lxml import etree # if this fails, comment out this line and uncomment the next one
     #import xml.etree.cElementTree as etree # if using this line, search for cElementTree in this file and comment out the other part
 except ImportError as ex:
-    raise Exception("Cannot import lxml.etree with %s: %s. On hgwdev this module comes "
-        "from the system package python3-lxml. Check with: python3 -c 'import lxml.etree'. "
-        "If the system python does not have it, install it with 'pip install --user lxml' or "
-        "build a private environment, see the README in the otto/uniprot directory." %
+    raise Exception("Cannot import lxml.etree with %s: %s. For the otto pipeline, activate "
+        "the virtualenv ('source venv/bin/activate' in the otto/uniprot directory) or "
+        "rebuild it with ./makeVenv.sh there. Otherwise: pip install lxml" %
         (sys.executable, ex))
 
 debugMode=False
 
 # --- FASTA FILES ---
 class FastaReader:
     """ a class to parse a fasta file
     Example:
         fr = FastaReader(filename)
         for (id, seq) in fr.parse():
             print id,seq """
 
     def __init__(self, fname):
         if hasattr(fname, 'read'):
             self.f = fname