bee41315e53da6fb5671f96302233c655906dfd5 max Wed Sep 9 06:56:49 2026 -0700 UniProt otto: rebuild the venv rather than relying on a system lxml Correction to the previous commit: hgwdev has no system-wide lxml at all. The import I tested was resolving to my own ~/.local/lib/python3.9/site-packages, which cron never sees, because it runs the pipeline as otto. So the environment is a virtualenv again, but a reproducible one. makeVenv.sh deletes venv/ and rebuilds it from /usr/bin/python3, installs lxml, opens up the permissions for otto, and then checks that lxml imports with an empty environment so we know the venv stands on its own instead of borrowing from whoever ran it. Built with --copies, so venv/bin/python is a real copy rather than a symlink that would silently follow a system python upgrade while its compiled modules stayed behind. doUpdate.sh activates venv/ again and says to run makeVenv.sh if it is missing or if the parser will not start. Verified: /usr/bin/python3 without per-user packages cannot import lxml, the venv can, and after activation the parser runs and converts real 2026_02 records. Also shortened the README to how the pipeline is started and how it works, and trimmed the history out of the code comments, leaving the ticket as the pointer. refs #38300 diff --git src/hg/utils/otto/uniprot/README.txt src/hg/utils/otto/uniprot/README.txt index a51fac40043..1a89dc56e93 100644 --- src/hg/utils/otto/uniprot/README.txt +++ src/hg/utils/otto/uniprot/README.txt @@ -1,68 +1,66 @@ UniProt mapping pipeline, Max 2016/2017, updates in 2021 -This is the automated pipeline that updates the UniProt tracks from UniProt.org. -UniProt puts out a new release every month, see http://www.uniprot.org/news/ +Updates the UniProt tracks from UniProt.org, which puts out a new release every +month. See http://www.uniprot.org/news/ -Two copies of these scripts exist and they are not the same file: - the tree copy, src/hg/utils/otto/uniprot in the kent repo, is the one to edit - the live copy, /hive/data/outside/otto/uniprot, is the one cron runs -Edit the tree copy, commit, then "make install" to push it out. "make diff" lists -files that have drifted apart, which is worth checking now and then: the rsync in -"make install" uses -u and so refuses to overwrite a live file that is newer than -the tree, which means a hand edit of the live copy wins silently and forever. +Two copies of these scripts exist. Edit the one in the kent tree, +src/hg/utils/otto/uniprot, commit, then "make install" to push it out. The one +cron runs is /hive/data/outside/otto/uniprot. "make diff" lists files that have +drifted apart, worth checking now and then: the rsync in "make install" uses -u +and will not overwrite a live file that is newer than the tree. + +How it is started: + +Cron, from otto's crontab, on the 26th of the month: + + 00 07 26 * * /hive/data/outside/otto/uniprot/doUpdate.sh run + +doUpdate.sh activates venv/, checks that the parser can start, runs doUniprot and +reports the outcome. To start a run by hand, use doUpdate.sh, not doUniprot, so +the environment and the logging are the same as under cron. Python environment: -uniprotToTab needs the lxml XML parser, which is not in the python standard -library. On hgwdev it comes from the system package python3-lxml, and the two are -upgraded together, so nothing here has to be maintained. Check it with: +uniprotToTab needs the lxml XML parser. It is not in the python standard library +and hgwdev has no system-wide copy, so it lives in a virtualenv in venv/. Build +or rebuild it with: - python3 -c 'import lxml.etree' + cd /hive/data/outside/otto/uniprot && ./makeVenv.sh -Do not put a private virtualenv or conda environment in the path of this pipeline -without reading redmine #38300 first. There used to be a venv/ directory here, and -uniprotToTab appended a personal conda site-packages directory to sys.path. Both -were built for python 3.6 and a venv's python is only a symlink to the system one, -so when hgwdev moved to python 3.9 the compiled lxml stopped loading. Every monthly -run then died at the parse step and the tracks sat on release 2024_06 for nineteen -months. If the system python ever loses lxml, the pipeline now says so loudly on -the very first second of the run; the repair is to get python3-lxml back, or, as a -last resort, to build a self-contained conda environment (not a venv, which does -not carry its own python) and point the shebang of uniprotToTab at it. +Note that ~/.local is not enough: cron runs this as otto, which does not see +anyone else's per-user python packages. -Did it run? Failures: +Did it run? -doUpdate.sh appends one line per run to runLog.txt in the live directory, and that -file is never truncated, so it is the history of the job: +runLog.txt gets one line per run and is never truncated, so it is the history of +the job: START a run began NOCHANGE UniProt had no new release, nothing to do, no mail sent OK new release, tracks rebuilt FAIL the run died, exit code and log named on the line -lastRun.log holds the log of the most recent run and is overwritten every month. -When a run fails, doUpdate.sh keeps a copy as lastFail.log so the evidence is still -there next month, and mails the last 25 lines to the addresses on the MAILTO line -in otto's crontab. A month with no new UniProt release sends no mail at all, which -is the normal otto behaviour, so silence means "nothing to do", not "it worked". +lastRun.log is the log of the most recent run and is overwritten every month. A +failing run is kept as lastFail.log, and its last 25 lines are mailed to the +MAILTO addresses in otto's crontab. A month without a new UniProt release sends +no mail, so silence means "nothing to do", not "it worked". version.txt in each bigBed/<db> directory is what the trackDb dataVersion setting shows on the track description page. It is only rewritten when the release string -actually changes, so its date on disk is the date the data last moved, not the date -the pipeline last ran. Do not "fix" this by rewriting it every run: an unchanged -file with a fresh date is what made the nineteen-month stall look healthy. +changes, so its date on disk is the date the data last moved, not the date the +pipeline last ran. Directories: fasta - current protein sequences and their sizes, named by taxon geneMaps/ - a mapping of the current protein sequences to the genome, one for each species and database. Includes the md5 of the fasta files, so we do not have to recompute when sequences have not changed These files are used by pslMap. Also includes everything needed for a bigPsl file of these psls and the bigPsls themselves. bigBed - one bigBed for every subTrack Pipeline: The main driver script is doUniprot. It requires the parameter "run" to do @@ -116,17 +114,15 @@ The more complicated part is the mapping from UniProt to Genome. It's handled by the shell script makeUniProtPsl.sh. The script makes various assumptions that may need tweaking one day: - it uses mapUniprot_doBlast (a TCL script! HT to MarkD) for the BLAST cluster jobs. Note that we use /cluster/bin/blast/x86_64/blast-2.2.16/bin/blastall, an older BLAST version, to align UniProt protein sequences against transcript DNA sequences with tblastn - it uses various filters on the results, then pslMap's these to the genome through the transcript.psl. - the minimum percent ID of the alignments is 95%, because not all proteins match at 100% to the transcripts - it keeps only the top 1% of the alignments using pslCDnaFilter - any transcripts on _hap/_alt/_fix sequences are removed, to avoid that annotations are "sucked away" from the main chromosomes (should it use /hive/data/genomes/hg38/jkStuff/hg38.haplotypes.psl instead and the -hap option in pslCDnaFilter?) - I am not sure how to speed up the alignment. NCBI suggests to change the chunking, and create e.g. one query file for 10 queries and one query file for 10 targets, and align only queries to known targets. Was too much work, so for now the BLAST runs are a bit slow. - -