af613a331e6839c6513c3e366abcb67af0fe8386 max Wed Sep 9 06:47:14 2026 -0700 UniProt otto: get the monthly update running again and make a stalled run visible The monthly UniProt job had produced nothing since January 2025. The tracks served release 2024_06 while the download sitting on disk was at 2026_02, on every assembly the job builds. Cause: uniprotToTab appended a personal conda site-packages directory to sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A venv's python is only a symlink to the system one, so when hgwdev moved to python 3.9 the compiled lxml in there stopped loading and every run died at the parse step. Removed both. The system python3 has lxml from python3-lxml and the two are upgraded together, so there is nothing left here to go stale. Verified by parsing real 2026_02 records under python 3.9 with lxml 5.4. Why nobody noticed for nineteen months: - doUpdate.sh read $? after an intervening echo, so it captured the echo's exit code and mailed "Big Uniprot update OK" every month while the job was dying. It now reads the real exit code, says FAILED, prints the tail of the log and exits nonzero. A month with no new UniProt release stays silent, which is the normal otto behaviour, so silence again means "nothing to do". - The logs were overwritten on every run, so a failure left no trace on disk. doUpdate.sh now appends one line per run to runLog.txt, which is never truncated, and keeps a failing log as lastFail.log. - version.txt in each bigBed directory was rewritten on every run even when the release string was identical. That is the file the trackDb dataVersion setting shows, and its date is what people check to decide whether a pipeline is still alive, so a stalled track could look freshly updated. It is now written only when the release actually changes. Also, so this cannot come back: - doUniprot checks that uniprotToTab can start before the download, instead of finding out 35 minutes later. - pylint on hgwdev is itself pinned to pythons that no longer exist, so "make install" aborted on its first line and could not be used. Replaced with a syntax check that needs nothing but python3; pylint stays best-effort. - uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from /hive/data/outside/otto/uniprot without being in the makefile's copy list. The tree copy of uniprotToTab was still python 2 from 2021. All are now listed and in sync, and "make diff" reports drift. - Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh and the pslMap -inType/-mapType flags. refs #38300 diff --git src/hg/utils/otto/uniprot/README.txt src/hg/utils/otto/uniprot/README.txt index 9e4a1c847ee..a51fac40043 100644 --- src/hg/utils/otto/uniprot/README.txt +++ src/hg/utils/otto/uniprot/README.txt @@ -1,21 +1,132 @@ -This is the automated pipeline to update uniprot tracks from UniProt.org - -UniProt updates its files every month, see http://www.uniprot.org/news/ - -Our process is this (doUniprot): - -- compare local date against ftp and do nothing is not new -- if the file has not been updated on the uniprot.org server, exit -- parse the uniprot data from XML to tab files, one per organism -- create an alignment uniprotId -> genome with MarkD's pslMap and pslSelect - These are stored in the directory protToGenome -- for pslSelect, the process is described in detail in the track description page: - use uniProt's annotated transcripts first, then try those annotated to the NCBI Entrez Gene, - then finally let pslDnaFilter use the best match (so no entry for pslSelect, this is OK because - of the -qPass option we use for pslSelect here) -- the MINALI global variable set how much proteins must align, it is set to 0.93 by default -- Creates an archive directory and a hub.txt for it -- Finally goes over assemblies and flips the bigBed files -- Produces detailed mapInfo.json files in the bigBed directory +UniProt mapping pipeline, Max 2016/2017, updates in 2021 + +This is the automated pipeline that updates the UniProt tracks from UniProt.org. +UniProt puts out a new release every month, see http://www.uniprot.org/news/ + +Two copies of these scripts exist and they are not the same file: + the tree copy, src/hg/utils/otto/uniprot in the kent repo, is the one to edit + the live copy, /hive/data/outside/otto/uniprot, is the one cron runs +Edit the tree copy, commit, then "make install" to push it out. "make diff" lists +files that have drifted apart, which is worth checking now and then: the rsync in +"make install" uses -u and so refuses to overwrite a live file that is newer than +the tree, which means a hand edit of the live copy wins silently and forever. + +Python environment: + +uniprotToTab needs the lxml XML parser, which is not in the python standard +library. On hgwdev it comes from the system package python3-lxml, and the two are +upgraded together, so nothing here has to be maintained. Check it with: + + python3 -c 'import lxml.etree' + +Do not put a private virtualenv or conda environment in the path of this pipeline +without reading redmine #38300 first. There used to be a venv/ directory here, and +uniprotToTab appended a personal conda site-packages directory to sys.path. Both +were built for python 3.6 and a venv's python is only a symlink to the system one, +so when hgwdev moved to python 3.9 the compiled lxml stopped loading. Every monthly +run then died at the parse step and the tracks sat on release 2024_06 for nineteen +months. If the system python ever loses lxml, the pipeline now says so loudly on +the very first second of the run; the repair is to get python3-lxml back, or, as a +last resort, to build a self-contained conda environment (not a venv, which does +not carry its own python) and point the shebang of uniprotToTab at it. + +Did it run? Failures: + +doUpdate.sh appends one line per run to runLog.txt in the live directory, and that +file is never truncated, so it is the history of the job: + + START a run began + NOCHANGE UniProt had no new release, nothing to do, no mail sent + OK new release, tracks rebuilt + FAIL the run died, exit code and log named on the line + +lastRun.log holds the log of the most recent run and is overwritten every month. +When a run fails, doUpdate.sh keeps a copy as lastFail.log so the evidence is still +there next month, and mails the last 25 lines to the addresses on the MAILTO line +in otto's crontab. A month with no new UniProt release sends no mail at all, which +is the normal otto behaviour, so silence means "nothing to do", not "it worked". + +version.txt in each bigBed/<db> directory is what the trackDb dataVersion setting +shows on the track description page. It is only rewritten when the release string +actually changes, so its date on disk is the date the data last moved, not the date +the pipeline last ran. Do not "fix" this by rewriting it every run: an unchanged +file with a fresh date is what made the nineteen-month stall look healthy. + +Directories: + +fasta - current protein sequences and their sizes, named by taxon +geneMaps/ - a mapping of the current protein sequences to the genome, one for each species and database. + Includes the md5 of the fasta files, so we do not have to recompute when sequences have not changed + These files are used by pslMap. + + Also includes everything needed for a bigPsl file of these psls and the bigPsls themselves. + +bigBed - one bigBed for every subTrack + +Pipeline: + +The main driver script is doUniprot. It requires the parameter "run" to do +anything. It goes through these steps: +- downloads UniProt XML with lftp. This takes 2-3 days. Skip this step with -l for development. + [ It's hard to speed this up, as the EBI FTP server does not allow parallel connections, at the time + of writing. ] +- converts it to tab-sep and fasta files using uniprotToTab. This takes 3-4 days! Skip this step with + -p whenever you can for debugging or development. + [ jIt is hard to speed this up, as there is only a single huge XML file, without an index. The EBI + has a pilot where they provide offsets into the XML but it's not a real produce yet. ] +- for each UniProt taxon ID, find the relevant UCSC db identifiers, adding a few manual overrides, + e.g. 9606 always uses both hg19 and hg38. wuhCor1 is skipped, etc. + You can limit the script to only certain dbs with e.g. --onlyDbs=hg19 + You can display the current mapping with 'doUniprot --db' + This will also show the correct trackDb make command if you want to remake all trackDbs. +- for each assembly, try to guess a transcript gene track and find transcript + sequences for it and finds or fakes a transcript.psl file for the transcripts. + Supported transcript tracks are: ncbiRefSeq, ensembl, augustus. + They are tried in this order. Hg19 is hardcoded to refGene because CSAG3 NM_001129826.3 + exists only on chrX_jh159150_fix in ncbiRefSeq, Terence confirmed this is an issue. + hg38 is hardcoded to ncbiRefSeq. see findBestGeneTable() +- tries to create a UniProtId <-> transcriptId pairs table, if possible (from the UniProt xref fields) + This massively reduces the false positives, for protein families with a lot of almost + identical transcripts. There often a small mismatches between this table and + the actual transcript set, so the file is cleaned up (and later pslSelect is run + with -qPass, so any alignments that do not appear in the file just go through) +- aligns the UniProt fasta files against the transcript + sequences with BLAST, using the script makeUniProtPsl.sh, on the cluster, + which uses mapUniprot_doBlast for the cluster jobs. + This can take 1-2 hours, even using the cluster. + The output are PSL "lift" files for pslMap, one per assembly. The file names include + the gene track MD5 and the pslSelect MD5 and the old results are reused, if the MD5s + match (CPU-heavy BLAST alignment). See section below for more details. + If the old mapping can be reused, a run of the pipeline takes < 1min per assembly. +- Writes the UniProt features to PSL files, then lifts those with pslMap to the + genome using the lift files that were just created on the cluster +- Converts the resulting UniProt annotation PSLs to BED and then to bigBed + The pslMap alignments are converted to bigPsl with various extra fields. + These bigPsl files are split into SwissProt/Trembl, two subtracks, because + our filters don't work on bigPsl and also because the Trembl sequences on human + are pretty useless anyways. +- If everything was successful for all assemblies: + - link the new bigBeds into /gbdb/ + - create little version.txt files in every bigBed directory to indicate the UniProt release + - copy the new files to /usr/local/apache/htdocs-hgdownload/goldenPath/archive/hg19/uniprot/<release>/ + and update the 'current' symlink there. Throw in the UniProt -> Genome PSL file. + +Alignments: + +The more complicated part is the mapping from UniProt to Genome. +It's handled by the shell script makeUniProtPsl.sh. The script makes various +assumptions that may need tweaking one day: +- it uses mapUniprot_doBlast (a TCL script! HT to MarkD) for the BLAST cluster jobs. + Note that we use /cluster/bin/blast/x86_64/blast-2.2.16/bin/blastall, an older + BLAST version, to align UniProt protein sequences against transcript DNA sequences with tblastn +- it uses various filters on the results, then pslMap's these to the genome through the transcript.psl. +- the minimum percent ID of the alignments is 95%, because not all proteins match at 100% to the transcripts +- it keeps only the top 1% of the alignments using pslCDnaFilter +- any transcripts on _hap/_alt/_fix sequences are removed, to avoid that annotations are "sucked away" from the main + chromosomes (should it use /hive/data/genomes/hg38/jkStuff/hg38.haplotypes.psl instead and the -hap option in pslCDnaFilter?) +- I am not sure how to speed up the alignment. NCBI suggests to change the chunking, and create + e.g. one query file for 10 queries and one query file for 10 targets, and align only + queries to known targets. Was too much work, so for now the BLAST runs are a bit slow. +