af613a331e6839c6513c3e366abcb67af0fe8386 max Wed Sep 9 06:47:14 2026 -0700 UniProt otto: get the monthly update running again and make a stalled run visible The monthly UniProt job had produced nothing since January 2025. The tracks served release 2024_06 while the download sitting on disk was at 2026_02, on every assembly the job builds. Cause: uniprotToTab appended a personal conda site-packages directory to sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A venv's python is only a symlink to the system one, so when hgwdev moved to python 3.9 the compiled lxml in there stopped loading and every run died at the parse step. Removed both. The system python3 has lxml from python3-lxml and the two are upgraded together, so there is nothing left here to go stale. Verified by parsing real 2026_02 records under python 3.9 with lxml 5.4. Why nobody noticed for nineteen months: - doUpdate.sh read $? after an intervening echo, so it captured the echo's exit code and mailed "Big Uniprot update OK" every month while the job was dying. It now reads the real exit code, says FAILED, prints the tail of the log and exits nonzero. A month with no new UniProt release stays silent, which is the normal otto behaviour, so silence again means "nothing to do". - The logs were overwritten on every run, so a failure left no trace on disk. doUpdate.sh now appends one line per run to runLog.txt, which is never truncated, and keeps a failing log as lastFail.log. - version.txt in each bigBed directory was rewritten on every run even when the release string was identical. That is the file the trackDb dataVersion setting shows, and its date is what people check to decide whether a pipeline is still alive, so a stalled track could look freshly updated. It is now written only when the release actually changes. Also, so this cannot come back: - doUniprot checks that uniprotToTab can start before the download, instead of finding out 35 minutes later. - pylint on hgwdev is itself pinned to pythons that no longer exist, so "make install" aborted on its first line and could not be used. Replaced with a syntax check that needs nothing but python3; pylint stays best-effort. - uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from /hive/data/outside/otto/uniprot without being in the makefile's copy list. The tree copy of uniprotToTab was still python 2 from 2021. All are now listed and in sync, and "make diff" reports drift. - Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh and the pslMap -inType/-mapType flags. refs #38300 diff --git src/hg/utils/otto/uniprot/trackDb.template.txt src/hg/utils/otto/uniprot/trackDb.template.txt new file mode 100644 index 00000000000..173f1998fac --- /dev/null +++ src/hg/utils/otto/uniprot/trackDb.template.txt @@ -0,0 +1,224 @@ +# this file is used for building the archive track hub, it is template, not the final file. +# doUniprot builds the final file. +# if you modify this file, make sure that you also modify the original of the native track in trackDb/uniprot.ra + +track uniprot$VER +shortLabel UniProt $VER +longLabel UniProt SwissProt/TrEMBL Protein Annotations, release $VER +#group genes +visibility hide +type bigBed 12 + +exonNumbers off +compositeTrack on +allButtonPair on +mouseOverField comments +urls uniProtId="http://www.uniprot.org/uniprot/$$#section_features" pmids="https://www.ncbi.nlm.nih.gov/pubmed/$$" +itemRgb on +#dataVersion /gbdb/$D/uniprot/version.txt +hideEmptySubtracks on + + track unipAliSwissprot$VER + shortLabel SwissProt Aln. + longLabel UCSC alignment of SwissProt proteins to genome - $VER + priority 1 + parent uniprot$VER + visibility hide + type bigPsl + bigDataUrl $VER/unipAliSwissprot.bb + searchIndex name,acc + urls acc="http://www.uniprot.org/uniprot/$$" hgncId="https://www.genenames.org/cgi-bin/gene_symbol_report?hgnc_id=$$" + labelFields acc,uniprotName,geneName,hgncSym,refSeq,refSeqProt,ensProt,uniprotName + mouseOverField protFullNames + itemRgb off + color 2,12,120 + indelDoubleInsert on + indelQueryInsert on + baseColorTickColor contrastingColor + baseColorUseCds given + baseColorDefault genomicCodons + showDiffBasesAllScales on + + track unipAliTrembl$VER + shortLabel TrEMBL Aln. + longLabel UCSC alignment of TrEMBL proteins to genome - $VER + priority 2 + parent uniprot$VER + visibility dense + type bigPsl + bigDataUrl $VER/unipAliTrembl.bb + searchIndex name,acc + urls acc="http://www.uniprot.org/uniprot/$$" hgncId="https://www.genenames.org/cgi-bin/gene_symbol_report?hgnc_id=$$" + labelFields acc,uniprotName,geneName,hgncSym,refSeq,refSeqProt,ensProt,uniprotName + mouseOverField protFullNames + itemRgb off + color 0,150,250 + indelDoubleInsert on + indelQueryInsert on + baseColorTickColor contrastingColor + baseColorUseCds given + baseColorDefault genomicCodons + showDiffBasesAllScales on + + track unipLocSignal$VER + shortLabel Signal Peptide + longLabel UniProt Signal Peptides - $VER + priority 3 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipLocSignal.bb + color 255,0,150 + itemRgb off + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipLocExtra$VER + shortLabel Extracellular + longLabel UniProt Extracellular Domain - $VER + priority 4 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipLocExtra.bb + color 0,150,255 + itemRgb off + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipLocTransMemb$VER + shortLabel Transmembrane + longLabel UniProt Transmembrane Domains - $VER + priority 5 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipLocTransMemb.bb + color 0,150,0 + itemRgb off + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipLocCytopl$VER + shortLabel Cytoplasmic + longLabel UniProt Cytoplasmic Domains - $VER + priority 6 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipLocCytopl.bb + color 255,150,0 + itemRgb off + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipChain$VER + shortLabel Chains + longLabel UniProt Mature Protein Products (Polypeptide Chains) - $VER + priority 7 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipChain.bb + urls uniProtId="http://www.uniprot.org/uniprot/$$#ptm_processing" pmids="https://www.ncbi.nlm.nih.gov/pubmed/$$" + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipInterest$VER + shortLabel Interest + longLabel UniProt Regions of Interest - $VER + priority 4 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipInterest.bb + itemRgb off + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipDomain$VER + shortLabel Domains + longLabel UniProt Domains - $VER + priority 8 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipDomain.bb + urls uniProtId="http://www.uniprot.org/uniprot/$$#family_and_domains" pmids="https://www.ncbi.nlm.nih.gov/pubmed/$$" + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipDisulfBond$VER + shortLabel Disulf. Bonds + longLabel UniProt Disulfide Bonds - $VER + priority 8 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipDisulfBond.bb + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipModif$VER + shortLabel AA Modifications + longLabel UniProt Amino Acid Modifications - $VER + priority 9 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipModif.bb + urls uniProtId="http://www.uniprot.org/uniprot/$$#aaMod_section" pmids="https://www.ncbi.nlm.nih.gov/pubmed/$$" + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipMut$VER + shortLabel Mutations + longLabel UniProt Amino Acid Mutations - $VER + priority 10 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipMut.bb + urls uniProtId="http://www.uniprot.org/uniprot/$$#pathology_and_biotech" pmids="https://www.ncbi.nlm.nih.gov/pubmed/$$" variationId="http://www.uniprot.org/uniprot/$$" + + track unipStruct$VER + shortLabel Structure + longLabel UniProt Protein Primary/Secondary Structure Annotations - $VER + priority 11 + parent uniprot$VER + visibility hide + type bigBed 12 + + bigDataUrl $VER/unipStruct.bb + urls uniProtId="http://www.uniprot.org/uniprot/$$#structure" pmids="https://www.ncbi.nlm.nih.gov/pubmed/$$" + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipOther$VER + shortLabel Other Annot. + longLabel UniProt Other Annotations - $VER + priority 11 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipOther.bb + urls uniProtId="http://www.uniprot.org/uniprot/$$#family_and_domains" pmids="https://www.ncbi.nlm.nih.gov/pubmed/$$" + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipRepeat$VER + shortLabel Repeats + longLabel UniProt Repeats - $VER + priority 12 + parent uniprot$VER + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipRepeat.bb + urls uniProtId="http://www.uniprot.org/uniprot/$$#family_and_domains" pmids="https://www.ncbi.nlm.nih.gov/pubmed/$$" + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + + track unipConflict$VER + shortLabel Seq. Conflicts + longLabel UniProt Sequence Conflicts - $VER + priority 13 + parent uniprot$VER off + visibility dense + type bigBed 12 + + bigDataUrl $VER/unipConflict.bb + urls uniProtId="http://www.uniprot.org/uniprot/$$#Sequence_conflict_section" pmids="https://www.ncbi.nlm.nih.gov/pubmed/$$" + filterValues.status Manually reviewed (Swiss-Prot),Unreviewed (TrEMBL) + +#searchTable unipAliSwissprot +#searchType bigBed +#searchDescription UniProt/SwissProt manually curated Proteins mapped to the genome by UCSC + +#searchTable unipAliTrembl +#searchType bigBed +#searchDescription UniProt/TrEMBL predicted Proteins mapped to the genome by UCSC