d033cea2063e9362949baf5b4d8b837597173a0d
max
Thu Sep 10 05:16:05 2026 -0700
Address the code review of the Sep 9 commits
Faceted composite: text that comes from a hub - a metadata column's
description, the name and title of a data type, and the values quoted back in
the "could not load the metadata" row - is put on the page as text rather than
as markup. The three places built their markup from template strings, so a
value carrying angle brackets or a quote was read as HTML: the column
description now goes through the shared htmlEncode() once where the header is
parsed, and the other two build their elements as nodes. The error row reads
better for it as well, since a value with brackets in it used to disappear from
the message that was meant to show it.
The saved UI state keys on the assembly as well as the metadata id. localStorage
is per-origin, so two assemblies whose tracks share a name were sharing one
entry, and a row order dragged on one came back on the other over a different
set of samples. hgTrackUi passes the database down for it. State saved under the
old key is dropped, which costs a facet selection or a page length.
Imprinting: the five subtrack description pages link back to the container as
hgTrackUi?db=$db&g=$parentTrack, without the hgsid. Native trackDb html is
substituted by hgTrackDb as it loads the table, where there is no cart, so
${hgsid} came out empty and the link read 'hgsid=&g=...'. Matches what the
Fiber-seq pages already do. The makeDoc note that described the old form is
updated with the reason.
UniProt otto: README.txt lists all eight things that reach runLog.txt. It had
four, and was missing LOCKED, along with PREFLIGHT-FAIL, END and INTERRUPTED.
refs #36210
refs #37599
refs #38300
diff --git src/hg/utils/otto/uniprot/README.txt src/hg/utils/otto/uniprot/README.txt
index 1a89dc56e93..11a2dec2488 100644
--- src/hg/utils/otto/uniprot/README.txt
+++ src/hg/utils/otto/uniprot/README.txt
@@ -1,128 +1,134 @@
UniProt mapping pipeline, Max 2016/2017, updates in 2021
Updates the UniProt tracks from UniProt.org, which puts out a new release every
month. See http://www.uniprot.org/news/
Two copies of these scripts exist. Edit the one in the kent tree,
src/hg/utils/otto/uniprot, commit, then "make install" to push it out. The one
cron runs is /hive/data/outside/otto/uniprot. "make diff" lists files that have
drifted apart, worth checking now and then: the rsync in "make install" uses -u
and will not overwrite a live file that is newer than the tree.
How it is started:
Cron, from otto's crontab, on the 26th of the month:
00 07 26 * * /hive/data/outside/otto/uniprot/doUpdate.sh run
doUpdate.sh activates venv/, checks that the parser can start, runs doUniprot and
reports the outcome. To start a run by hand, use doUpdate.sh, not doUniprot, so
the environment and the logging are the same as under cron.
Python environment:
uniprotToTab needs the lxml XML parser. It is not in the python standard library
and hgwdev has no system-wide copy, so it lives in a virtualenv in venv/. Build
or rebuild it with:
cd /hive/data/outside/otto/uniprot && ./makeVenv.sh
Note that ~/.local is not enough: cron runs this as otto, which does not see
anyone else's per-user python packages.
Did it run?
runLog.txt gets one line per run and is never truncated, so it is the history of
the job:
+ PREFLIGHT-FAIL the venv or the parser was not usable, doUniprot never ran
START a run began
+ END doUniprot returned, exit code on the line
+ LOCKED another doUniprot holds the lock file, this run did nothing
NOCHANGE UniProt had no new release, nothing to do, no mail sent
OK new release, tracks rebuilt
FAIL the run died, exit code and log named on the line
+ INTERRUPTED a signal killed the run, the lock file was removed
+
+A normal month is START, END, then one of LOCKED, NOCHANGE, OK or FAIL.
lastRun.log is the log of the most recent run and is overwritten every month. A
failing run is kept as lastFail.log, and its last 25 lines are mailed to the
MAILTO addresses in otto's crontab. A month without a new UniProt release sends
no mail, so silence means "nothing to do", not "it worked".
version.txt in each bigBed/<db> directory is what the trackDb dataVersion setting
shows on the track description page. It is only rewritten when the release string
changes, so its date on disk is the date the data last moved, not the date the
pipeline last ran.
Directories:
fasta - current protein sequences and their sizes, named by taxon
geneMaps/ - a mapping of the current protein sequences to the genome, one for each species and database.
Includes the md5 of the fasta files, so we do not have to recompute when sequences have not changed
These files are used by pslMap.
Also includes everything needed for a bigPsl file of these psls and the bigPsls themselves.
bigBed - one bigBed for every subTrack
Pipeline:
The main driver script is doUniprot. It requires the parameter "run" to do
anything. It goes through these steps:
- downloads UniProt XML with lftp. This takes 2-3 days. Skip this step with -l for development.
[ It's hard to speed this up, as the EBI FTP server does not allow parallel connections, at the time
of writing. ]
- converts it to tab-sep and fasta files using uniprotToTab. This takes 3-4 days! Skip this step with
-p whenever you can for debugging or development.
[ jIt is hard to speed this up, as there is only a single huge XML file, without an index. The EBI
has a pilot where they provide offsets into the XML but it's not a real produce yet. ]
- for each UniProt taxon ID, find the relevant UCSC db identifiers, adding a few manual overrides,
e.g. 9606 always uses both hg19 and hg38. wuhCor1 is skipped, etc.
You can limit the script to only certain dbs with e.g. --onlyDbs=hg19
You can display the current mapping with 'doUniprot --db'
This will also show the correct trackDb make command if you want to remake all trackDbs.
- for each assembly, try to guess a transcript gene track and find transcript
sequences for it and finds or fakes a transcript.psl file for the transcripts.
Supported transcript tracks are: ncbiRefSeq, ensembl, augustus.
They are tried in this order. Hg19 is hardcoded to refGene because CSAG3 NM_001129826.3
exists only on chrX_jh159150_fix in ncbiRefSeq, Terence confirmed this is an issue.
hg38 is hardcoded to ncbiRefSeq. see findBestGeneTable()
- tries to create a UniProtId <-> transcriptId pairs table, if possible (from the UniProt xref fields)
This massively reduces the false positives, for protein families with a lot of almost
identical transcripts. There often a small mismatches between this table and
the actual transcript set, so the file is cleaned up (and later pslSelect is run
with -qPass, so any alignments that do not appear in the file just go through)
- aligns the UniProt fasta files against the transcript
sequences with BLAST, using the script makeUniProtPsl.sh, on the cluster,
which uses mapUniprot_doBlast for the cluster jobs.
This can take 1-2 hours, even using the cluster.
The output are PSL "lift" files for pslMap, one per assembly. The file names include
the gene track MD5 and the pslSelect MD5 and the old results are reused, if the MD5s
match (CPU-heavy BLAST alignment). See section below for more details.
If the old mapping can be reused, a run of the pipeline takes < 1min per assembly.
- Writes the UniProt features to PSL files, then lifts those with pslMap to the
genome using the lift files that were just created on the cluster
- Converts the resulting UniProt annotation PSLs to BED and then to bigBed
The pslMap alignments are converted to bigPsl with various extra fields.
These bigPsl files are split into SwissProt/Trembl, two subtracks, because
our filters don't work on bigPsl and also because the Trembl sequences on human
are pretty useless anyways.
- If everything was successful for all assemblies:
- link the new bigBeds into /gbdb/
- create little version.txt files in every bigBed directory to indicate the UniProt release
- copy the new files to /usr/local/apache/htdocs-hgdownload/goldenPath/archive/hg19/uniprot/<release>/
and update the 'current' symlink there. Throw in the UniProt -> Genome PSL file.
Alignments:
The more complicated part is the mapping from UniProt to Genome.
It's handled by the shell script makeUniProtPsl.sh. The script makes various
assumptions that may need tweaking one day:
- it uses mapUniprot_doBlast (a TCL script! HT to MarkD) for the BLAST cluster jobs.
Note that we use /cluster/bin/blast/x86_64/blast-2.2.16/bin/blastall, an older
BLAST version, to align UniProt protein sequences against transcript DNA sequences with tblastn
- it uses various filters on the results, then pslMap's these to the genome through the transcript.psl.
- the minimum percent ID of the alignments is 95%, because not all proteins match at 100% to the transcripts
- it keeps only the top 1% of the alignments using pslCDnaFilter
- any transcripts on _hap/_alt/_fix sequences are removed, to avoid that annotations are "sucked away" from the main
chromosomes (should it use /hive/data/genomes/hg38/jkStuff/hg38.haplotypes.psl instead and the -hap option in pslCDnaFilter?)
- I am not sure how to speed up the alignment. NCBI suggests to change the chunking, and create
e.g. one query file for 10 queries and one query file for 10 targets, and align only
queries to known targets. Was too much work, so for now the BLAST runs are a bit slow.