1682366b1827b7559f8e1e41635acff6c5ea15e9 max Wed Sep 9 06:05:05 2026 -0700 hprc2annot: move the makeDoc into its own directory and repoint the links The makeDoc has grown a companion (an hg38 pcLAI doc is in progress), so it moves from doc/contrib/hprc2annot.txt into doc/contrib/hprc2annot/, matching how the scripts and trackDb copies are already laid out. The file itself gains a section on the pcLAI scatterplot on the details page: where the reference panel comes from, the four ancestry centroids the discretized field takes across the release, and why the file is read through hgTrackUi rather than fetched by the browser. All seven track description pages linked to the old flat path and would have 404'd, so they are repointed. Six of them change only that link; pclai.html has further edits still in progress and keeps its own copy of the change. refs #35415 diff --git src/hg/makeDb/doc/contrib/hprc2annot.txt src/hg/makeDb/doc/contrib/hprc2annot/hprc2annot.txt similarity index 63% rename from src/hg/makeDb/doc/contrib/hprc2annot.txt rename to src/hg/makeDb/doc/contrib/hprc2annot/hprc2annot.txt index 61aec5f0fe9..e7b355b678c 100644 --- src/hg/makeDb/doc/contrib/hprc2annot.txt +++ src/hg/makeDb/doc/contrib/hprc2annot/hprc2annot.txt @@ -1,18 +1,21 @@ # 2026-07-16 Claude (max): HPRC Release 2 GenArk contributed track hub (refs #35415) # 2026-08-28 Claude (max): QA fixes -- liftoff transcript names, segdups strand, # pcLAI rebuild, stats retention, docs/trackDb into git +# 2026-09-09 Claude (max): pcLAI field pcaSegment renamed to centroid and its +# description corrected (see section 5c); genark +# detailsScript dataUrl double-prefix bug fixed # Builds a GenArk "contributed track" hub (contribTracks.html model) that adds # seven annotation tracks to the ~462 HPRC Release 2 assemblies already present as # GenArk assembly hubs: # catGenes liftoffGenes censat censatCentromeres segdups pclai methylation # # Hub location (served under hgdownload /hubs/ once a contrib symlink is added): # /hive/data/genomes/asmHubs/contrib/hprc2annot # Build work area (indexes, logs, per-job temp): # /hive/data/genomes/asmHubs/contrib/hprc2annot.build # Scripts (this directory): # ~/kent/src/hg/makeDb/scripts/hprc2annot # Track description pages and the trackDb stanza template (the master copies; # the hub's docs/ directory is refreshed from here): # ~/kent/src/hg/makeDb/trackDb/contrib/hprc2annot @@ -180,49 +183,182 @@ ############################################################################## # Master copies of the seven track description pages: # ~/kent/src/hg/makeDb/trackDb/contrib/hprc2annot/.html # hprc2annotMakeTrackDb.py copies them into the hub's docs/ directory, where all # 462 assemblies reference them. Edit the kent copy, never the hub copy. # References were added with /cluster/bin/scripts/getTrackReferences: # catGenes CAT PMID 29884752 # liftoff Liftoff PMID 33320174 # censat/cen cenSat PMID 35357911 # segdups SEDEF PMID 30423092 # pcLAI has no PMID; the preprint reference was taken from the Crossref record for # doi 10.64898/2026.03.23.713813 rather than written by hand: # curl -sH 'Accept: application/json' https://api.crossref.org/works/ # methylation: HPRC has not supplied a method citation. +############################################################################## +# 5b. pcLAI scatterplot on the details page +############################################################################## +# Clicking a pcLAI window shows where it sits in the ancestry space, as a +# scatterplot of the 1000 Genomes reference haplotypes with the window's own +# coordinate and its segment's coordinate marked. This uses the trackDb +# detailsScript mechanism with the scatterPlot plot type (hg/js/hgc.scatterPlot.js). +# +# The background points are the pcLAI authors' published reference panel: +# https://github.com/AI-sandbox/hprc-pclai/blob/main/reference_pca_metadata.tsv +# turned into the module's JSON by +# hprc2annotMakePclaiRefPanel.py reference_pca_metadata.tsv \ +# /hive/data/genomes/asmHubs/contrib/hprc2annot/pclaiRefPanel.json +# 3122 haplotypes, 21 populations, 94 KB. One file for the whole collection: it +# is the reference space, not per-assembly data. The script also repairs one +# mis-decoded en dash in the upstream population names ("Western Division D +# Mandinka"), which would otherwise show in the plot legend. +# +# The four values the centroid field ever takes across all 460 assemblies are the +# four ancestry cluster centres pcLAI discretizes to, which is worth knowing when +# reading the plot (nearest reference population to each -- our own reading of the +# panel, not a label the authors assign): +# (-1.743,0.207) African (Yoruba / Mende / Esan) +# (0.445,-1.314) European (Toscani / Iberian) +# (0.477,-0.507) South Asian (Sri Lankan Tamil / Telugu) +# (0.695,0.907) East Asian (Kinh / Han / Japanese) +# +# The file is not fetched by the browser from its own URL: hgc resolves the +# dataUrl against the track's bigDataUrl and the JS asks hgTrackUi for it, which +# only serves a path inside a hub attached to the cart. So it works for a hub +# loaded from a local path (the GenArk /gbdb hubs) and needs no CORS header, but +# it does have to sit inside the hub -- hence the symlink genark drops into each +# assembly's contrib/hprc2annot/ (see section 7). + +############################################################################## +# 5c. 2026-09-09: pcLAI "pcaSegment" -> "centroid", and what was wrong +############################################################################## +# Prompted by building the same annotation as a native hg38 track +# (doc/hg38/hprcPclai.txt) and reading the pcLAI authors' own format description +# for the first time: +# https://github.com/AI-sandbox/hprc-pclai (README, "Output format (BED)") +# +# What we had wrong. Column 10 of the source BED was documented here, in pclai.as +# and on the description page as "the PCA coordinate of the longer ancestry +# segment this window belongs to". It is not that. The authors call the column +# "centroid": the discretized pcLAI annotation of the window, written as the PCA +# centroid of the ancestry cluster the window falls into. That is why it only ever +# takes four values -- not because segments are megabases long and share a +# coordinate, but because there are four clusters. The old reading also implied a +# segmentation step in the method that does not exist: pcLAI predicts one +# coordinate per window and nothing else, and the visible blocks in the display +# are simply runs of windows with similar predictions. +# +# The README also settles three things that had been guesses: +# - windows are a fixed 1000 SNPs, not a fixed number of bases (~100 kb is the +# consequence, not the definition); +# - the format specifies thickStart == chromStart and thickEnd == chromEnd, so +# the occasional thickStart == chromStart-1 that hprc2annotBuildOne.sh works +# around is a bug in their files rather than something we misread; +# - low-confidence windows are dropped before publication, which is why the +# windows do not tile a sequence without gaps and why the score floor sits +# well above 0. +# +# What was changed: +# pclai.as pcaSegment -> centroid, descriptions corrected +# hprc2annot.trackDb.txt mouseOver + detailsScript exportFields +# trackDb/contrib/.../pclai.html description and methods rewritten +# hprc2annotBuildOne.sh converter comment only (columns unchanged) +# hprc2annotFixBed.sh comment only +# +# The field name and its description live inside each bigBed, so editing pclai.as +# is not enough -- the 460 built files had to be re-emitted. That is a pure .as +# swap with no column change, which neither hprc2annotBuild.sh (re-downloads +# everything) nor hprc2annotFixBed.sh (re-parses the name column, and refuses an +# already-converted file) will do, so hprc2annotRewriteAs.sh was added for it: +# cd /hive/data/genomes/asmHubs/contrib/hprc2annot +# ls */pclai.bb | parallel -j 16 hprc2annotRewriteAs.sh pclai {} +# # 460 OK, 11635199 items, unchanged; ~2 min +# It reads chrom.sizes out of the bigBed header (PanSN names, no assembly lookup), +# refuses a file whose column count does not match the .as, and checks the item +# count across the round trip. Unlike hprc2annotFixBed.sh it is safe to re-run. +# +# Worth knowing for next time: the .bb files on disk had been built from an older +# pclai.as than the one in the tree (the deployed files still carried the +# pre-reorganization field labels "PCA-space coordinates of this window|(PC1,PC2)" +# while the tree had "Window PCA|..."). Nothing detects that drift -- an .as edit +# alone silently does nothing to a built collection. hprc2annotRewriteAs.sh is +# the tool to run after any .as description edit. +# +# Then re-generate and re-wire, in this order: +# hprc2annotMakeTrackDb.py # hub trackDb.txt + docs/ from the kent copies +# genark addContrib hprc2annot # per-assembly stanzas inside the GenArk hubs +# The second step is easy to forget: hprc2annotMakeTrackDb.py only writes the +# contrib hub's own trackDb.txt. Each GenArk assembly hub holds a separate +# rewritten copy under /contrib/hprc2annot/hprc2annot.trackDb.txt, and +# until addContrib re-runs, that copy still has the old field name and the +# mouseOver renders the literal text instead of a value. + +############################################################################## +# 5d. genark addContrib bug: detailsScript dataUrl was prefixed twice +############################################################################## +# Found while checking the pcLAI details page after the rename: the scatterplot +# had never been drawing on the GenArk hubs. The request went to +# contrib/hprc2annot/contrib/hprc2annot/pclaiRefPanel.json +# and hgTrackUi's file fetch quietly returned nothing. +# +# Cause: rewriteTrackDb() in src/utils/genark/genark rebased the "...Url" inside a +# detailsScript value the same way it rebases bigDataUrl, to +# contrib//. But hgc (hg/hgc/bigBedClick.c) resolves a relative +# detailsScript Url against the track's own bigDataUrl when it builds the details +# page, and bigDataUrl had already been rebased -- so the prefix landed twice. +# In this layout the JSON is symlinked into contrib// right beside the .bb, +# so relative-to-the-.bb is just the bare file name. Fixed by giving detailsScript +# its own rebaseBeside() that reduces a local path to its basename and leaves a URL +# or an absolute path alone; it is idempotent, so addContrib can be re-run. +# +# Only shows up for a collection that uses detailsScript, i.e. only pcLAI today. +# Test after any change to that resolution path: load an assembly hub, click a +# pcLAI window, and confirm the "Window PCA" row draws a plot rather than sitting +# empty (an empty cell is exactly what a failed fetch looks like). +# +# While there: the makeDoc link on all seven description pages still pointed at +# doc/contrib/hprc2annot.txt, from before this file moved into its own directory. +# Repointed to doc/contrib/hprc2annot/hprc2annot.txt. + ############################################################################## # 6. Testing / serving ############################################################################## # For QA the hub is exposed via ~/public_html and loaded on the browser: # https://genome.ucsc.edu/cgi-bin/hgTracks?genome=GCA_041900255.1&hubUrl=https://hgwdev.gi.ucsc.edu/~max/hprc2annot/hub.txt # hubCheck https://hgwdev.gi.ucsc.edu/~max/hprc2annot/hub.txt # Production serving under hgdownload needs a hubs/contrib -> asmHubs/contrib # symlink (owned by the GenArk maintainer). ############################################################################## # 7. Wiring into the GenArk assembly hubs (src/utils/genark/genark) ############################################################################## # The general GenArk management tool genark installs the collection with one # command: # genark addContrib hprc2annot (genark addContrib --remove hprc2annot to undo) # For each accession it: # - creates /contrib/hprc2annot/ with symlinks to the data files # (.bb + methylation.bw -> the contrib hub) and the shared doc pages; # - writes a per-assembly hprc2annot.trackDb.txt whose bigDataUrl/html paths are # hub-root relative (contrib/hprc2annot/...); -# - wires that trackDb block into the assembly's useOneFile hub.txt, between -# "# BEGIN/END genark contrib: hprc2annot" markers (idempotent). +# - symlinks the collection's shared root-level data files (pclaiRefPanel.json) +# flat next to the docs, and rewrites a "...Url" inside a detailsScript value +# the same way as bigDataUrl, so contrib/hprc2annot/ resolves; +# - wires that trackDb block into the assembly's alpha.hub.txt, between +# "# BEGIN/END genark contrib: hprc2annot" markers (idempotent), first +# clearing any unmarked copy of our stanzas that the assembly build baked in +# (otherwise the hub would declare each track twice). +# Only the alpha tier is written for now. .singleFile.hub.txt, which +# /gbdb/genark//hub.txt points at and which is served as the assembly's +# default hub, is deliberately left alone. # A full GenArk hub rebuild regenerates hub.txt, so re-run addContrib after a # rebuild, or add hprc2annot handling to the build's asmHubTrackDb.sh for a # durable inclusion. # Ran on 462 assemblies (HG002 excluded). # # QA the installed collection with: # genark checkContrib hprc2annot # hubCheck a random sample of 5 # genark checkContrib hprc2annot --all # every assembly # genark checkContrib hprc2annot GCA_x ... # specific accessions # It runs hubCheck on each assembly hub and separates contrib-specific problems # from the assemblies' own pre-existing hub warnings (the HPRCv2 chain track names # contain periods, which hubCheck warns about -- not an hprc2annot issue).