fe02c55d26614f1d5059c884fb2594f9b4d5ad50 mspeir Wed Aug 12 08:02:19 2026 -0700 singleCellSignalsPeaks: point at the hub build's new home, fix the file-copy script The hub build now lives in the cellBrowser repo under ucsc/allTracksHub, so the makeDocs link there instead of a personal work dir. HUB_BUILD in these scripts is the build's output dir, not its code; comments say so now, paths unchanged. copySingleCellSignalsPeaksFiles.py was silently copying nothing: it compared a whole line against "parent ", but stanzas are now indented and read "parent off". Now dedents, matches the first token, and refuses to run on zero subtracks. Dry runs give 925 hg38 / 587 mm10, none missing. refs #37914 diff --git src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt index 26f2f102543..61e8dcf0770 100644 --- src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt +++ src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt @@ -1,45 +1,50 @@ # mm10 singleCellSignalsPeaks track - 2026-07-22 Claude (mspeir) refs #37914 # The native mm10 "singleCellSignalsPeaks" faceted composite is the Genome # Browser version of the mm10 signal-&-peaks composite (cellBrowserMm10) of the -# UCSC Cell Browser all-tracks super hub (built under -# /hive/users/mspeir/claude/cell-browser/all-tracks-hub-build). It gathers the -# per-cell-type ATAC-seq signal (bigWig) and peak (bigNarrowPeak) tracks from the +# UCSC Cell Browser all-tracks super hub, whose build lives in the cellBrowser +# repo, ucsc/allTracksHub: +# https://github.com/ucscGenomeBrowser/cellBrowser/tree/develop/ucsc/allTracksHub +# +# It gathers the per-cell-type ATAC-seq signal (bigWig) and peak (bigNarrowPeak) tracks from the # single-cell ATAC datasets in the Cell Browser and re-parents them under one # faceted composite. It is the mm10 counterpart of the hg38 track of the same # name (see doc/hg38/singleCellSignalsPeaks.txt, Redmine #37820). Histone marks, # cCREs, RNA/expression, interactions, and the separately-faceted motor-neuron # and brain-spatial sets each live in their own hub composites and are NOT part # of this track. ############################################################################## # 1. Source data ############################################################################## # The track mirrors the hub's main mm10 signal-&-peaks faceted composite # (cellBrowserMm10): 647 subtracks (641 bigWig, 6 bigNarrowPeak) from 9 datasets # - catlas-mouse-aging (234), catlas-mouse-brain (160), allen-basal-ganglia-atac # (131), olg-dyn-eae-multiome (28), mouse-lvcp-multiome (23, incl. the 6 peaks), # catlas-paired-tag (21), mouse-kidney-atac (19), olg-eae-ms (16) and # mouse-epi-juv-brain (15). That composite and its facet metadata are produced by # the hub build from the Cell Browser dataset tree: # -# cd /hive/users/mspeir/claude/cell-browser/all-tracks-hub-build +# cd $HOME/cellBrowser/ucsc/allTracksHub # python3 build_manifest.py # scan datasets -> manifest.tsv # python3 build_stanzas.py # manifest -> stanzas/mm10.trackDb.txt # # + meta/mm10.metadata.tsv # +# The build writes its output to $CBHUB_OUT, NOT next to the scripts (that dir is +# git-controlled). Default: /hive/users/mspeir/claude/cell-browser/all-tracks-hub-build +# # The per-track source files (abs_path column of manifest.tsv) are the files the # Cell Browser datasets already serve; nothing is regenerated here, only copied. ############################################################################## # 2. Copy the data files into place (bed dir, served via a /gbdb symlink) ############################################################################## # copySingleCellSignalsPeaksFiles.py copies every cellBrowserMm10 subtrack file # into /hive/data/genomes/mm10/bed/singleCellSignalsPeaks/, # keeping each file's served relative path, and copies the composite's facet # metadata to /singleCellSignalsPeaks_metadata.tsv. The served subpath is # preserved on purpose: some coverage/peak basenames (e.g. MOL.bw, OPC.bw) repeat # across datasets, so a flat directory would clobber them. 89.6 GB, 647 files. # # scriptDir=$HOME/kent/src/hg/makeDb/scripts/singleCellSignalsPeaks # python3 $scriptDir/copySingleCellSignalsPeaksFiles.py --assembly mm10 --dry-run @@ -164,44 +169,47 @@ # be made there too. # # One copy of the crosswalks: build_stanzas reads celltype-crosswalks/ out of the kent # tree (XWALK_ROOT, overridable), not the copy in the hub-build dir. There used to be # two independent copies and they drifted -- a stale local copy silently reverted the # cell-class fix above while the kent copy looked correct. # # The curation of record is archived with the scripts: # scripts/singleCellSignalsPeaks/build_celltype_crosswalks.py # scripts/singleCellSignalsPeaks/celltype-crosswalks/ (per-collection crosswalks, # celltype-palette.tsv, celltype-class.tsv, sea-ad-celltype-crosswalk.tsv) # scripts/singleCellSignalsPeaks/celltype-crosswalks/paper-decodes/ (the paper-curated # decode tables, each row's `note` giving its source justification) # build_celltype_crosswalks.py rebuilds the crosswalks + palette from paper-decodes/. # The general hub machinery that consumes them (build_manifest.py, build_stanzas.py, -# hub_config.json) lives in the all-tracks-hub-build dir noted above, not in the kent -# tree -- it builds the whole super hub, not just this track. +# build_hub.py, hub_config.json) lives in the cellBrowser repo, not in the kent tree -- +# it builds the whole super hub, not just this track: +# https://github.com/ucscGenomeBrowser/cellBrowser/tree/develop/ucsc/allTracksHub +# build_stanzas finds the crosswalks archived here via XWALK_ROOT, so the hub and these +# native tracks always color a cell class the same. ############################################################################## # 5. Redundant allen-brain-science copies removed ############################################################################## # The Allen basal-ganglia dataset serves each unchanged per-cluster bigWig from all # four of its grouping directories (bg_regrouping_cl, bg_merge_D1_D2, # bg_merge_dorsal_ventral, bg_merge_D1_D2_dorsal_ventral). For 14 basenames those # four copies are byte-identical (md5-verified), so they rendered as four # indistinguishable subtracks and cost 4.4 GB of duplicated storage. build_stanzas # keeps the bg_regrouping_cl copy -- the cluster-level source the merges are built # from -- and skips the other three (allen_duplicate_skip; see -# all-tracks-hub-build/allen-duplicates.log for the list). Where the four copies +# $CBHUB_OUT/allen-duplicates.log for the list). Where the four copies # genuinely differ (14 other basenames, different aggregations and different file # sizes) all four are kept and told apart by the grouping-level variant descriptor. # The 42 redundant files were deleted from the bed dir; the list is kept alongside # them in deleted-redundant-allen-copies.txt. Only the one directory-level /gbdb # symlink exists, so nothing needed cleaning up there. # # xargs -a deleted-redundant-allen-copies.txt rm ############################################################################## # Counts ############################################################################## # 647 files resolved from the hub manifest (0 missing) and copied to the bed dir. # 42 byte-identical Allen copies were then removed, leaving 605 data files. 18 # QC-cluster tracks (doublet / low-quality / batch) are dropped when the .ra is # generated, leaving 587 subtracks in the track (581 signal + 6 peak). Facet