fe02c55d26614f1d5059c884fb2594f9b4d5ad50 mspeir Wed Aug 12 08:02:19 2026 -0700 singleCellSignalsPeaks: point at the hub build's new home, fix the file-copy script The hub build now lives in the cellBrowser repo under ucsc/allTracksHub, so the makeDocs link there instead of a personal work dir. HUB_BUILD in these scripts is the build's output dir, not its code; comments say so now, paths unchanged. copySingleCellSignalsPeaksFiles.py was silently copying nothing: it compared a whole line against "parent <composite>", but stanzas are now indented and read "parent <composite> off". Now dedents, matches the first token, and refuses to run on zero subtracks. Dry runs give 925 hg38 / 587 mm10, none missing. refs #37914 diff --git src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt index 26f2f102543..61e8dcf0770 100644 --- src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt +++ src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt @@ -1,45 +1,50 @@ # mm10 singleCellSignalsPeaks track - 2026-07-22 Claude (mspeir) refs #37914 # The native mm10 "singleCellSignalsPeaks" faceted composite is the Genome # Browser version of the mm10 signal-&-peaks composite (cellBrowserMm10) of the -# UCSC Cell Browser all-tracks super hub (built under -# /hive/users/mspeir/claude/cell-browser/all-tracks-hub-build). It gathers the -# per-cell-type ATAC-seq signal (bigWig) and peak (bigNarrowPeak) tracks from the +# UCSC Cell Browser all-tracks super hub, whose build lives in the cellBrowser +# repo, ucsc/allTracksHub: +# https://github.com/ucscGenomeBrowser/cellBrowser/tree/develop/ucsc/allTracksHub +# +# It gathers the per-cell-type ATAC-seq signal (bigWig) and peak (bigNarrowPeak) tracks from the # single-cell ATAC datasets in the Cell Browser and re-parents them under one # faceted composite. It is the mm10 counterpart of the hg38 track of the same # name (see doc/hg38/singleCellSignalsPeaks.txt, Redmine #37820). Histone marks, # cCREs, RNA/expression, interactions, and the separately-faceted motor-neuron # and brain-spatial sets each live in their own hub composites and are NOT part # of this track. ############################################################################## # 1. Source data ############################################################################## # The track mirrors the hub's main mm10 signal-&-peaks faceted composite # (cellBrowserMm10): 647 subtracks (641 bigWig, 6 bigNarrowPeak) from 9 datasets # - catlas-mouse-aging (234), catlas-mouse-brain (160), allen-basal-ganglia-atac # (131), olg-dyn-eae-multiome (28), mouse-lvcp-multiome (23, incl. the 6 peaks), # catlas-paired-tag (21), mouse-kidney-atac (19), olg-eae-ms (16) and # mouse-epi-juv-brain (15). That composite and its facet metadata are produced by # the hub build from the Cell Browser dataset tree: # -# cd /hive/users/mspeir/claude/cell-browser/all-tracks-hub-build +# cd $HOME/cellBrowser/ucsc/allTracksHub # python3 build_manifest.py # scan datasets -> manifest.tsv # python3 build_stanzas.py # manifest -> stanzas/mm10.trackDb.txt # # + meta/mm10.metadata.tsv # +# The build writes its output to $CBHUB_OUT, NOT next to the scripts (that dir is +# git-controlled). Default: /hive/users/mspeir/claude/cell-browser/all-tracks-hub-build +# # The per-track source files (abs_path column of manifest.tsv) are the files the # Cell Browser datasets already serve; nothing is regenerated here, only copied. ############################################################################## # 2. Copy the data files into place (bed dir, served via a /gbdb symlink) ############################################################################## # copySingleCellSignalsPeaksFiles.py copies every cellBrowserMm10 subtrack file # into /hive/data/genomes/mm10/bed/singleCellSignalsPeaks/<served-relpath>, # keeping each file's served relative path, and copies the composite's facet # metadata to <bed>/singleCellSignalsPeaks_metadata.tsv. The served subpath is # preserved on purpose: some coverage/peak basenames (e.g. MOL.bw, OPC.bw) repeat # across datasets, so a flat directory would clobber them. 89.6 GB, 647 files. # # scriptDir=$HOME/kent/src/hg/makeDb/scripts/singleCellSignalsPeaks # python3 $scriptDir/copySingleCellSignalsPeaksFiles.py --assembly mm10 --dry-run @@ -164,44 +169,47 @@ # be made there too. # # One copy of the crosswalks: build_stanzas reads celltype-crosswalks/ out of the kent # tree (XWALK_ROOT, overridable), not the copy in the hub-build dir. There used to be # two independent copies and they drifted -- a stale local copy silently reverted the # cell-class fix above while the kent copy looked correct. # # The curation of record is archived with the scripts: # scripts/singleCellSignalsPeaks/build_celltype_crosswalks.py # scripts/singleCellSignalsPeaks/celltype-crosswalks/ (per-collection crosswalks, # celltype-palette.tsv, celltype-class.tsv, sea-ad-celltype-crosswalk.tsv) # scripts/singleCellSignalsPeaks/celltype-crosswalks/paper-decodes/ (the paper-curated # decode tables, each row's `note` giving its source justification) # build_celltype_crosswalks.py rebuilds the crosswalks + palette from paper-decodes/. # The general hub machinery that consumes them (build_manifest.py, build_stanzas.py, -# hub_config.json) lives in the all-tracks-hub-build dir noted above, not in the kent -# tree -- it builds the whole super hub, not just this track. +# build_hub.py, hub_config.json) lives in the cellBrowser repo, not in the kent tree -- +# it builds the whole super hub, not just this track: +# https://github.com/ucscGenomeBrowser/cellBrowser/tree/develop/ucsc/allTracksHub +# build_stanzas finds the crosswalks archived here via XWALK_ROOT, so the hub and these +# native tracks always color a cell class the same. ############################################################################## # 5. Redundant allen-brain-science copies removed ############################################################################## # The Allen basal-ganglia dataset serves each unchanged per-cluster bigWig from all # four of its grouping directories (bg_regrouping_cl, bg_merge_D1_D2, # bg_merge_dorsal_ventral, bg_merge_D1_D2_dorsal_ventral). For 14 basenames those # four copies are byte-identical (md5-verified), so they rendered as four # indistinguishable subtracks and cost 4.4 GB of duplicated storage. build_stanzas # keeps the bg_regrouping_cl copy -- the cluster-level source the merges are built # from -- and skips the other three (allen_duplicate_skip; see -# all-tracks-hub-build/allen-duplicates.log for the list). Where the four copies +# $CBHUB_OUT/allen-duplicates.log for the list). Where the four copies # genuinely differ (14 other basenames, different aggregations and different file # sizes) all four are kept and told apart by the grouping-level variant descriptor. # The 42 redundant files were deleted from the bed dir; the list is kept alongside # them in deleted-redundant-allen-copies.txt. Only the one directory-level /gbdb # symlink exists, so nothing needed cleaning up there. # # xargs -a deleted-redundant-allen-copies.txt rm ############################################################################## # Counts ############################################################################## # 647 files resolved from the hub manifest (0 missing) and copied to the bed dir. # 42 byte-identical Allen copies were then removed, leaving 605 data files. 18 # QC-cluster tracks (doublet / low-quality / batch) are dropped when the .ra is # generated, leaving 587 subtracks in the track (581 signal + 6 peak). Facet