fe02c55d26614f1d5059c884fb2594f9b4d5ad50 mspeir Wed Aug 12 08:02:19 2026 -0700 singleCellSignalsPeaks: point at the hub build's new home, fix the file-copy script The hub build now lives in the cellBrowser repo under ucsc/allTracksHub, so the makeDocs link there instead of a personal work dir. HUB_BUILD in these scripts is the build's output dir, not its code; comments say so now, paths unchanged. copySingleCellSignalsPeaksFiles.py was silently copying nothing: it compared a whole line against "parent ", but stanzas are now indented and read "parent off". Now dedents, matches the first token, and refuses to run on zero subtracks. Dry runs give 925 hg38 / 587 mm10, none missing. refs #37914 diff --git src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt index 26f2f102543..61e8dcf0770 100644 --- src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt +++ src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt @@ -1,208 +1,216 @@ # mm10 singleCellSignalsPeaks track - 2026-07-22 Claude (mspeir) refs #37914 # The native mm10 "singleCellSignalsPeaks" faceted composite is the Genome # Browser version of the mm10 signal-&-peaks composite (cellBrowserMm10) of the -# UCSC Cell Browser all-tracks super hub (built under -# /hive/users/mspeir/claude/cell-browser/all-tracks-hub-build). It gathers the -# per-cell-type ATAC-seq signal (bigWig) and peak (bigNarrowPeak) tracks from the +# UCSC Cell Browser all-tracks super hub, whose build lives in the cellBrowser +# repo, ucsc/allTracksHub: +# https://github.com/ucscGenomeBrowser/cellBrowser/tree/develop/ucsc/allTracksHub +# +# It gathers the per-cell-type ATAC-seq signal (bigWig) and peak (bigNarrowPeak) tracks from the # single-cell ATAC datasets in the Cell Browser and re-parents them under one # faceted composite. It is the mm10 counterpart of the hg38 track of the same # name (see doc/hg38/singleCellSignalsPeaks.txt, Redmine #37820). Histone marks, # cCREs, RNA/expression, interactions, and the separately-faceted motor-neuron # and brain-spatial sets each live in their own hub composites and are NOT part # of this track. ############################################################################## # 1. Source data ############################################################################## # The track mirrors the hub's main mm10 signal-&-peaks faceted composite # (cellBrowserMm10): 647 subtracks (641 bigWig, 6 bigNarrowPeak) from 9 datasets # - catlas-mouse-aging (234), catlas-mouse-brain (160), allen-basal-ganglia-atac # (131), olg-dyn-eae-multiome (28), mouse-lvcp-multiome (23, incl. the 6 peaks), # catlas-paired-tag (21), mouse-kidney-atac (19), olg-eae-ms (16) and # mouse-epi-juv-brain (15). That composite and its facet metadata are produced by # the hub build from the Cell Browser dataset tree: # -# cd /hive/users/mspeir/claude/cell-browser/all-tracks-hub-build +# cd $HOME/cellBrowser/ucsc/allTracksHub # python3 build_manifest.py # scan datasets -> manifest.tsv # python3 build_stanzas.py # manifest -> stanzas/mm10.trackDb.txt # # + meta/mm10.metadata.tsv # +# The build writes its output to $CBHUB_OUT, NOT next to the scripts (that dir is +# git-controlled). Default: /hive/users/mspeir/claude/cell-browser/all-tracks-hub-build +# # The per-track source files (abs_path column of manifest.tsv) are the files the # Cell Browser datasets already serve; nothing is regenerated here, only copied. ############################################################################## # 2. Copy the data files into place (bed dir, served via a /gbdb symlink) ############################################################################## # copySingleCellSignalsPeaksFiles.py copies every cellBrowserMm10 subtrack file # into /hive/data/genomes/mm10/bed/singleCellSignalsPeaks/, # keeping each file's served relative path, and copies the composite's facet # metadata to /singleCellSignalsPeaks_metadata.tsv. The served subpath is # preserved on purpose: some coverage/peak basenames (e.g. MOL.bw, OPC.bw) repeat # across datasets, so a flat directory would clobber them. 89.6 GB, 647 files. # # scriptDir=$HOME/kent/src/hg/makeDb/scripts/singleCellSignalsPeaks # python3 $scriptDir/copySingleCellSignalsPeaksFiles.py --assembly mm10 --dry-run # python3 $scriptDir/copySingleCellSignalsPeaksFiles.py --assembly mm10 ############################################################################## # 3. Generate the trackDb .ra ############################################################################## # makeSingleCellSignalsPeaksRa.py reads the hub's mm10 stanzas, keeps the # cellBrowserMm10 subtracks, renames the composite to singleCellSignalsPeaks, # repoints every bigDataUrl at the local /gbdb copy, and writes the .ra with # group=regulation (ATAC signal/peaks sit with the ENCODE regulatory tracks). # The generator also: drops any subtrack whose source path is under a deprecated # "*.old/" dir, sets every subtrack "off" by default (users pick tracks from the # faceted selector), and assigns "priority" by cell class so same-class tracks # group together in the display. Labels/colors/facets are already resolved by # build_stanzas (see section 4); the generator carries them through unchanged. # # python3 $scriptDir/makeSingleCellSignalsPeaksRa.py --assembly mm10 # # https://github.com/ucscGenomeBrowser/kent/tree/master/src/hg/makeDb/scripts/singleCellSignalsPeaks ############################################################################## # 4. Labels, colors, and facets (build_stanzas.py) ############################################################################## # build_stanzas.py derives the display metadata rather than copying the source # hub's cryptic cluster labels. The same logic runs for hg38 (see # doc/hg38/singleCellSignalsPeaks.txt). # # Cell type: the source cluster label is cleaned and, for datasets with coded # cluster names, mapped to a canonical name via a paper-curated crosswalk # (celltype-crosswalks/.tsv; codes decoded from Li 2021 PMID 34616068, # Zhang 2022 PMID 36207411, Zhu 2021 PMID 33589836, and the Allen whole-brain # taxonomy). Redundant synonyms are merged, QC clusters (doublet / low-quality / # batch) are dropped, and commas become ';' (the faceted UI tokenizes cell # values on commas). The full cell type is a searchable table column (_Cell_type), # not a facet -- there are too many values. # # Cell class: each cell type is assigned one of ~22 broad classes # (celltype-crosswalks/celltype-class.tsv, built by build_celltype_crosswalks.py). # The class drives BOTH the track color (one colorblind-conscious palette, # celltype-palette.tsv, shared by hg38 and mm10 so a class is the same color on # both assemblies) and the subtrack priority. Cell class is the primary cell facet. # # Tissue / Life stage / Condition: filled per collection from hub_config.json # (tissue_from_path, collection_lifestage, collection_condition) plus dataset- # specific parsing in build_stanzas.py. SEA-AD, for example, gets its region # (seaad_MTG/PFC -> middle temporal gyrus / dorsolateral prefrontal cortex), # Aged life stage (donors 65-102), and ADNC neuropathology level (no AD / low / # intermediate / high) from the path + "ADNC" filename prefix; regions and # ADNC categories per Gabitto 2024 (PMID 39402379) and Hawrylycz 2024 (PMID # 39402332). # # longLabel: rebuilt from the harmonized facets as # [, ][, ] # [, ][, ] () # so the cryptic source short labels decode, e.g. "ADNC0 Astro" -> # "Astrocyte, ADNC 0 (no AD), middle temporal gyrus (SEA-AD Brain ATAC)", and # "LM.Macg.03" -> "Macrophage, muscle, 3 months (CATLAS Mouse Aging Brain)". The # CATLAS aging age-in-months (the ".NN" filename suffix) is the descriptor # -- kept in the label, not made a facet, so Life stage stays coarse (Adult/Aged) # and consistent across datasets. # # Variant descriptors (track_variant): the cell-type harmonization above drops # detail on purpose to keep the facets compact -- collapse_celltype strips trailing # cluster numbers, rollup_celltype strips parenthetical qualifiers, and # celltype_strip_suffix_regex strips peak-method suffixes. That detail is real, and # without it hundreds of genuinely different subtracks collapsed onto one label # (before this was added, hg38 had 925 subtracks sharing only 401 distinct # longLabels, mm10 629 sharing 359). The variant descriptors put the meaningful # distinctions back: how the peaks were called, which grouping level the track # belongs to, which cohort, and signal vs peaks. They compose, since a peak file # from one cohort needs both markers. # # Cluster identity (disambiguate_labels): any label still shared after the variant # descriptors is qualified with its source cluster code, which is unique and is # what the source paper calls the cluster. CATLAS mouse brain alone rolls # MSGA1..MSGA13 into one "Medial septum GABAergic neuron", and ITL4GL1/ITL5GL2 into # one "Cortical IT excitatory neuron" even though those encode different cortical # layers -- right for a facet, wrong for a label. Every longLabel is now unique on # both assemblies. # # shortLabel: rebuilt too (the source shortLabels were not in fact length-limited -- # they ran to 50 chars, carried underscores, ArchR filename tails and R-mangled # names like X1126_HY_PAL_STR_Folh1_1). Built from the harmonized cell type, # abbreviated word-by-word through a curated table (CT_ABBREV) only when it does # not fit, plus the variant token and, where the longLabel distinction would # otherwise be invisible, a compact facet token (SEA-AD region + ADNC level, # CATLAS aging age: "Astrocyte MTG A0", "Basophil 3mo"). Budget is 22 chars: there # is no hard limit in the tree (excluding this track, hg38's trackDb runs a median # of 18 and 54% of tracks exceed the classic 17), but the left label area crowds # past that and 22 leaves room for the distinguishing token. # # Spelling: known source misspellings are corrected before the plural/case merge so # a typo'd variant lands in the same group as its correctly-spelled twin instead of # becoming a second facet value ("Ventricular cardioyocyte" vs "Ventricular # cardiomyocyte" were two separate Cell_type values). See CELLTYPE_SPELLING. # The plural/case merge picks the singular form first and only then the most # frequent -- frequency alone made the facet inconsistent, choosing the singular # for 15 of 17 merged groups but the plural for Megakaryocytes/Oligodendrocytes. # # IMPORTANT, if you add to CELLTYPE_SPELLING: the misspelled strings are load-bearing # *keys* in the curated tables (celltype-class.tsv, and hub_config.json's # celltype_rollup_overrides / celltype_expansions / celltype_expansions_global / # celltype_tissue). Correcting a cell type without correcting those keys makes the # lookup miss silently, and the track loses its rollup, tissue, broad class and color. # That happened: correcting Broncial / Hematopoeitic / Syncitio / Glutaminergic, plus # canonicalizing Megakaryocytes to the singular, quietly cost 16 hg38 tracks their # Cell_class and color. The tables now normalize their keys on load (class_key, # _ct_lower_key, _ct_matchkey) so either spelling matches, and any cell type with no # broad class is reported in unclassified-celltypes.log plus counted in # facet-coverage.md rather than silently becoming "unknown". # # Cell classes: Nephron progenitor used to come out as class "Neural progenitor". The # decode tables carry a bare broad class of "Progenitor", and build_celltype_crosswalks # mapped that to "Neural progenitor" -- right for the radial glia and neuroblasts that # make up the rest of the class, wrong for kidney. Nephron progenitors are Six2+ cap # mesenchyme, so they are now classed Stromal (NON_NEURAL_PROGENITOR); the HTML legend # no longer lists them under Neural progenitor. Note celltype-class.tsv is hand-curated # (the union of the mm10 crosswalk classes with paper-decodes/hg38_ct_class.tsv), NOT # generated -- build_celltype_crosswalks.py does not write it, so a class change has to # be made there too. # # One copy of the crosswalks: build_stanzas reads celltype-crosswalks/ out of the kent # tree (XWALK_ROOT, overridable), not the copy in the hub-build dir. There used to be # two independent copies and they drifted -- a stale local copy silently reverted the # cell-class fix above while the kent copy looked correct. # # The curation of record is archived with the scripts: # scripts/singleCellSignalsPeaks/build_celltype_crosswalks.py # scripts/singleCellSignalsPeaks/celltype-crosswalks/ (per-collection crosswalks, # celltype-palette.tsv, celltype-class.tsv, sea-ad-celltype-crosswalk.tsv) # scripts/singleCellSignalsPeaks/celltype-crosswalks/paper-decodes/ (the paper-curated # decode tables, each row's `note` giving its source justification) # build_celltype_crosswalks.py rebuilds the crosswalks + palette from paper-decodes/. # The general hub machinery that consumes them (build_manifest.py, build_stanzas.py, -# hub_config.json) lives in the all-tracks-hub-build dir noted above, not in the kent -# tree -- it builds the whole super hub, not just this track. +# build_hub.py, hub_config.json) lives in the cellBrowser repo, not in the kent tree -- +# it builds the whole super hub, not just this track: +# https://github.com/ucscGenomeBrowser/cellBrowser/tree/develop/ucsc/allTracksHub +# build_stanzas finds the crosswalks archived here via XWALK_ROOT, so the hub and these +# native tracks always color a cell class the same. ############################################################################## # 5. Redundant allen-brain-science copies removed ############################################################################## # The Allen basal-ganglia dataset serves each unchanged per-cluster bigWig from all # four of its grouping directories (bg_regrouping_cl, bg_merge_D1_D2, # bg_merge_dorsal_ventral, bg_merge_D1_D2_dorsal_ventral). For 14 basenames those # four copies are byte-identical (md5-verified), so they rendered as four # indistinguishable subtracks and cost 4.4 GB of duplicated storage. build_stanzas # keeps the bg_regrouping_cl copy -- the cluster-level source the merges are built # from -- and skips the other three (allen_duplicate_skip; see -# all-tracks-hub-build/allen-duplicates.log for the list). Where the four copies +# $CBHUB_OUT/allen-duplicates.log for the list). Where the four copies # genuinely differ (14 other basenames, different aggregations and different file # sizes) all four are kept and told apart by the grouping-level variant descriptor. # The 42 redundant files were deleted from the bed dir; the list is kept alongside # them in deleted-redundant-allen-copies.txt. Only the one directory-level /gbdb # symlink exists, so nothing needed cleaning up there. # # xargs -a deleted-redundant-allen-copies.txt rm ############################################################################## # Counts ############################################################################## # 647 files resolved from the hub manifest (0 missing) and copied to the bed dir. # 42 byte-identical Allen copies were then removed, leaving 605 data files. 18 # QC-cluster tracks (doublet / low-quality / batch) are dropped when the .ra is # generated, leaving 587 subtracks in the track (581 signal + 6 peak). Facet # metadata rows match the subtracks 1:1, and all 587 longLabels are unique.