8bb883ed32e24a1007bf9f97ee1793a46bf47bd0 mspeir Thu Aug 27 10:00:43 2026 -0700 singleCellSignalsPeaks: classify the SEA-AD and Allen tracks, add BrainVar stages 72 hg38 SEA-AD and 89 mm10 allen-basal-ganglia tracks had lost their broad cell class and were drawing in dataset colours instead of the shared palette; both assemblies are now 0 unclassified. Fix is in the hub build (cellBrowser ucsc/allTracksHub). Adds an "Other glia" class, since "Other" was doing four unrelated jobs. BrainVar goes 4 -> 13 hg38 subtracks with the new stage-split pseudobulk. Scaling moves to "autoScale group" on the composite so selected subtracks share one scale, and every bigWig now carries a data range -- without one the 13 BrainVar tracks drew as flat lines against hgTracks' built-in 0:127. Description pages and makeDocs updated, including removing the stale claim that SEA-AD is coloured by its own subclass palette. refs #37914, refs #37820 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> diff --git src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt index 8e426b25a3c..69a7ad01e47 100644 --- src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt +++ src/hg/makeDb/doc/mm10/singleCellSignalsPeaks.txt @@ -71,36 +71,50 @@ # 4. Labels, colors, and facets (build_stanzas.py) ############################################################################## # build_stanzas.py derives the display metadata rather than copying the source # hub's cryptic cluster labels. The same logic runs for hg38 (see # doc/hg38/singleCellSignalsPeaks.txt). # # Cell type: the source cluster label is cleaned and, for datasets with coded # cluster names, mapped to a canonical name via a paper-curated crosswalk # (celltype-crosswalks/<collection>.tsv; codes decoded from Li 2021 PMID 34616068, # Zhang 2022 PMID 36207411, Zhu 2021 PMID 33589836, and the Allen whole-brain # taxonomy). Redundant synonyms are merged, QC clusters (doublet / low-quality / # batch) are dropped, and commas become ';' (the faceted UI tokenizes cell # values on commas). The full cell type is a searchable table column (_Cell_type), # not a facet -- there are too many values. # -# Cell class: each cell type is assigned one of ~22 broad classes +# Cell class: each cell type is assigned one of 23 broad classes # (celltype-crosswalks/celltype-class.tsv, built by build_celltype_crosswalks.py). # The class drives BOTH the track color (one colorblind-conscious palette, # celltype-palette.tsv, shared by hg38 and mm10 so a class is the same color on # both assemblies) and the subtrack priority. Cell class is the primary cell facet. # +# "Other glia" (88,160,88) was added because "Other" was doing four unrelated +# jobs. Of 40 tracks corpus-wide sitting in it: 14 real glia with nowhere else to +# go, 14 peripheral glia deliberately placed there (Schwann / enteric -- left +# alone), 5 that are not cell types at all (All cell types / All cells / +# Neuronal and glial -- aggregate peak calls, a separate defect NOT addressed), +# and 6 genuinely other. The new class takes the glial aggregate labels (Glia, +# Glial), olfactory ensheathing cells, and the five fly glia subtypes; Schwann and +# enteric glia stay under "Other". Named for symmetry with the existing +# "Other neuron". The color came from a grid search maximizing minimum dE against +# the 22 existing entries under normal, deuteranope and protanope simulation, +# restricted to the widest free hue band; nearest neighbours are Epithelial +# (dE 17) and Astrocyte (dE 20), both above the palette's own closest pair +# (Other vs Unknown, 12.7). Before this, Glia and Non-Neuronal were both grey. +# # Tissue / Life stage / Condition: filled per collection from hub_config.json # (tissue_from_path, collection_lifestage, collection_condition) plus dataset- # specific parsing in build_stanzas.py. SEA-AD, for example, gets its region # (seaad_MTG/PFC -> middle temporal gyrus / dorsolateral prefrontal cortex), # Aged life stage (donors 65-102), and ADNC neuropathology level (no AD / low / # intermediate / high) from the path + "ADNC<n>" filename prefix; regions and # ADNC categories per Gabitto 2024 (PMID 39402379) and Hawrylycz 2024 (PMID # 39402332). # # longLabel: rebuilt from the harmonized facets as # <cell type>[, <condition if not healthy>][, <tissue if a specific region>] # [, <extra>][, <variant>] (<dataset>) # so the cryptic source short labels decode, e.g. "ADNC0 Astro" -> # "Astrocyte, ADNC 0 (no AD), middle temporal gyrus (SEA-AD Brain ATAC)", and # "LM.Macg.03" -> "Macrophage, muscle, 3 months (CATLAS Mouse Aging Brain)". The @@ -175,42 +189,114 @@ # # The curation of record is archived with the scripts: # scripts/singleCellSignalsPeaks/build_celltype_crosswalks.py # scripts/singleCellSignalsPeaks/celltype-crosswalks/ (per-collection crosswalks, # celltype-palette.tsv, celltype-class.tsv, sea-ad-celltype-crosswalk.tsv) # scripts/singleCellSignalsPeaks/celltype-crosswalks/paper-decodes/ (the paper-curated # decode tables, each row's `note` giving its source justification) # build_celltype_crosswalks.py rebuilds the crosswalks + palette from paper-decodes/. # The general hub machinery that consumes them (build_manifest.py, build_stanzas.py, # build_hub.py, hub_config.json) lives in the cellBrowser repo, not in the kent tree -- # it builds the whole super hub, not just this track: # https://github.com/ucscGenomeBrowser/cellBrowser/tree/develop/ucsc/allTracksHub # build_stanzas finds the crosswalks archived here via XWALK_ROOT, so the hub and these # native tracks always color a cell class the same. +############################################################################## +# 4b. Two collections whose source hub stanzas are ignored +############################################################################## +# hub_config.json "collection_ignore_hub_stanzas" lists collections built as if they +# had no curated hub at all: the cell type comes from the manifest original_track_name +# (the source FILENAME) and the stanza is generated rather than copied. Normally the +# wrangler-written hub stanza is the better source and is preferred everywhere -- +# corpus-wide its shortLabel classifies 1245 tracks that the filename cannot -- so this +# is per-collection and must stay that way. +# +# Currently sea-ad-brain-atac and allen-basal-ganglia-atac. Both name their subtracks in +# a DIFFERENT scheme than every piece of curation written for them, and both failed in +# two independent ways: +# +# Labels. SEA-AD files are ADNC0L2-3IT.bw -> original_track_name "ADNC0 L2-3IT", but +# its hub stanzas are named "L2/3IT_ADNC:0" -- the ADNC token moves from the front to +# the back and the layer name loses its hyphen. So label_sub (^ADNC(\d+)), +# celltype_leading_strip_regex (which strips a LEADING ADNC token) and the keys of +# sea-ad-celltype-crosswalk.tsv (L2-3IT, L5-6NP, Lamp5-Lhx6, Sst-Chodl) ALL miss, and +# 72 hg38 cortical-layer tracks plus 89 mm10 allen-basal-ganglia tracks lost their +# broad cell class and their shared-palette color. Note the trailing " ADNC:0" is NOT +# what breaks it: lookup_class shortens by whitespace prefix, so "Lamp5 ADNC:0" -> +# "Lamp5" still resolves. The failures are the 9 spellings that differ between the two +# schemes (lookup_class("L2/3IT") is None; lookup_class("L2/3 IT") is Excitatory neuron). +# +# Scaling. Both hubs declare a bare "type bigWig" with no data range, so 184 hg38 and +# 89 mm10 tracks silently lost the default viewLimits bigWigInfo would have supplied. +# +# Add a collection here only when its hub stanzas are genuinely a different naming scheme +# from its curation. The fix for a merely MISSING crosswalk row is to add the row. + +############################################################################## +# 4c. Vertical scaling: autoScale group on the composite +############################################################################## +# The composite parent carries "autoScale group", and NO subtrack sets autoScale of its +# own. Every selected subtrack then shares one vertical scale, so two tracks drawn at the +# same locus can be compared directly; with per-track autoScale, tracks whose values +# differ by orders of magnitude both drew full height and looked equally strong. +# +# It has to sit on the parent. hgTracks groups by tdb->parent (wigTrack.c setMinMax +# accumulates each drawn track's limits into the shared parent tdb, and wigDrawPredraw +# reads them back), and hubCheck errAborts on an individual bigWig that declares +# "autoScale group" -- it "belongs in the parent composite stanza instead". 637 subtracks +# had inherited exactly that verbatim from their source hubs, so build_stanzas now drops +# autoScale via DROP_KEYS. A per-subtrack autoScale would also override the inherited +# parent setting. +# +# Limits come from preDrawAutoScale scanning the preDraw array, i.e. the data in the +# CURRENT WINDOW, not genome-wide -- so an outlying region elsewhere cannot flatten the +# view. That is why grouping beats fixed per-dataset viewLimits, which was considered and +# rejected: the spread of per-track maxima WITHIN one dataset reaches 20,117x (retina, +# 0.80 to 16,092) and the maxima are outlier-dominated (a cortex-dev coverage track peaks +# at 908,985), so a per-dataset ceiling would draw many real tracks flat. +# +# Grouping PER DATASET is not expressible here. autoScale group groups by tdb->parent and +# a subtrack inside a view has the view as its parent, but a faceted composite cannot have +# views: facetedCompositeUi walks tdb->subtracks exactly one level and builds <track>_sel +# cart vars from the immediate children (hgTrackUi.c), so with a view level the leaf +# bigWigs are never enumerated. None of the faceted composites in the tree use views. +# Accepted cost: selecting across datasets with different normalizations (BrainVar +# gene-activity maxing near 6 next to SEA-AD coverage near 72) draws the smaller tracks +# small. The description page says comparisons are most meaningful within a dataset. +# +# Separately, a bigWig whose type line carries no data range gets one from bigWigInfo, so +# the browser always has default viewLimits. Source hubs often set autoScale on the +# COMPOSITE PARENT and this build flattens subtracks into one composite without copying +# the parent, so the intent was lost in transit -- 232 subtracks corpus-wide had no range, +# no autoScale and no viewLimits, and BrainVar (max ~6) drew as a flat line against +# hgTracks' built-in 0:127. + ############################################################################## # 5. Redundant allen-brain-science copies removed ############################################################################## # The Allen basal-ganglia dataset serves each unchanged per-cluster bigWig from all # four of its grouping directories (bg_regrouping_cl, bg_merge_D1_D2, # bg_merge_dorsal_ventral, bg_merge_D1_D2_dorsal_ventral). For 14 basenames those # four copies are byte-identical (md5-verified), so they rendered as four # indistinguishable subtracks and cost 4.4 GB of duplicated storage. build_stanzas # keeps the bg_regrouping_cl copy -- the cluster-level source the merges are built # from -- and skips the other three (allen_duplicate_skip; see # $CBHUB_OUT/allen-duplicates.log for the list). Where the four copies # genuinely differ (14 other basenames, different aggregations and different file # sizes) all four are kept and told apart by the grouping-level variant descriptor. # The 42 redundant files were deleted from the bed dir; the list is kept alongside # them in deleted-redundant-allen-copies.txt. Only the one directory-level /gbdb # symlink exists, so nothing needed cleaning up there. # # xargs -a deleted-redundant-allen-copies.txt rm ############################################################################## # Counts ############################################################################## # 647 files resolved from the hub manifest (0 missing) and copied to the bed dir. # 42 byte-identical Allen copies were then removed, leaving 605 data files. 18 # QC-cluster tracks (doublet / low-quality / batch) are dropped when the .ra is # generated, leaving 587 subtracks in the track (581 signal + 6 peak). Facet # metadata rows match the subtracks 1:1, and all 587 longLabels are unique. +# Every subtrack has a broad cell class and a palette color (0 unclassified); +# the 89 allen-basal-ganglia-atac tracks were the ones fixed by section 4b.