bdf448295c10bee74b6baa773f6733ce33171996
markd
  Thu Oct 1 20:56:37 2026 -0700
Store the ProCapNet minus strand negated, and fix group autoscale. refs #35528

Two of the three problems Jairo found on the composite.

The negate control did not work on ProCapNet. Its minus-strand files held
positive values and were flipped at display time with trackDb negateValues. The
composite's control sets one shared value, which replaced the per-track setting
and sent both strands the same way, with no route back to the default short of a
cart reset. encode4ProCap was fine because ENCODE publishes its minus strand
negative already. So put the sign in the data: proCapNetPredToFixedStep gains
--negate, proCapNetPredBuild passes it for the minus strand, and negateValues is
gone from the stanzas. The twelve existing files were rewritten rather than
rebuilt, since the downloads had been deleted; each was checked against its
original for identical nBasesCovered, a mirrored value range and exactly negated
values. The originals are kept in previousPred, about 197 GB, until QA is done.

Group autoscale on the composite did nothing, which was my error: the multiWig
containers and the contribution leaves each carried autoScale on, and the child
setting wins. The wiggle settings now live only on the composite.

The third, an empty row when both strands of a cell line are deselected, is
#38441 and is not fixed here.

Note in both makedocs and in the generator that this arrangement only partly
works and is expected to become a superTrack of multiWig overlays: under a
composite, hgTrackUi lists leaves rather than containers, so an overlay gets no
inline config block and leaves an empty row when deselected.

diff --git src/hg/makeDb/doc/hg38/transcriptionStart.txt src/hg/makeDb/doc/hg38/transcriptionStart.txt
index 806dd58a709..4d27b960c4f 100644
--- src/hg/makeDb/doc/hg38/transcriptionStart.txt
+++ src/hg/makeDb/doc/hg38/transcriptionStart.txt
@@ -1,18 +1,28 @@
 # Transcription start sites: ENCODE 4 PRO-cap and ProCapNet, #35528, Claude
 Thu Sep 17 2026 (Claude/markd)
 
+# STATUS: the composite only partly works, and the plan is to replace it with a
+# superTrack holding the multiWig overlays directly, the way fantom5 does.  A
+# multiWig under a composite draws correctly and the composite's wiggle controls
+# apply, but hgTrackUi lists descendant leaves rather than containers, so the
+# overlays get no inline config block and turning off both strands of a cell line
+# leaves an empty row where the overlay was (#38441).  Under a superTrack each
+# overlay is a track in its own right, with its own full configuration page and
+# no flattening.  The cost is losing the subtrack matrix and the sample class
+# filter.
+
 # Scripts are in ~/kent/src/hg/makeDb/outside/proCapNet.  The cell lines, their
 # ENCODE accessions and their colors are in proCapNetExperiments.tsv there.
 
 ##############################################################################
 # ProCapNet predictions and sequence-contribution scores
 ##############################################################################
 
 # The predictions are 24 GB per file and are re-encoded, so they are fetched to
 # a working area outside the track directory.  The contribution scores are used
 # unaltered and go straight to their final home.
 
 mkdir -p /hive/data/outside/proCapNet/hg38 /hive/data/genomes/hg38/bed/proCapNet
 cd /hive/data/genomes/hg38/bed/proCapNet
 ~/kent/src/hg/makeDb/outside/proCapNet/proCapNetDownload hg38 \
     ~/kent/src/hg/makeDb/outside/proCapNet/proCapNetExperiments.tsv \
@@ -21,56 +31,90 @@
 
 # Re-encode the predictions from one bedGraph interval per base into fixedStep
 # sections, which cuts the size by about a third without changing a value, and
 # drop the bases holding a literal NaN.  Those NaNs are all in unresolved (N)
 # reference sequence and make the track's autoScale and summary statistics NaN.
 ~/kent/src/hg/makeDb/outside/proCapNet/proCapNetPredBuild \
     /hive/data/genomes/hg38/chrom.sizes /hive/data/outside/proCapNet/hg38/pred pred 6
 # all 12 files report the same counts:
 #   24 chroms, 3088269832 bases, 2924001250 with data,
 #   164268582 dropped as no-data or NaN   (5.3%)
 # That is more than the 150610728 gap bases on the primary chromosomes, because
 # no prediction was made where most of a 2114 bp window was unresolved, which
 # also takes out sequence flanking each gap.
 # about 24 GB in, 15.7 to 15.9 GB out per file
 
+# proCapNetPredBuild writes the minus strand negated, so it draws below the
+# baseline from its own values.  Doing it in the data rather than with the
+# trackDb negateValues setting is what makes the composite's own negate control
+# work: that control sets one shared value for the composite, which replaced the
+# per-track negateValues and sent both strands the same way, with no way back to
+# the default short of a cart reset.  encode4ProCap never had the problem,
+# because ENCODE already publishes its minus strand negative.
+
 # confirm the summary statistics are numbers again, they were nan before
 bigWigInfo pred/K562.proCapNet.pos.bw | egrep 'basesCovered|mean|std'
 #   basesCovered: 2,924,001,250
 #   mean: 0.019478
 #   std: 0.300502
 
 # spot check that the re-encoding is lossless, comparing random windows of the
 # converted file against the download
 python3 - <<'PYEOF'
 import numpy as np, pyBigWig, random
 src = pyBigWig.open("/hive/data/outside/proCapNet/hg38/pred/K562.pos.bigWig")
 out = pyBigWig.open("pred/K562.proCapNet.pos.bw")
 random.seed(7)
 bad = 0
 for chrom in ("chr1", "chr7", "chr14", "chr21", "chrX"):
     for _ in range(3):
         s = random.randrange(0, src.chroms()[chrom] - 200000)
         a = src.values(chrom, s, s + 200000, numpy=True)
         b = out.values(chrom, s, s + 200000, numpy=True)
         na, nb = np.isnan(a), np.isnan(b)
         if not (na == nb).all() or not (a[~na] == b[~nb]).all():
             bad += 1
 print("mismatched windows:", bad)
 PYEOF
 # mismatched windows: 0
 
+##############################################################################
+# negating the minus strand, after the fact
+##############################################################################
+
+# The prediction files were first built with the minus strand positive and
+# flipped at display time with the trackDb negateValues setting.  That broke the
+# composite's negate control, as described above, so the twelve minus-strand
+# files were rewritten with negated values rather than rebuilt from the
+# downloads, which had already been deleted:
+#
+#   for db in hg38 hs1 ; do
+#       base=/hive/data/genomes/$db/bed/proCapNet
+#       mkdir -p $base/previousPred
+#       ls $base/pred/*.neg.bw | while read f ; do
+#           ~/kent/src/hg/makeDb/outside/proCapNet/proCapNetPredToFixedStep \
+#               --negate /hive/data/genomes/$db/chrom.sizes $f $base/negated/$(basename $f)
+#       done
+#   done
+#
+# Each output was checked against its input: same nBasesCovered, value range
+# mirrored, sampled values exactly negated.  The originals were then moved to
+# previousPred/ and the negated files put in their place, so /gbdb needs no new
+# symlinks.  previousPred/ is about 197 GB and can be removed once the track has
+# been through QA.  A rebuild from the downloads does not need any of this:
+# proCapNetPredBuild negates the minus strand itself.
+
 ##############################################################################
 # ENCODE 4 PRO-cap
 ##############################################################################
 
 mkdir -p /hive/data/genomes/hg38/bed/encode4ProCap
 cd /hive/data/genomes/hg38/bed/encode4ProCap
 
 # Resolve the portal metadata: for each of the six experiments take only the
 # plus and minus strand signal of unique reads files belonging to that
 # experiment's default analysis, which drops files superseded by a later
 # reprocessing.
 ~/kent/src/hg/makeDb/outside/proCapNet/proCapNetEncodeMeta \
     ~/kent/src/hg/makeDb/outside/proCapNet/proCapNetExperiments.tsv encodeFiles.tsv
 #   A673    ENCSR046BCI  4 files
 #   Caco-2  ENCSR100LIJ  4 files