1de43b26fd18d5be3585829158d9b65944979094
lrnassar
  Wed Aug 5 17:19:10 2026 -0700
Harden popEVE build drivers per code review. refs #37950 refs #37791

- Add "set -o pipefail" to both build scripts so a bedToBigBed failure piped to tail is not
masked (runBuildDense.sh reported the previous file's size on failure).
- runBuild.sh (sparse) now writes popEve_sparse.bb instead of popEve.bb, so re-running it
cannot silently clobber the dense popEve.bb produced by runBuildDense.sh. Makedoc updated.

diff --git src/hg/makeDb/doc/hg38/popEve.txt src/hg/makeDb/doc/hg38/popEve.txt
index 8fdcd8b45aa..978a5b425eb 100644
--- src/hg/makeDb/doc/hg38/popEve.txt
+++ src/hg/makeDb/doc/hg38/popEve.txt
@@ -44,56 +44,57 @@
 cd /hive/data/genomes/hg38/bed/popEve
 # Full build (download already done): strand map, extract, color anchors, sort, convert,
 # filter to chrom.sizes, bigBed. See runBuild.sh in this directory.
 bash runBuild.sh
 
 # runBuild.sh steps, in order:
 #   1. NP_->strand map:
 #        hgsql hg38 -N -e "select distinct l.protAcc, g.chrom, g.strand from ncbiRefSeqLink l
 #          join ncbiRefSeq g on g.name=l.mrnaAcc where l.protAcc like 'NP\_%'"
 #        -> np_strand.tsv (prefer the strand on a primary chromosome)
 #   2. extract: zcat ...vcf.gz | extractPopEve.py > popEve_records.tsv
 #   3. color anchors: p0.5 / p99.5 of popEVE over all records -> anchors.txt
 #   4. sort -t$'\t' -k1,1 -k4,4n -S 4G -T sorttmp popEve_records.tsv > popEve_sorted.tsv
 #   5. vcfToPopEveHeatmap.py popEve_sorted.tsv np_strand.tsv popEve_raw.bed $LO $HI
 #   6. bedSort + awk filter to chrom.sizes -> popEve_filtered.bed
-#   7. bedToBigBed -type=bed12+ -tab -as=popEve_heatmap.as popEve_filtered.bed chrom.sizes popEve.bb
+#   7. bedToBigBed -type=bed12+ -tab -as=popEve_heatmap.as popEve_filtered.bed chrom.sizes popEve_sparse.bb
+#      (distinct name from the dense popEve.bb so re-running this sparse build cannot clobber it)
 # Scripts (all in ~/kent/src/hg/makeDb/scripts/popEve/): extractPopEve.py,
 #   vcfToPopEveHeatmap.py, popEve_heatmap.as, and the build drivers runBuild.sh (sparse /
 #   coordinate foundation) and runBuildDense.sh (dense rebuild).
 
 # Notes:
 #  - The genomic VCF contains only single-nucleotide-reachable substitutions (~6-9 of 19 per
 #    codon). The DENSE REBUILD below supersedes this with the full per-amino-acid matrix; the
 #    VCF-derived popEve_sorted.tsv remains the source of genomic codon coordinates and strand.
 #  - The heatmap renderer parses the score array (chopCommas, keeps trailing empty) and the
 #    label array (chopByCharRespectDoubleQuotesKeepEmpty, drops one trailing empty)
 #    differently. When the last cell (last row, last column) is empty the counts disagree and
 #    the track aborts; the converter sets a non-empty placeholder label on that one trailing
 #    cell (its score stays empty so the cell is uncolored). trailingFix count below.
 
 # Build results:
 #   records extracted: 66,400,085 (nan skipped: 15,156)
 #   color anchors: loAnchor=-5.742 hiAnchor=-2.287 (popEVE p0.5 / p99.5; median -3.358)
 #   proteins: 18,968 (18,343 distinct gene symbols; remainder are RefSeq isoforms)
 #   amino-acid positions: 10,114,809;  bases covered: 886,638,890
 #   strand: 0 inferred-vs-RefSeq mismatches; 25 proteins not in the RefSeq map (strand from
 #     coordinate inference); 0 with no strand signal; 0 dropped by chrom.sizes filter
 #   trailingFix (trailing empty cell relabeled): 15,555 proteins
 #   mouseover: HTML multi-line labels (<br>/<b>); component scores rounded to 3 dp;
 #     missing/nan components shown as NA
-#   popEve.bb size: 1,592,586,266 bytes
+#   popEve_sparse.bb size: 1,592,586,266 bytes
 
 # ============================================================================
 # DENSE REBUILD (2026-07-23) - full per-amino-acid matrix, per collaborator request
 # ============================================================================
 # The popEVE authors confirmed the per-transcript CSV download carries the COMPLETE
 # per-amino-acid matrix (all 19 substitutions per position), unlike the genomic VCF above
 # (single-nucleotide-reachable only, ~6 of 19 per codon). This rebuild takes the SCORES from
 # the per-transcript CSVs and REUSES the genomic codon coordinates, strand, and blocks
 # already derived from the VCF (popEve_sorted.tsv), producing dense heatmaps. Proteins present
 # in the VCF but absent from the (older, March 2025) CSV release fall back to sparse VCF scores.
 
 cd /hive/data/outside/popEve
 wget -nv https://data.evemodel.org/popeve/v1.1/downloads/popEVE_ukbb_20250312.zip
 #   2915714283 bytes, Last-Modified 2025-10-29
 mkdir -p popeve_csv