5ad55adbb6a5cc72a393700130584aa87fef2c89
lrnassar
  Tue Jun 30 06:15:44 2026 -0700
varFreqs: add Top 3 source AFs to mouseOvers; audit excludes SGDP and SVatalog. refs #36642

Adds a Top 3 source AFs ranking to the varFreqsAffected and varFreqsBackground
mouseOvers. Alongside the pooled allele frequency, the mouseOver now lists the
three cohorts/arms with the highest per-source AF, formatted as
"Source (AF), Source (AF), Source (AF)". Disease cohorts with phenotype splits
carry the arm label (SPARK ASD, SCHEMA case, GREGoR unaffected); population
cohorts use the bare key. Per-population sub-ancestries are deliberately
excluded so a high sub-pop AF cannot crowd out actual project-level signals.

vcfToBigBed.py adds a top_n_source_afs helper, collects per-arm AFs into
affected_arm_afs / background_arm_afs, and emits two new fields
topAffectedSources and topBackgroundSources. AS schema field count 163 -> 165.

An AF-distribution sweep across all 28 source cohorts identified SGDP and
SVatalog as encoding allele counts per genotyped individual (small N, AF
defaults near 0.5), making their per-source AF unreliable for the ranking.
Adds a skip_top_ranking column (col 9) to databases.tsv, set to 1 for SGDP
and SVatalog, and gates the per-arm AF append in vcfToBigBed.py on this
flag. Both cohorts still contribute to pooled backgroundAC/AN/AF and still
appear in backgroundSources; they are only suppressed from the Top 3.

Description pages varFreqsAffected.html and varFreqsBackground.html document
the ranking; the latter also documents the SGDP/SVatalog exclusion. Build
documentation in varFreqs.txt is updated.

diff --git src/hg/makeDb/doc/hg38/varFreqs.txt src/hg/makeDb/doc/hg38/varFreqs.txt
index 52c925634a8..079576ea50f 100644
--- src/hg/makeDb/doc/hg38/varFreqs.txt
+++ src/hg/makeDb/doc/hg38/varFreqs.txt
@@ -1075,15 +1075,72 @@
 cp varFreqsAffected.bb   varFreqsAffected.bb.preAfPool.bak
 cp varFreqsBackground.bb varFreqsBackground.bb.preAfPool.bak
 # Build:
 python3 ~/kent/src/hg/makeDb/scripts/varFreqs/vcfToBigBed.py \
     --annotated-vcf merged.annotated.vcf.gz \
     --output-prefix varFreqs \
     --split-affected \
     --threads 8 \
     --work-dir /hive/data/genomes/hg38/bed/varFreqs/all
 # Spot-check (APOE rs429358, chr19:44908683-44908684 T>C):
 #   pre  : affectedAC=33238 affectedAF=0.181730 (max, GA4K-dominated)
 #                            backgroundAF=1.000000 (max, AllOfUs_OCE artifact)
 #   post : affectedAC=32782 affectedAN=213150 affectedAF=0.153798
 #                            backgroundAC=341065 backgroundAN=2751112
 #                            backgroundAF=0.123974
+
+##########
+# 2026-06-27 Lou (Claude)
+# Added Top 3 source AFs to the varFreqsAffected and varFreqsBackground
+# mouseOver, modeled loosely on gnomAD's PopMaxAF but ranking PROJECTS,
+# not populations. Motivation: variants common in one specific cohort
+# (e.g. the East-Asian founder allele rs4986893 at ~13% in ToMMo Japan
+# and ~10% in KOVA Korea) surface immediately, even when the pooled rate
+# across all contributing arms is much smaller.
+#
+# Implementation: vcfToBigBed.py now tracks per-arm AF contributions
+# alongside the existing pooled AC/AN sums and emits two new string
+# columns, topAffectedSources and topBackgroundSources, formatted as
+# "Source (AF), Source (AF), Source (AF)". For disease cohorts with a
+# phenotype split, the source label carries the arm (e.g. "SPARK ASD",
+# "SCHEMA case", "GREGoR unaffected"); for population cohorts the label
+# is the bare cohort key. Per-population sub-ancestries of population
+# cohorts (e.g. gnomAD HGDP+1kG continental groups) are deliberately
+# excluded so sub-populations don't crowd out actual project signals.
+# AS schema field count: 163 -> 165.
+#
+# Rebuild used the existing merged.annotated.vcf.gz (no re-merge needed);
+# only the Phase 1 per-cohort extract and Phase 2 BED build re-ran.
+cd /hive/data/genomes/hg38/bed/varFreqs/all
+cp varFreqsAffected.bb   varFreqsAffected.bb.preTop3.bak
+cp varFreqsBackground.bb varFreqsBackground.bb.preTop3.bak
+python3 ~/kent/src/hg/makeDb/scripts/varFreqs/vcfToBigBed.py \
+    --annotated-vcf merged.annotated.vcf.gz \
+    --output-prefix varFreqs \
+    --split-affected \
+    --threads 8 \
+    --work-dir /hive/data/genomes/hg38/bed/varFreqs/all
+
+#
+# 2026-06-29: re-rebuild after Top-3 source audit.
+# A broad AF-distribution sweep across all 28 source cohorts identified two
+# whose per-source AF was unreliable for the Top-3 ranking introduced on
+# 2026-06-27: SGDP and SVatalog both encode AC/AN per genotyped site (small
+# N producing AF=0.5 or AF=1.0 at most variants), so they otherwise
+# dominated the Top-3 list with meaningless values. Pooled AF is unaffected
+# since their AC/AN contributions are tiny relative to the larger cohorts.
+# Added a new "skip_top_ranking" column to databases.tsv (col 9); set to 1
+# for SGDP and SVatalog, 0 (or blank) for every other cohort. Modified
+# vcfToBigBed.py to skip the per-arm AF append into affected_arm_afs /
+# background_arm_afs when skip_top_ranking is set; the cohort still
+# contributes to the pooled AC/AN and still appears in backgroundSources.
+# Description page update: varFreqsBackground.html now documents this
+# exclusion in the "Top population sources by AF" section.
+cd /hive/data/genomes/hg38/bed/varFreqs/all
+cp varFreqsAffected.bb   varFreqsAffected.bb.preTopAudit.bak
+cp varFreqsBackground.bb varFreqsBackground.bb.preTopAudit.bak
+python3 ~/kent/src/hg/makeDb/scripts/varFreqs/vcfToBigBed.py \
+    --annotated-vcf merged.annotated.vcf.gz \
+    --output-prefix varFreqs \
+    --split-affected \
+    --threads 8 \
+    --work-dir /hive/data/genomes/hg38/bed/varFreqs/all