af9d8fe39e88f7b7cec3792ea797dab44f1416b0 max Tue May 19 04:36:02 2026 -0700 varFreqs: rebuild varFreqsAll with WBBC/TPMI/ChinaMAP/GenomeIndia, drop IndiGen, harden build pipeline Rebuilds /gbdb/hg38/varFreqs/_all/varFreqsAll.bb to fold in the four new subtracks registered earlier in May (WBBC 78.6M, TPMI 672k, ChinaMAP 147M, GenomeIndia 130M) and to drop IndiGen, which ships only a VRT bit and contributed an always-empty AC/AF column. New bb is 47 GB / 147 fields / 1.34 billion items (was 44 GB / 133 / 1.22B). Two pipeline fixes were necessary mid-rebuild: - bcftools 1.22 csq is stricter than earlier versions. Added --unify-chr-names chr,-,chr (Ensembl GFF3 uses bare "1" while merged VCF + FASTA use "chr1") and --force (5 SCHEMA alt contigs end up in the merge but aren't annotated in the GFF3) to the csq invocation in mergeAndAnnotate.sh. Four follow-up cleanups to the build scripts (no track change, just safer next rebuild): - mergeAndAnnotate.sh now reads VCF paths directly from databases.tsv in both the per-VCF strip+norm step and the merge step. The previous "files.txt + find normalized/" model could silently re-merge stale norm cache entries after a database was dropped from databases.tsv. - vcfToBigBed.py concat step streams sort stdout straight to disk instead of capture_output=True, which buffered the whole sorted chromosome (~24 GB for chr1) in Python RAM. - vcfToBigBed.py generate_trackdb_fragment() now emits the three customizations that used to have to be added on top of the auto-fragment by hand: filterType.consequence multipleListOr, the expanded consequence buckets (3_prime_utr, 5_prime_utr, non_coding, others), and skipEmptyFields on. - trackDb/human/varFreqs.ra updated to match the new bb columns (WBBC/TPMI/ChinaMAP/GenomeIndia AC+AF filters, WBBC 4-region population filters, IndiGen filter removed). refs #36642 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> diff --git src/hg/makeDb/trackDb/human/varFreqs.ra src/hg/makeDb/trackDb/human/varFreqs.ra index c6509b2a780..9f22710443c 100644 --- src/hg/makeDb/trackDb/human/varFreqs.ra +++ src/hg/makeDb/trackDb/human/varFreqs.ra @@ -5,71 +5,69 @@ type bed 12 visibility hide superTrack on track varFreqsAll shortLabel All Databases Combined longLabel Variant Frequencies: All Databases Combined with Consequence Annotations type bigBed 9 + parent varFreqs on bigDataUrl /gbdb/$D/varFreqs/_all/varFreqsAll.bb visibility pack itemRgb on maxWindowToDraw 5000000 priority 0.1 mouseOver <b>Var:</b> $name<br><b>AA change:</b> $aaChange<br><b>Var type:</b> $varType<br><b>Conseq:</b> $consequence<br><b>Max AF:</b> $maxAF<br><b>Total AC:</b> $totalAC<br><b>Sources:</b> $sources + # Source database filter + filterValues.sources AllOfUs|AllOfUs,SPARK|SPARK WES,SFARI_WGS|SFARI WGS,GenomeAsia|GenomeAsia SNVs,GenomeAsiaIndel|GenomeAsia Indels,NPM|NPM Singapore,KOVA|KOVA Korea,ToMMo|ToMMo Japan,FinnGen|FinnGen Finland,Saudi|Saudi,SweGen|SweGen Sweden,TOPMed|TOPMed,ABraOM|ABraOM Brazil,ALFA|ALFA,MGRB|MGRB Australia,HRC|HRC,MexBB|Mexico Biobank,SGDP|SGDP,HGDP1kG|gnomAD HGDP+1kG,GREGoR|GREGoR,SCHEMA|SCHEMA,GA4K|GA4K PacBio LR,CoLoRSdb|CoLoRSdb PacBio LR,SVatalog|SVatalog 101 10XG SR,Tishkoff180|Tishkoff 180 African WGS,WBBC|WBBC China,TPMI|TPMI Taiwan,ChinaMAP|China ChinaMAP,GenomeIndia|GenomeIndia 9.7k WGS + filterType.sources multipleListOr + filterLabel.sources Source Database # Variant type and consequence filters filterValues.varType SNV|SNV,INS|Insertion,DEL|Deletion,MNV|MNV filterLabel.varType Variant Type filterValues.consequence missense|Missense,synonymous|Synonymous,stop_gained|Stop Gained,frameshift|Frameshift,splice_donor|Splice Donor,splice_acceptor|Splice Acceptor,intron|Intron,3_prime_utr|3' UTR,5_prime_utr|5' UTR,non_coding|Non-coding,.|Intergenic,others|Other filterType.consequence multipleListOr filterLabel.consequence Consequence - # Source database filter - filterValues.sources AllOfUs|AllOfUs,SPARK|SPARK WES,SFARI_WGS|SFARI WGS,GenomeAsia|GenomeAsia SNVs,GenomeAsiaIndel|GenomeAsia Indels,KOVA|KOVA Korea,ToMMo|ToMMo Japan,IndiGen|IndiGenomes India,GenomeIndia|GenomeIndia 9.7k WGS,FinnGen|FinnGen Finland,Saudi|Saudi,SweGen|SweGen Sweden,TOPMed|TOPMed,ABraOM|ABraOM Brazil,ALFA|ALFA,MGRB|MGRB Australia,HRC|HRC,MexBB|Mexico Biobank,SGDP|SGDP,HGDP1kG|gnomAD HGDP+1kG,GREGoR|GREGoR,SCHEMA|SCHEMA,GA4K|GA4K PacBio LR,CoLoRSdb|CoLoRSdb PacBio LR,SVatalog|SVatalog 101 10XG SR,Tishkoff180|Tishkoff 180 African WGS,NPM|NPM Singapore - filterType.sources multipleListOr - filterLabel.sources Source Database # Length filters filterByRange.refLen on filterLabel.refLen Reference Length filterByRange.altLen on filterLabel.altLen Alternate Length filterByRange.varLen on filterLabel.varLen Length Change # Max AF filter filterByRange.maxAF on filterLabel.maxAF Max Allele Frequency filterLimits.maxAF 0:1 # Total AC filter filterByRange.totalAC on filterLabel.totalAC Total Allele Count (all databases) # Per-database AF filters filterByRange.AllOfUsAF on filterLabel.AllOfUsAF AllOfUs AF filterByRange.SPARKAF on filterLabel.SPARKAF SPARK WES AF filterByRange.SFARI_WGSAF on filterLabel.SFARI_WGSAF SFARI WGS AF filterByRange.GenomeAsiaAF on filterLabel.GenomeAsiaAF GenomeAsia SNVs AF filterByRange.GenomeAsiaIndelAF on filterLabel.GenomeAsiaIndelAF GenomeAsia Indels AF filterByRange.KOVAAF on filterLabel.KOVAAF KOVA Korea AF filterByRange.ToMMoAF on filterLabel.ToMMoAF ToMMo Japan AF - filterByRange.IndiGenAF on - filterLabel.IndiGenAF IndiGenomes India AF filterByRange.FinnGenAF on filterLabel.FinnGenAF FinnGen Finland AF filterByRange.SaudiAF on filterLabel.SaudiAF Saudi AF filterByRange.SweGenAF on filterLabel.SweGenAF SweGen Sweden AF filterByRange.TOPMedAF on filterLabel.TOPMedAF TOPMed AF filterByRange.ABraOMAF on filterLabel.ABraOMAF ABraOM Brazil AF filterByRange.ALFAAF on filterLabel.ALFAAF ALFA AF filterByRange.MGRBAF on filterLabel.MGRBAF MGRB Australia AF filterByRange.HRCAF on @@ -82,49 +80,53 @@ filterLabel.HGDP1kGAF gnomAD HGDP+1kG AF (4k cohort) filterByRange.GREGoRAF on filterLabel.GREGoRAF GREGoR AF filterByRange.SCHEMAAF on filterLabel.SCHEMAAF SCHEMA AF filterByRange.GA4KAF on filterLabel.GA4KAF GA4K PacBio LR AF filterByRange.CoLoRSdbAF on filterLabel.CoLoRSdbAF CoLoRSdb PacBio LR AF filterByRange.SVatalogAF on filterLabel.SVatalogAF SVatalog 101 10XG SR AF filterByRange.Tishkoff180AF on filterLabel.Tishkoff180AF Tishkoff 180 African WGS AF filterByRange.NPMAF on filterLabel.NPMAF NPM Singapore AF + filterByRange.WBBCAF on + filterLabel.WBBCAF WBBC China AF + filterByRange.TPMIAF on + filterLabel.TPMIAF TPMI Taiwan AF + filterByRange.ChinaMAPAF on + filterLabel.ChinaMAPAF China ChinaMAP AF filterByRange.GenomeIndiaAF on filterLabel.GenomeIndiaAF GenomeIndia 9.7k WGS AF # Per-database AC filters filterByRange.AllOfUsAC on filterLabel.AllOfUsAC AllOfUs AC filterByRange.SPARKAC on filterLabel.SPARKAC SPARK WES AC filterByRange.SFARI_WGSAC on filterLabel.SFARI_WGSAC SFARI WGS AC filterByRange.GenomeAsiaAC on filterLabel.GenomeAsiaAC GenomeAsia SNVs AC filterByRange.GenomeAsiaIndelAC on filterLabel.GenomeAsiaIndelAC GenomeAsia Indels AC filterByRange.KOVAAC on filterLabel.KOVAAC KOVA Korea AC filterByRange.ToMMoAC on filterLabel.ToMMoAC ToMMo Japan AC - filterByRange.IndiGenAC on - filterLabel.IndiGenAC IndiGenomes India AC filterByRange.FinnGenAC on filterLabel.FinnGenAC FinnGen Finland AC filterByRange.SaudiAC on filterLabel.SaudiAC Saudi AC filterByRange.SweGenAC on filterLabel.SweGenAC SweGen Sweden AC filterByRange.TOPMedAC on filterLabel.TOPMedAC TOPMed AC filterByRange.ABraOMAC on filterLabel.ABraOMAC ABraOM Brazil AC filterByRange.ALFAAC on filterLabel.ALFAAC ALFA AC filterByRange.MGRBAC on filterLabel.MGRBAC MGRB Australia AC filterByRange.HRCAC on @@ -137,30 +139,36 @@ filterLabel.HGDP1kGAC gnomAD HGDP+1kG AC (4k cohort) filterByRange.GREGoRAC on filterLabel.GREGoRAC GREGoR AC filterByRange.SCHEMAAC on filterLabel.SCHEMAAC SCHEMA AC filterByRange.GA4KAC on filterLabel.GA4KAC GA4K PacBio LR AC filterByRange.CoLoRSdbAC on filterLabel.CoLoRSdbAC CoLoRSdb PacBio LR AC filterByRange.SVatalogAC on filterLabel.SVatalogAC SVatalog 101 10XG SR AC filterByRange.Tishkoff180AC on filterLabel.Tishkoff180AC Tishkoff 180 African WGS AC filterByRange.NPMAC on filterLabel.NPMAC NPM Singapore AC + filterByRange.WBBCAC on + filterLabel.WBBCAC WBBC China AC + filterByRange.TPMIAC on + filterLabel.TPMIAC TPMI Taiwan AC + filterByRange.ChinaMAPAC on + filterLabel.ChinaMAPAC China ChinaMAP AC filterByRange.GenomeIndiaAC on filterLabel.GenomeIndiaAC GenomeIndia 9.7k WGS AC # Population-specific AF filters # AllOfUs local-ancestry populations # NB: these are local-ancestry-stratified frequencies (per-position, per-haplotype-class), # NOT the AllOfUs paper's global Rye ancestry categories. See varFreqs.html for details. filterByRange.AllOfUsAF_AFR on filterLabel.AllOfUsAF_AFR AllOfUs African AF (local ancestry) filterByRange.AllOfUsAF_AMR on filterLabel.AllOfUsAF_AMR AllOfUs Indigenous American AF (local ancestry) filterByRange.AllOfUsAF_EAS on filterLabel.AllOfUsAF_EAS AllOfUs East Asian AF (local ancestry) filterByRange.AllOfUsAF_EUR on filterLabel.AllOfUsAF_EUR AllOfUs European AF (local ancestry) filterByRange.AllOfUsAF_OCE on @@ -265,30 +273,47 @@ filterByRange.GREGoRAC_UNK on filterLabel.GREGoRAC_UNK GREGoR Unknown AC # NPM Singapore ancestry groups filterByRange.NPMAF_Chinese on filterLabel.NPMAF_Chinese NPM Singapore Chinese AF filterByRange.NPMAF_Malay on filterLabel.NPMAF_Malay NPM Singapore Malay AF filterByRange.NPMAF_Indian on filterLabel.NPMAF_Indian NPM Singapore Indian AF filterByRange.NPMAC_Chinese on filterLabel.NPMAC_Chinese NPM Singapore Chinese AC filterByRange.NPMAC_Malay on filterLabel.NPMAC_Malay NPM Singapore Malay AC filterByRange.NPMAC_Indian on filterLabel.NPMAC_Indian NPM Singapore Indian AC + # WBBC China populations + filterByRange.WBBCAF_North on + filterLabel.WBBCAF_North WBBC North Han AF + filterByRange.WBBCAF_Central on + filterLabel.WBBCAF_Central WBBC Central Han AF + filterByRange.WBBCAF_South on + filterLabel.WBBCAF_South WBBC South Han AF + filterByRange.WBBCAF_Lingnan on + filterLabel.WBBCAF_Lingnan WBBC Lingnan Han AF + filterByRange.WBBCAC_North on + filterLabel.WBBCAC_North WBBC North Han AC + filterByRange.WBBCAC_Central on + filterLabel.WBBCAC_Central WBBC Central Han AC + filterByRange.WBBCAC_South on + filterLabel.WBBCAC_South WBBC South Han AC + filterByRange.WBBCAC_Lingnan on + filterLabel.WBBCAC_Lingnan WBBC Lingnan Han AC skipEmptyFields on track allofus shortLabel AllOfUs v7 245k WGS longLabel Variant Frequencies: AllOfUs v7 - 245k WGS, local-ancestry-stratified, AC>=20 type vcfTabix parent varFreqs on bigDataUrl /gbdb/$D/varFreqs/_allofus/allOfUs.locAncFreq.vcf.gz dataVersion V7 visibility hide tableBrowser off priority 0.5 #track me #shortLabel Regeneron Million Exomes 983k WES