af9a5b388259e680dd34bc47b2cad4ff6e3d162f lrnassar Sat Jun 13 03:00:51 2026 -0700 varFreqs: pre-release polish from comprehensive sanity check. * Sync the new combined-track shortLabels into the four description pages: "Affected/Case Individuals" -> "Disease cohorts" and "Population + Unaffected" -> "Population reference" (matches the trackdb shortLabels users now see). * Add a paragraph in the supertrack Methods section describing the pooled affectedAF / backgroundAF formulation (sum AC / sum AN) and the default_an configuration that handles AF-only cohorts. * Update the in-track Methods paragraphs on varFreqsAffected.html and varFreqsBackground.html: replace "summed/maximized" with "pooled". * Fix supertrack table downloadability column to match the underscore-prefix convention: allofus "Yes" -> "No" (description page already says license restricted); gregor "No" -> "Yes" (description page says VCF is on our download server, and the gbdb path is not underscore-prefixed). * Add a 2026-06-12 makedoc section documenting the pooled-AF rebuild, the default_an mechanism, the new affectedAN/backgroundAN columns, the before/after spot-check at APOE rs429358, and the build commands. refs #36642 diff --git src/hg/makeDb/trackDb/human/varFreqs.html src/hg/makeDb/trackDb/human/varFreqs.html index 4bfa6e2345d..3c715c1b35f 100644 --- src/hg/makeDb/trackDb/human/varFreqs.html +++ src/hg/makeDb/trackDb/human/varFreqs.html @@ -20,47 +20,55 @@ Their VCF versions below show only the isolate frequency per variant. </p> <p>Please contact us (<a href="mailto:genome@soe.ucsc.edu">genome@soe.ucsc.edu</a><!-- above address is genome at soe.ucsc.edu -->) if you know of a project that we should add. So far, Regeneron's Million Exomes and Mexico City Studies (request rejected) and Taiwan Biobank (pending). </p> <h2>Combined Tracks</h2> <p> Three combined tracks merge variants from the individual subtracks into single bigBed files with predicted protein consequences and cross-database filtering. All three use the same filter conventions (variant type, consequence, source database, allele frequency, allele count, and per-database AF/AC). </p> <ul> - <li><a href="hgTrackUi?g=varFreqsAffected"><b>Affected/Case Individuals</b></a> — + <li><a href="hgTrackUi?g=varFreqsBackground"><b>Population reference</b></a> — the + default summary view: variants seen in the population reference cohorts (gnomAD + HGDP+1kG, TOPMed, ALFA, HRC and the national WGS projects) and in the + unaffected/control arms of the disease cohorts. Excludes the genotyping-array + cohorts.</li> + <li><a href="hgTrackUi?g=varFreqsAffected"><b>Disease cohorts</b></a> — variants seen in the affected or case arm of five disease-study cohorts (SFARI SPARK WES and WGS autism probands, SCHEMA schizophrenia cases, GREGoR affected, GA4K - rare-disease). Each variant also carries its frequency in the background, so - case-enriched variants can be isolated.</li> - <li><a href="hgTrackUi?g=varFreqsBackground"><b>Population + Unaffected</b></a> — - the matched background: variants seen in the population reference cohorts (gnomAD - HGDP+1kG, TOPMed, ALFA, HRC and the national WGS projects) and in the - unaffected/control arms of the disease cohorts. Showing this together with the - Affected track lets you compare case versus background frequency across a gene. Both - tracks exclude the genotyping-array cohorts.</li> + rare-disease). Each variant also carries its background frequency, so case-enriched + variants can be isolated by filtering Background AF.</li> <li><a href="hgTrackUi?g=varFreqsArray"><b>Genotyping Array Databases Combined</b></a> — 14.7 million variants from three array cohorts (TPMI Taiwan, Mexico Biobank, UK Biobank imputed). Kept separate because chip data has different per-variant confidence than sequencing.</li> </ul> +<p> +On the Disease and Population reference tracks, <b>Affected AF</b> and <b>Background AF</b> +are pooled across contributing cohort arms (sum of allele counts divided by sum of allele +numbers), not the maximum across arms, so the displayed frequency matches the carrier-count +scale and a small cohort with a high local frequency does not dominate the value. See the +"Pooled allele frequency" section on each combined track's description page for +which cohorts contribute to the pool numerator and denominator. +</p> + <h3>Consequence filter — the "Other" bucket</h3> <p> All three combined tracks share the same Consequence filter (Missense, Synonymous, Stop Gained, Frameshift, Splice Donor, Splice Acceptor, Intron, 3' UTR, 5' UTR, Non-coding, Intergenic, Other). The filter uses OR logic across the comma-separated consequence tokens on each variant: a variant tagged <code>stop_gained,frameshift</code> is selected by either the "Stop Gained" or the "Frameshift" filter. The "Other" bucket catches the less common <a href="http://www.sequenceontology.org/" target="_blank">Sequence Ontology</a> consequence terms emitted by <code>bcftools csq</code> that don't fit the named buckets above. Examples include <code>splice_region</code> (variant near a splice site but outside the canonical donor/acceptor), <code>start_lost</code> / <code>stop_lost</code> (variant disrupts the start codon or replaces the stop codon with a coding amino acid), <code>stop_retained</code> (variant changes the stop codon but keeps it a stop), <code>inframe_insertion</code> / <code>inframe_deletion</code> (in-frame indel that adds or @@ -70,65 +78,65 @@ </p> <h3>Available Datasets</h3> <table class="stdTbl"> <tr> <th>Database</th> <th>Region</th> <th>N</th> <th>Data Type</th> <th>Cohort</th> <th>Sub-populations</th> <th>Downloadable from UCSC</th> </tr> <tr> - <td><a href="hgTrackUi?g=varFreqsAffected">Affected/Case Individuals</a></td> + <td><a href="hgTrackUi?g=varFreqsAffected">Disease cohorts</a></td> <td>Sequencing-based disease cohorts</td> <td>~130k</td> <td>WGS/WES/long-read</td> <td>Affected/case arms of SFARI SPARK WES/WGS, SCHEMA, GREGoR, GA4K</td> <td>Affected/case AF and AC; background AF for contrast</td> <td>No</td> </tr> <tr> - <td><a href="hgTrackUi?g=varFreqsBackground">Population + Unaffected</a></td> + <td><a href="hgTrackUi?g=varFreqsBackground">Population reference</a></td> <td>Sequencing-based, population + unaffected</td> <td>~1.5mil</td> <td>WGS/WES/long-read</td> <td>Population cohorts + unaffected/control arms</td> <td>Background AF and AC; per-cohort and ancestry breakdowns</td> <td>No</td> </tr> <tr> <td><a href="hgTrackUi?g=varFreqsArray">Genotyping Array Databases Combined</a></td> <td>TPMI, MexBB, UKBB</td> <td>~530k</td> <td>Array / imputed</td> <td>14.7M variants</td> <td>—</td> <td>No</td> </tr> <tr> <td><a href="hgTrackUi?g=allofus">AllOfUs v7</a></td> <td>USA</td> <td>245k</td> <td>WGS</td> <td>General population, diverse</td> <td>African, Indigenous American, East Asian, European, Oceanian, South Asian (<b>local ancestry</b>; see Notes below)</td> - <td>Yes</td> + <td>No</td> </tr> <tr> <td><a href="hgTrackUi?g=topmed">TOPMED Freeze 10</a></td> <td>USA</td> <td>151k</td> <td>WGS</td> <td>Heart, lung, blood, sleep disorder cohorts</td> <td>—</td> <td>No</td> </tr> <tr> <td><a href="hgTrackUi?g=sfariSparkExomes">SFARI SPARK WES</a></td> <td>USA</td> <td>140k</td> <td>WES</td> @@ -330,31 +338,31 @@ <td><a href="hgTrackUi?g=sgdpFreq">SGDP</a></td> <td>Global</td> <td>279</td> <td>WGS</td> <td>142 diverse populations worldwide</td> <td>By population</td> <td>Yes</td> </tr> <tr> <td><a href="hgTrackUi?g=gregor">GREGoR R4</a></td> <td>USA</td> <td>3.6k</td> <td>WGS</td> <td>Rare disease families (10.7k participants, 4.4k families)</td> <td>—</td> - <td>No</td> + <td>Yes</td> </tr> <tr> <td><a href="hgTrackUi?g=hgdp1kFreq">gnomAD HGDP+1kG</a></td> <td>Global</td> <td>4k</td> <td>WGS</td> <td>80 populations (HGDP + 1000 Genomes reprocessed)</td> <td>4k-cohort total AF only; per-population AF columns are <b>full gnomAD v3.1.2</b> release values (~76k genomes), see Notes below</td> <td>Yes</td> </tr> <tr> <td><a href="hgTrackUi?g=ga4kSnv">GA4K</a></td> <td>USA</td> <td>552</td> @@ -442,42 +450,48 @@ letters. All VCF files are normalized, with one allele per annotation (no multi-allele lines). </p> <h2>Methods</h2> <p> Each subtrack includes the upstream project's VCF largely as-released; per-subtrack pipelines (coordinate liftover, format conversion, header normalization) are documented on each subtrack's own description page and recorded in the <a href="https://github.com/ucscGenomeBrowser/kent/blob/master/src/hg/makeDb/doc/hg38/varFreqs.txt" target="_blank">build documentation</a>. The conversion scripts (<em>e.g.</em> <code>finngen_to_vcf.py</code>, <code>kovaToVcf.py</code>, <code>schema_addAcAnAf.py</code>, <code>svatalogFreqToVcf.py</code>) live alongside the makedoc in the <a href="https://github.com/ucscGenomeBrowser/kent/tree/master/src/hg/makeDb/scripts/varFreqs" target="_blank">scripts directory</a>. </p> <p> -The combined Affected and Background tracks are built by a separate pipeline: -each per-subtrack VCF is normalized (<code>bcftools norm</code>), all sites are merged into a single -callset, consequence annotations are recomputed against Ensembl with <code>bcftools csq</code>, -and the merged callset is split by phenotype into the two bigBed files via -<code>vcfToBigBed.py</code> + <code>bedToBigBed</code>. The mapping from upstream INFO fields to -bigBed columns is driven by two configuration files in the scripts directory: -<code>databases.tsv</code> (one row per source dataset, flagging which cohorts study a disease) -and <code>populations.tsv</code> (per-population AC/AF columns within each source, including the -affected and unaffected arm of each disease cohort). Editing those two files and rerunning -<code>mergeAndAnnotate.sh</code> followed by <code>vcfToBigBed.py --split-affected</code> rebuilds -the two tracks. The Genotyping Array Databases Combined track is built the same way from the -array cohorts only. +The combined Disease cohorts and Population reference tracks are built by a separate +pipeline: each per-subtrack VCF is normalized (<code>bcftools norm</code>), all sites are +merged into a single callset, consequence annotations are recomputed against Ensembl with +<code>bcftools csq</code>, and the merged callset is split by phenotype into the two bigBed +files via <code>vcfToBigBed.py</code> + <code>bedToBigBed</code>. Within each combined +track, the <b>Affected AF</b> and <b>Background AF</b> columns are +<i>pooled</i> across contributing cohort arms (sum of allele counts divided by sum of +allele numbers, with the per-arm AN derived from each cohort's AC and AF), so the displayed +frequency matches the carrier-count scale and a small cohort with a high local AF cannot +dominate the value. The mapping from upstream INFO fields to bigBed columns is driven by +two configuration files in the scripts directory: <code>databases.tsv</code> (one row per +source dataset, flagging which cohorts study a disease, and optionally a +<code>default_an</code> for cohorts that publish only AF) and <code>populations.tsv</code> +(per-population AC/AF columns within each source, including the affected and unaffected arm +of each disease cohort). Editing those two files and rerunning +<code>mergeAndAnnotate.sh</code> followed by <code>vcfToBigBed.py --split-affected</code> +rebuilds the two tracks. The Genotyping Array Databases Combined track is built the same +way from the array cohorts only. </p> <h2>Data Access</h2> <p>All the data is publicly available. The table above indicates if we are allowed to distribute it in VCF format. Most of the databases do not allow us to redistribute the data files directly from our website, but it can always be downloaded from the original websites in some form. Click the database link in the table above and see the "Data Access" section of the respective track for a description of where to download the data. When the data is freely available from our website, the Data Access section will also indicate the VCF file location on our download server. Because it contains some licensed data, the combined track is not available for download, but can be recreated using the conversion scripts in our <a href="https://github.com/ucscGenomeBrowser/kent/tree/master/src/hg/makeDb/scripts/varFreqs" target="_blank">GitHub repository</a> and the accompanying <a href="https://github.com/ucscGenomeBrowser/kent/blob/master/src/hg/makeDb/doc/hg38/varFreqs.txt" target="_blank">documentation file</a>. </p> <h2>Credits</h2> <p>This track is only possible thanks to the data from millions of volunteers around the world, who donated blood, signed consent forms and provided health information about themselves and sometimes their families. Click any of the tracks in the list above to see the specific credits for each project. Thanks to Alex Ioannidis, UCSC, for the inspiration for this track and to Andreas Lahner, MGZ, for feedback.</p> <h2>References</h2> <p> All of Us Research Program Genomics Investigators. <a href="https://doi.org/10.1038/s41586-023-06957-x" target="_blank">