Commits for max

v499_preview to v499_preview2 (2026-05-18 to 2026-05-25) v499

a95508a8614511de9cb72966524ef62d8f85517c Mon May 18 07:38:56 2026 -0700

• NMDetective-AI track polish: shorter longLabels, taller bigWig, item label without gene prefix. Drop the "Veiner et al., " parenthetical from both AI subtrack longLabels, bump the AI bigWig default height to 120 px, add the AI tracks to the NMD supertrack description page (under NMDetective scores), and strip the "<gene>:" prefix from the per-variant bigBed item labels (gene symbol is still in the geneName column and the mouseover). refs #33737 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
  • src/hg/makeDb/scripts/nmd/nmdDetectAi/makeBigBed.py - lines changed 4, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hg38/nmd.html - lines changed 5, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hg38/nmd.ra - lines changed 3, context: html, text, full: html, text

57c0c8dd69163aac91d9aa7ad7d301baf6feb584 Mon May 18 08:20:26 2026 -0700

• varFreqs: enable SweGen swefreq subtrack on hs1 with CHM13-specific supertrack description page, refs #36642 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
  • src/hg/makeDb/doc/hs1/varFreqs.txt - lines changed 22, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hs1/html/varFreqs.html - lines changed 25, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hs1/trackDb.ra - lines changed 2, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hs1/varFreqs.ra - lines changed 2, context: html, text, full: html, text

af9d8fe39e88f7b7cec3792ea797dab44f1416b0 Tue May 19 04:36:02 2026 -0700

• varFreqs: rebuild varFreqsAll with WBBC/TPMI/ChinaMAP/GenomeIndia, drop IndiGen, harden build pipeline Rebuilds /gbdb/hg38/varFreqs/_all/varFreqsAll.bb to fold in the four new subtracks registered earlier in May (WBBC 78.6M, TPMI 672k, ChinaMAP 147M, GenomeIndia 130M) and to drop IndiGen, which ships only a VRT bit and contributed an always-empty AC/AF column. New bb is 47 GB / 147 fields / 1.34 billion items (was 44 GB / 133 / 1.22B). Two pipeline fixes were necessary mid-rebuild: - bcftools 1.22 csq is stricter than earlier versions. Added --unify-chr-names chr,-,chr (Ensembl GFF3 uses bare "1" while merged VCF + FASTA use "chr1") and --force (5 SCHEMA alt contigs end up in the merge but aren't annotated in the GFF3) to the csq invocation in mergeAndAnnotate.sh. Four follow-up cleanups to the build scripts (no track change, just safer next rebuild): - mergeAndAnnotate.sh now reads VCF paths directly from databases.tsv in both the per-VCF strip+norm step and the merge step. The previous "files.txt + find normalized/" model could silently re-merge stale norm cache entries after a database was dropped from databases.tsv. - vcfToBigBed.py concat step streams sort stdout straight to disk instead of capture_output=True, which buffered the whole sorted chromosome (~24 GB for chr1) in Python RAM. - vcfToBigBed.py generate_trackdb_fragment() now emits the three customizations that used to have to be added on top of the auto-fragment by hand: filterType.consequence multipleListOr, the expanded consequence buckets (3_prime_utr, 5_prime_utr, non_coding, others), and skipEmptyFields on. - trackDb/human/varFreqs.ra updated to match the new bb columns (WBBC/TPMI/ChinaMAP/GenomeIndia AC+AF filters, WBBC 4-region population filters, IndiGen filter removed). refs #36642 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
  • src/hg/makeDb/doc/hg38/varFreqs.txt - lines changed 66, context: html, text, full: html, text
  • src/hg/makeDb/scripts/varFreqs/mergeAndAnnotate.sh - lines changed 20, context: html, text, full: html, text
  • src/hg/makeDb/scripts/varFreqs/vcfToBigBed.py - lines changed 12, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/varFreqs.ra - lines changed 41, context: html, text, full: html, text

2c38ad38c6edbcc5c71099acf373fe99386b1ecd Tue May 19 06:24:48 2026 -0700

• ncOrfs: color by Kozak strength, MANE+GENCODE intron grafting, hidden _teRgb Color: itemRgb now reflects categorical Kozak strength (Strong=golden yellow, Moderate=steel blue, Weak=medium gray, non-ATG=black, no-context =light gray) instead of the Noderer TE gradient. The TE-bin color is preserved as a hidden _teRgb field (underscore prefix => skipped by hgc) for a future second-color rendering. Every subtrack html now has its own swatch legend; the parent page legend was converted to the same style. UTRannotator intron grafting: addIntrons.py now uses MANE Select / MANE Plus Clinical as the primary donor, with the full GENCODE comprehensive set (gencodeV49) as fallback when every MANE candidate is rejected. Containment was relaxed to overlap so uORFs that extend past either end of the host transcript still get the host introns inside the overlap plus a single bridging block for the orphan portion. uORFs whose endpoint falls in a host intron are left single-block to avoid inventing exon structure. The chosen donor transcript ID is stored in intronsSource. Replaced the previous multi-database matching against gencNcOrfs/nuorfdb/metamorf/openprot. Counts on 44,435 uORFs: 5,934 MANE-grafted + 2,795 GENCODE-grafted = 8,729 spliced (was 5,318 with the old multi-db matching), 4,843 rejected for endpoint-in-intron, 143 with no host in either pool. refs #35101 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
  • src/hg/makeDb/doc/hg38/ncOrfs.txt - lines changed 44, context: html, text, full: html, text
  • src/hg/makeDb/scripts/ncOrfs/addIntrons.py - lines changed 238, context: html, text, full: html, text
  • src/hg/makeDb/scripts/ncOrfs/colorByKozak.py - lines changed 75, context: html, text, full: html, text
  • src/hg/makeDb/scripts/ncOrfs/gencNcOrf.as - lines changed 2, context: html, text, full: html, text
  • src/hg/makeDb/scripts/ncOrfs/gencNcOrfPhase2.as - lines changed 2, context: html, text, full: html, text
  • src/hg/makeDb/scripts/ncOrfs/metamorf.as - lines changed 2, context: html, text, full: html, text
  • src/hg/makeDb/scripts/ncOrfs/nuorfdb.as - lines changed 2, context: html, text, full: html, text
  • src/hg/makeDb/scripts/ncOrfs/openprot.as - lines changed 2, context: html, text, full: html, text
  • src/hg/makeDb/scripts/ncOrfs/run_kozak.sh - lines changed 3, context: html, text, full: html, text
  • src/hg/makeDb/scripts/ncOrfs/utrAnnotUorfs.as - lines changed 3, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hg38/gencNcOrfs.html - lines changed 8, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hg38/metamorf.html - lines changed 8, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hg38/ncOrfs.html - lines changed 35, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hg38/ncOrfs.ra - lines changed 3, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hg38/nuorfdb.html - lines changed 9, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hg38/openprot.html - lines changed 8, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hg38/utrAnnotUorfs.html - lines changed 24, context: html, text, full: html, text

1525618a814bcf6eb64c4147b99c520509a55d60 Tue May 19 07:37:29 2026 -0700

• Refresh mpraVarDB to 2026-05-19 snapshot; fix hg19 pos in name field. Upstream landed the Mouri/Tewhey pvalue correction (5,092 rows with pvalue > 1 -> 0) and replaced 47,156 placeholder fdr=1.0 values with NaN. Rebuild from the new CSV; remove the temporary "pending upstream fix" note in mpraVarDb.html. Also closes the pre-existing "hg19 pos in non-rs name field" issue: mpravardbToBed.py gained a post-liftOver step that rewrites the chr:pos prefix inside non-rs names with the hg38 coordinates. 47,160 names were rewritten in this build (e.g. chr1:1403972:C>CG at hg38 chr1:1468591 now reads chr1:1468592:C>CG). itemCount preserved at 239,028. refs #37359 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
  • src/hg/makeDb/doc/hg38/mpra.txt - lines changed 28, context: html, text, full: html, text
  • src/hg/makeDb/scripts/mpravardb/mpravardbToBed.py - lines changed 27, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hg38/mpraVarDb.html - lines changed 15, context: html, text, full: html, text

2197f6d5208aff4c48ccbe42e61a116d988ac392 Tue May 19 08:23:54 2026 -0700

• hubApi: add /blat endpoint with apiKey gating, format=hgblat, and known-agent bypass New src/hg/hubApi/blat.c implements /blat/<type> (dna, protein, transRna, transDna, guess) backed by the same gfServer logic as hgBlat. Key details: - Requires an apiKey for rate-limiting; botException() and botExceptionUserAgent() exempt IPs/user-agents in hg.conf (same policy as captcha bypass elsewhere in the browser stack). - Invalid apiKey returns a clean JSON 403 rather than an HTML 500 (pre-validated in hubApi.c main() before hgBotDelayTimeFrac runs). - Extra bot-delay fraction (default 0.3, 10x hubApi default) is configurable via hubApi.blatDelayFraction in hg.conf. - format=text/psl -> PSL text; format=hgblat -> byte-for-byte hgBlat?output=json shape; jsonOutputArrays=1 -> hubApi envelope with arrays (parallel to getData behaviour); default -> objects. - botExceptionUserAgent() carved out of cart.c's static isUserAgentException() into botDelay.c so non-cart callers can use it. - Cross-reference comments added in hgBlat.c and blat.c noting the shared logic so fixes get applied to both. refs #36315 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
  • src/hg/hgBlat/hgBlat.c - lines changed 101, context: html, text, full: html, text
  • src/hg/hubApi/blat.c - lines changed 427, context: html, text, full: html, text
  • src/hg/hubApi/dataApi.h - lines changed 11, context: html, text, full: html, text
  • src/hg/hubApi/hubApi.c - lines changed 18, context: html, text, full: html, text
  • src/hg/hubApi/makefile - lines changed 2, context: html, text, full: html, text
  • src/hg/hubApi/tests/makefile - lines changed 29, context: html, text, full: html, text
  • src/hg/inc/blatServers.h - lines changed 21, context: html, text, full: html, text
  • src/hg/inc/botDelay.h - lines changed 3, context: html, text, full: html, text
  • src/hg/lib/blatServers.c - lines changed 85, context: html, text, full: html, text
  • src/hg/lib/botDelay.c - lines changed 23, context: html, text, full: html, text
  • src/hg/lib/cart.c - lines changed 19, context: html, text, full: html, text
  • src/hg/lib/makefile - lines changed 1, context: html, text, full: html, text

03a7221f2973a860ee4363183fc5c582ce0d6178 Tue May 19 10:25:26 2026 -0700

• tons of changes to hubtools, committing now for chris
  • src/utils/hubtools/hubtools - lines changed 979, context: html, text, full: html, text

79e7ed894b39f65202ef820ae81af514f62e43dd Wed May 20 13:38:35 2026 -0700

• hubApi/blat: errCatch gfServer calls, fix comment, maxSeqCount from hg.conf, Content-Type spacing - Wrap gfServer block (mustOpen→gfDisconnect) in errCatch so a down or unreachable BLAT server returns a JSON 500 instead of a bare Apache error. - Fix comment: hgBlat's CGI interface and behavior remain unchanged (it was refactored internally in the previous commit). - Replace hardcoded maxSeqCount=25 with cfgOptionDefault("hgBlat.maxSequenceCount") to share the same hg.conf knob as hgBlat itself. - Remove spurious space in "Content-Type:text/plain" to match rest of hubApi. refs #36315 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
  • src/hg/hubApi/blat.c - lines changed 25, context: html, text, full: html, text

0ef7d0ffeea25371b4828fc23051cdd27598439f Wed May 20 14:57:51 2026 -0700

• varFreqs: add Genome of the Netherlands (GoNL) subtrack on hg38, GRCh38 re-analysis v1.0, 498 unrelated parents (250 fathers + 248 mothers), 36.4M biallelic records after dropping decoy/EBV contigs and bcftools norm split+left-align, refs #36642 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
  • src/hg/makeDb/doc/hg38/varFreqs.txt - lines changed 36, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/gonl.html - lines changed 76, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/varFreqs.ra - lines changed 10, context: html, text, full: html, text

5c88f2b10f443d95e3f9c8e4bbc81534b78d9222 Thu May 21 00:40:38 2026 -0700

• varFreqs: rename supertrack to "SNV Frequencies"; add GoNL (Netherlands, ~13x SR) to databases.tsv and to the varFreqsAll filter UI so the next combined-track rebuild picks it up. refs #36642 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
  • src/hg/makeDb/doc/hg38/varFreqs.txt - lines changed 3, context: html, text, full: html, text
  • src/hg/makeDb/scripts/varFreqs/databases.tsv - lines changed 1, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/varFreqs.ra - lines changed 7, context: html, text, full: html, text

9bfd58221b1539193cb7f0a317b4e959c1c7e49a Thu May 21 01:00:45 2026 -0700

• varFreqs: AI generated text sounds bad, hard to read, so remove typical AI language. "humanizer" pass on all 31 varFreqs description pages — cut em dashes, copula avoidance ("serves as", "stands as"), "-ing" puffery, and boilerplate filler ("We provide documentation that indicates how..."). Title-case headings and meaningful <b> emphasis preserved. No facts/URLs/counts/versions changed. tpmi.html added as a new file (was previously uncommitted). refs #36642 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
  • src/hg/makeDb/trackDb/human/abraom.html - lines changed 5, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/alfaVcf.html - lines changed 2, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/allofus.html - lines changed 2, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/chinamap.html - lines changed 13, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/colorsDbSnv.html - lines changed 9, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/finngen.html - lines changed 6, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/ga4kSnv.html - lines changed 8, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/gasp.html - lines changed 2, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/genomeindia.html - lines changed 11, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/gregor.html - lines changed 13, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hgdp1kFreq.html - lines changed 8, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/hrc.html - lines changed 4, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/indigenomes.html - lines changed 3, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/kova.html - lines changed 12, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/mgrb.html - lines changed 9, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/mxbFreq.html - lines changed 5, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/npm.html - lines changed 8, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/saudi.html - lines changed 4, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/schema.html - lines changed 1, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/sfariSparkExomes.html - lines changed 1, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/sgdpFreq.html - lines changed 5, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/svatalogSnv.html - lines changed 4, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/swefreq.html - lines changed 3, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/tishkoff180.html - lines changed 3, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/tommo60kjpn.html - lines changed 3, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/topmed.html - lines changed 1, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/tpmi.html - lines changed 135, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/ukbb.html - lines changed 6, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/varFreqs.html - lines changed 21, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/varFreqsAll.html - lines changed 7, context: html, text, full: html, text
  • src/hg/makeDb/trackDb/human/wbbc.html - lines changed 15, context: html, text, full: html, text