d069df7855eb648b5b53c51ef5218fe2e4a1d087 lrnassar Wed May 13 12:48:53 2026 -0700 mpraVarDb script rebuild: standardize NA, fix upstream typos, add pvalue note. refs #37359 Build-script changes (mpravardbToBed.py + mpravardb.as): - sanitize_text() now normalizes "None"/"NA"/"N/A"/"null"/"NULL"/"nan" to empty string. Removes 55,108 stale sentinels: 53,144 disease= "None" eQTL rows, 1,964 disease="NA" Kircher rows, 44 ref/alt=NA Myint rows. - Literal-replacement table for three upstream typos: "30 UTR" -> "3'UTR" (26,546 Schuster description rows), "Familial hypercholesterol emia" -> "Familial hypercholesterolemia" (2,176 Kircher disease rows), "Alchol use disorder" -> "Alcohol use disorder" (88 Rao disease rows). - fmt_mo() renders NaN floats as "NA" in mouseOver helper fields; 30,921 rows that previously showed literal "nan" now read "NA". - Tightened name+rsid handling: treat a value as an rsID only if it starts with "rs". 2,088 hg19-coord-style names like "1_1403972_CG" are reformatted to "chr<X>:<hg38pos>:<ref>><alt>" and the rsid field cleared to "" so the dbSNP linkout doesn't fire on a bogus value. - Removed the 250-char truncation that cut Griesemer descriptions mid-sentence; mpravardb.as switched description and mpraStudy from "string" to "lstring" to admit the full upstream text. Description page (mpraVarDb.html): - Added a "Note (pending upstream fix)" paragraph in Methods explaining the 5,092 rows with pvalue > 1 (test statistic mislabeled in the pvalue field by upstream curators for Mouri 2022 and Tewhey 2016). Bracketed by an HTML comment "TEMP: remove once Tao Wang fixes..." for future cleanup when the next snapshot lands. bigBed rebuilt; itemCount 239,028 preserved. Pre-rebuild backup: /hive/data/genomes/hg38/bed/mpra/mpravardb/mpravardb.bb.pre-2026-05-14-backup Makedoc updated with the QA-2 build-script rebuild section. diff --git src/hg/makeDb/doc/hg38/mpra.txt src/hg/makeDb/doc/hg38/mpra.txt index ed45ad17294..ab6a5195d24 100644 --- src/hg/makeDb/doc/hg38/mpra.txt +++ src/hg/makeDb/doc/hg38/mpra.txt @@ -128,36 +128,54 @@ # the corrected primary citation (Jin et al. 2024, PMID 39325859 in # Bioinformatics; the previous "Wang T, Matreyek KA, Yang X." citation # was fabricated -- not the actual authors of either the preprint # PMID 38617248 or the published paper). # - 7 studies-table row counts corrected to match data (Tewhey, Griesemer, # Abell, Mouri, McAfee, Cooper, Lu). # - HTML mouseOver upgraded to bold/multi-line. # - dataVersion "MPRAVarDB snapshot 2026-03-10" added to stanza. # - urls rsid="https://www.ncbi.nlm.nih.gov/snp/$$" added so rsIDs are # clickable linkouts. # - Methods + Display Conventions paragraphs added: scoring methodology # differs across studies, post-transcriptional vs transcriptional # distinction (Griesemer/Schuster 3'UTR), Kircher saturation # mutagenesis structure, log2FC interpretation. # -# Punted to Redmine for Max / Tao Wang (potential next-rebuild items): +# Punted to Redmine for Max / Tao Wang (status as of 2026-05-14): # - 5,092 rows (Mouri, Tewhey) have pvalue > 1 (impossible; max 8.96). # FDR appears valid; pvalue field looks like a t-statistic mislabeled. -# mouseOver/details show misleading p-values for ~2% of significant -# rows. -# - 30,921 rows display literal "nan" / "None" in mouseOver and -# details fields where NA-coded values were preserved verbatim. Two -# sentinel conventions coexist ("None" in 53k eQTL rows, "NA" in 2k -# Kircher rows). -# - Upstream typos in 28,810 rows: "30 UTR" in 26,546 Schuster -# descriptions, "Familial hypercholesterol emia" in 2,176 Kircher -# diseases, "Alchol use disorder" in 88 Rao diseases. +# Upstream curators acknowledged; fix is weeks out. We added a +# "Note (pending upstream fix)" paragraph to mpraVarDb.html bracketed +# by an HTML comment "TEMP: remove once Tao Wang fixes..." -- +# remove that paragraph when the next CSV snapshot lands. # - 60,860 rows have description="GWAS" (no detail) -- upstream limit. # - 1,069 rows have multi-allelic alt collapsed into one row (e.g. -# "T/A,G") with one log2FC/pvalue. -# - 2,088 rows preserve hg19-coord-style names (e.g. "1_1403972_CG") -# post-liftOver to hg38; coordinates and name no longer match. +# "T/A,G") with one log2FC/pvalue. Upstream-collapsed; per-allele +# values not recoverable from the CSV. # - 969 rows are colored red (FDR<0.05) but pvalue=nan -- mouseOver -# reads "FDR: 0.001 / p-value: nan" which looks contradictory. -# - 250-char truncation in mpravardbToBed.py cuts Griesemer -# descriptions mid-sentence; should raise/remove the cap. +# reads "FDR: 0.001 / p-value: NA" (NA now, formerly "nan"). +# Defensible; FDR can be reported without per-test p. +# +# QA-2 build-script rebuild 2026-05-14 (RM #37359, Lou): +# Items handled in mpravardbToBed.py + mpravardb.as, single rebuild: +# - sanitize_text now maps "None"/"NA"/"N/A"/"null"/"NULL"/"nan" to +# empty string after the existing UTF-8 sanitization. Removed +# 55,108 stale sentinels (53,144 disease="None" eQTL rows + +# 1,964 disease="NA" Kircher rows + 44 ref/alt=NA Myint rows). +# - sanitize_text applies a literal-replacement table for three +# upstream typos: "30 UTR" -> "3'UTR" (26,546 Schuster description +# rows), "Familial hypercholesterol emia" -> "Familial +# hypercholesterolemia" (2,176 Kircher disease rows), "Alchol use +# disorder" -> "Alcohol use disorder" (88 Rao disease rows). +# - New fmt_mo() renders NaN floats as "NA" in the mouseOver helper +# fields rather than literal "nan"; 30,921 rows fixed. +# - Name + rsid handling tightened: a value is treated as an rsID +# only if it starts with "rs". 2,088 hg19-coord-style names like +# "1_1403972_CG" are now reformatted to "chr<X>:<hg38pos>:<ref>><alt>" +# and the rsid field is set to "" so the dbSNP linkout does not +# fire on a bogus value. +# - Removed the 250-char truncation that was cutting Griesemer +# descriptions mid-sentence; mpravardb.as switched the description +# and mpraStudy fields from "string" to "lstring" to allow full +# upstream text. +# - Pre-rebuild backup: mpravardb.bb.pre-2026-05-14-backup +# - itemCount preserved: 239,028.