All File Changes
v504_preview to v504_preview2 (2026-09-07 to 2026-09-14) v504
Show details
- .gitignore
- lines changed 4, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- confs/asia.hg.conf
- lines changed 3, context: html, text, full: html, text
297c479ca3a30cff1b1669b231a8f76928f15d6e Sun Sep 13 01:11:22 2026 -0700
Installing updated hg.conf files from UCSC servers
- confs/euro.hg.conf
- lines changed 3, context: html, text, full: html, text
297c479ca3a30cff1b1669b231a8f76928f15d6e Sun Sep 13 01:11:22 2026 -0700
Installing updated hg.conf files from UCSC servers
- confs/hgwdev.hg.conf
- lines changed 3, context: html, text, full: html, text
992aeef92fea7be25a2acd916578898649212a32 Mon Sep 7 11:31:07 2026 -0700
quickLift: gate the alignment lift behind an hg.conf flag, refs #38249
Add browser.quickLiftAlignments, default FALSE, so the alignment lift ships
dark and a machine turns it on with browser.quickLiftAlignments=on. It sits
beside browser.quickLift, the gate on the rest of the feature.
quickLiftAlignmentsEnabled() in hg/lib/quickLift.c is the one read, and
validateOneTdb in hg/lib/trackHub.c is the one place that asks it, before an
alignment track may enter a quickLift hub. That is the only door:
quickLiftUrl and quickLiftDb, the pair every lift path keys off, are written
by the quickLift hub writer and by nothing else, so with the flag off an
alignment track never gets them and the lifting, drawing and details code
behind them cannot be reached. pslTrack.c, chainTrack.c, wigMafTrack.c,
bigBedTrack.c and hgc.c are unchanged.
With the flag off hgConvert lists psl, bigPsl, chain, bigChain, maf, bigMaf
and wigMaf tracks in its "type is not supported by QuickLift" table, which is
what it did before this work. A hub built while the flag was on keeps working
after it is turned off, since its stanzas are already in the hub file in
trash, so this holds the feature back from people who have not used it rather
than switching off a session that has.
Read the hg.conf half with a literal cfgOptionBooleanDefault rather than
cartOrCfgOption so harvestHgConf.py can see it; a cart variable of the same
name still overrides it. Register the flag in hgConfCatalog.py with
role="gate" so the sunset report tracks it, and turn it on in
confs/hgwdev.hg.conf.
- lines changed 19, context: html, text, full: html, text
297c479ca3a30cff1b1669b231a8f76928f15d6e Sun Sep 13 01:11:22 2026 -0700
Installing updated hg.conf files from UCSC servers
- src/ameme/ameme.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/ameme/amemeFloat.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/ameme/fixpAmeme.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/ameme/makefile
- lines changed 1, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/affyTransciptome/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/blastToPsl/makefile
- lines changed 4, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/cartDump/cartDump.c
- lines changed 32, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- src/hg/cgilib/api.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/cgilib/bedCart.c
- lines changed 9, context: html, text, full: html, text
53a8c1ac1513a14c299dc27940a45d97e6e5456f Tue Sep 8 09:33:22 2026 -0700
bedItemRgb: let an explicit "itemRgb on" beat the presence of a "color" setting
A stanza could once say both "itemRgb on" and "color" and get both: items drawn
from the file's own RGB column, labels drawn in the color setting. Since 2025 the
color setting wins outright and the two tracks in Gerardo's test hub, one with
"color" alone and one with both settings, render identically.
The cause is the order of the tests in bedItemRgb(), not a missing feature. The
"color" test is only about whether to turn itemRgb on by DEFAULT, but it sat in the
same early return as the "itemRgb off" test, above the test for an explicit
"itemRgb on" -- so that test was unreachable for any stanza that set a color, and
an explicit setting could be overridden by the mere presence of one.
5448842337e added the color rule while a later block still honoured an explicit
setting; 88d620e6c82 folded the two tests together and dropped that block;
c54077c4044 added it back, but below the color test.
Moving the color test below both explicit tests restores the old behavior. Only one
of the four cases changes: "itemRgb on" plus "color" now returns TRUE. "itemRgb
off" still returns FALSE, "color" alone still suppresses the default, and a stanza
that says neither still follows the alwaysItemRgb hg.conf default. The label keeps
taking its color from the color setting either way, since that comes from
colorFromCart() rather than from here.
Measured before and after with a four-track hub whose items all carry a pure blue
itemRgb column and whose color settings are pure green: the both-settings track
went from green items to blue items, with its center label green throughout. The
other three tracks are unchanged. A Docent regression test asserts all four rows,
kent/src/hg/utils/docent/tests/regress/rm36212.xfail.docent.yaml.
refs #36212
- src/hg/cgilib/cartJson.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/cgilib/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/cirm/cdw/cdwGetFile/cdwGetFile.c
- lines changed 14, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/cirm/cdw/cdwGetMetadataAsFile/cdwGetMetadataAsFile.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/cirm/cdw/cdwServeTagStorm/cdwServeTagStorm.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/cirm/cdw/cdwWebBrowse/cdwWebBrowse.c
- lines changed 4, context: html, text, full: html, text
08228c9a2d84f3d0a30474f1f24598dbb293dfa5 Mon Sep 7 11:29:09 2026 -0700
cdwWebBrowse: do not build 'id IN ()' when no file matches the filters
When a download filter matched no files, findDownloadableFiles() built
SELECT * FROM cdwFile WHERE id IN () which MariaDB rejects with error
1064. Return an empty list instead; both callers already handle it.
Found in the apache error log, email from Erich.
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/cirm/cdw/lib/cdwLib.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/cirm/gateway/htdocs/sspsygeneTimeline.html
- lines changed 26, context: html, text, full: html, text
b6a47b7ad276b0fc44d624d9038edb096bb52c0c Wed Sep 9 15:05:26 2026 -0700
Adding MiNND Year4 milestone items to timeline
- src/hg/das/das.c
- lines changed 5, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/encode/docId/lib/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/encode3/encodeDataWarehouse/edwScriptSubmitStatus/edwScriptSubmitStatus.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/encode3/encodeDataWarehouse/edwWebAuthLogin/edwWebAuthLogin.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/encode3/encodeDataWarehouse/edwWebAuthLogout/edwWebAuthLogout.c
- lines changed 3, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/encode3/encodeDataWarehouse/lib/edwLib.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/genePredToMafFrames/makefile
- lines changed 5, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/hgBlat/hgBlat.c
- lines changed 10, context: html, text, full: html, text
595531f894a27e59e18b32436f6cb932fa4374a0 Tue Sep 8 13:03:44 2026 -0700
BLAT form character counter: per-type limits passed through from the C constants. refs #38293
The counter showed the 75,000 DNA limit for every query type; protein and translated queries
are capped at 10,000, so an oversized protein paste looked fine until the server rejected it.
The per-sequence limits are now named constants in hgBlat.c, emitted into hgBlatFormData and
read by the counter, which keys on the Query type dropdown and recounts when it changes -
so the numbers cannot drift apart again. BLAT's guess counts against the DNA limit.
- lines changed 2, context: html, text, full: html, text
c0a6706dea55f019923332c5f51dfe64fc420dc8 Wed Sep 9 12:15:27 2026 -0700
Comment touch-ups from CR: reattach delayFraction's continuation comment, drop an imprecise history aside in the counter comment. refs #38293
- src/hg/hgChooseDb/hgChooseDb.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgCollection/hgCollection.c
- lines changed 6, context: html, text, full: html, text
59e2bbf2d3a99e5f18436a756e1e5c8b64308e33 Tue Sep 8 09:21:02 2026 -0700
Copy a track collection's hub file when the program that writes it asks for a copy, instead of on every session load.
cartCopyLocalHubs ran on every session load, in every CGI and in four more places in
hgSession. For each customComposite-<db> cart variable it copied the collection's hub
file to a fresh trash name and registered the copy in hgcentral.hubStatus to get a hub
id. Nothing removed the old row. 81% of the 3.1 million rows in hubStatus on the RR are
these dead registrations, and each load also took the central_hubStatus advisory lock
and wrote to a MyISAM table, which serializes every concurrent load of a shared session
that carries a collection.
The copy itself is needed. A saved session's hub file lives under sessionDataDir and
every load of that session names it, so hgCollection must not write it in place. It was
just being made for every load, and almost no load is followed by an edit. hgCollection
is the only program that writes one of these files, so it now asks for its own copy:
main() calls cartRequestLocalHubCopy() before it opens the cart, and cartNew() makes the
copy for it.
That has to be a property of the program rather than of the request, and the copy has to
be made at cart open, above hubConnectLoadHubs. Copying the file gives the hub a new id,
the hubs are loaded during cart open, and trackList carries that id in every track name,
so a copy made any later leaves printTrackDbListToHub looking for the collection under
an id trackList does not have and the hub file comes out with a header and no tracks. A
condition that instead tries to work out whether this particular request will write
cannot be correct either: "cmd" is not in hgTracks' excludeVars, so it persists in the
cart and hgCollection dispatches on the cart rather than on the CGI variable, a settings
file can carry it, and a command-line run has no SCRIPT_NAME to test.
Three other parts. copyLocalHubs skips a hub the cart already owns, so repeated edits
reuse one file and one hub id rather than renumbering the hub on every drag; the test
requires a plain file, because saveTrackFile leaves the trash name behind as a symbolic
link to the durable copy and writing through that link would rewrite the saved session's
own file. copyLocalHubs also screens the path with isServerUserFilePath before opening
it, the way getHubName does, and selects on customComposite-<db> with the dash so that
the names it acts on are the ones fileNameCartVarPrefixes screens. saveTrackFile copies
when the source is a local hub outside trash that is not under this session's own
directory, so loading one session and saving it under a new name gives the new session
its own file instead of a reference into the first one's directory. pathIsUnderDir is no
longer static in trashDir.c.
All of it is behind the hg.conf gate collectionHubCopyOnWrite, default off, which
reproduces the old behavior exactly. Retiring the gate is not uniform: the body of
cartCopyLocalHubsOnSessionLoad is the old behavior and that function and its five
callers go away with the gate, while the other three tests lose only the gate term.
Measured on hgcentraltest with one binary. With the gate off, five loads of a session
carrying a collection add five hubStatus rows and three later edits add none; with it
on, the five loads add none and the first edit adds one. Saving a loaded session costs
two rows off and one on. The resulting hub file is byte identical either way apart from
the hub id, and the file the saved session names is unchanged by md5 and mtime,
including after an edit made straight after a save, and including when the cart names a
trash symbolic link into session storage.
refs #38273
- lines changed 9, context: html, text, full: html, text
543c9ee045ba1832faaa9b75c6dc1e369dffce5a Thu Sep 10 05:21:42 2026 -0700
Login and Sign out come back to a page that was reached by POST, refs #38192
Clicking a track name in the list below the browser image submits the
hgTracks form to hgTrackUi, so the request is a POST even though the
track name sits in the URL. The return URL builder threw the query
string away for anything that was not a GET, which left a returnto of
hgTrackUi?hgsid= alone, and hgTrackUi cannot draw a page from that
because the track name is deliberately not kept in the cart. Login and
Sign out therefore ended in an error instead of coming back.
The query string of a POST lives in the form's action URL, which is the
address the browser is showing, so returning to it is no different from
the visitor pressing reload. Only the form body is left behind, and the
cart already holds what mattered from it. hgTracks stays the exception:
its query string can hold a one-shot zoom or drag.
Also, hgTrackUi now says which parameter is missing when it is reached
without a track name, rather than failing on a bare hash lookup, and
hgCollection's own "you must be logged in" link brings the visitor back
to hgCollection instead of the sessions page.
- src/hg/hgConvert/hgConvert.c
- lines changed 5, context: html, text, full: html, text
f219460db8db952415b5201ba01df99ce4999004 Thu Sep 10 13:46:42 2026 -0700
genark: pass the liftOver accession list as an slName list, refs #38328
genarkLiftOverDbs() took a pre-quoted SQL fragment that its callers
assembled. It now takes a struct slName list and builds the query
itself with sqlDyStringCreate, so no caller writes SQL text.
Accessions that do not start with GC are skipped, since nothing else
can match the table. hdb.c and hgConvert.c updated for the new
signature.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgCustom/hgCustom.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgGateway/hgGateway.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgGenome/hgGenome.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgHubConnect/hgHubConnect.c
- lines changed 3, context: html, text, full: html, text
1ea512b92ecb6ae35dae658eac7eb1dc5a03550e Thu Sep 10 05:58:37 2026 -0700
hgHubConnect: let mirrors show the API key section without hubSpace, refs #38323
The API key section was gated on storeUserFiles && showHubApiKey, so a
mirror could only offer key generation by also turning on the whole
hubSpace upload stack (tusd endpoint, tusdDataDir, hubSpaceUrl). But keys
are also used to bypass the download CAPTCHA and live in each machine's
own hgcentral, so botDelay tells a genome-euro user that keys are
server-specific and sends them to euro's hgHubConnect, where the section
was not being printed. Gate on showHubApiKey alone, as before e9a3e487007.
Mirrors still need showHubApiKey=on in hg.conf.
- src/hg/hgHubConnect/hooks/makefile
- lines changed 1, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/hgLogin/hgLogin.c
- lines changed 10, context: html, text, full: html, text
800ff0dc7028712146bb5ea2a691ae42d299d25b Wed Sep 9 06:40:29 2026 -0700
hgLogin: tighten activation link handling - treat a missing or empty token as invalid, and apply the seven-day expiry that the confirmation mail already promises, refs #38302
- lines changed 13, context: html, text, full: html, text
5a249cd50f592a3b7598110eec792cfc942fab3f Wed Sep 9 08:52:06 2026 -0700
Address the v504 code review
hgSession: an anonymous share name that arrives with the request is saved only
when it is not already in the table. Every anonymous link sits under the one
reserved user "l", so a name already there stays as it is and the caller is told
so. The top-right Share dialog is unaffected, since it passes a name it has
just reserved and such a name does not exist yet. Snapshot names are now left
out of both My Sessions listings, which is what their "__" prefix has claimed
all along.
Share dialog: "Create link & copy" reports the copy instead of promising it.
copyToClipboard says whether the text reached the clipboard, the dialog passes
that on when a browser refuses, and it tries the asynchronous clipboard API
before giving up. The preview is built with the same encoding the server uses,
so a name holding a hyphen or a slash previews as the link that really gets
made. Cancelling out of the name editor no longer copies a second time, and a
reply with no link in it says so rather than showing "undefined".
hgBlat: a second click on the share button while the first request is still out
no longer mints a second snapshot session, and a box dismissed during the wait
stays closed.
Also: a snapshot moves a cart value into durable storage only when it is a
trash path, the way sessionData's own callers check; sqlAddressMatch keeps to
its own documented precondition when handed an empty address; alphaGenomeToWig
compares its output with its input rather than with itself, rejects a position
that is not all digits and skips an empty score; and hgc's default iframe width
reaches the browser as one percent sign.
refs #38294
- src/hg/hgMenubar/hgMenubar.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgPhyloPlace/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/hgSearch/hgSearch.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgSession/backup.c
- lines changed 5, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- lines changed 28, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgSession/hgSession.c
- lines changed 8, context: html, text, full: html, text
59e2bbf2d3a99e5f18436a756e1e5c8b64308e33 Tue Sep 8 09:21:02 2026 -0700
Copy a track collection's hub file when the program that writes it asks for a copy, instead of on every session load.
cartCopyLocalHubs ran on every session load, in every CGI and in four more places in
hgSession. For each customComposite-<db> cart variable it copied the collection's hub
file to a fresh trash name and registered the copy in hgcentral.hubStatus to get a hub
id. Nothing removed the old row. 81% of the 3.1 million rows in hubStatus on the RR are
these dead registrations, and each load also took the central_hubStatus advisory lock
and wrote to a MyISAM table, which serializes every concurrent load of a shared session
that carries a collection.
The copy itself is needed. A saved session's hub file lives under sessionDataDir and
every load of that session names it, so hgCollection must not write it in place. It was
just being made for every load, and almost no load is followed by an edit. hgCollection
is the only program that writes one of these files, so it now asks for its own copy:
main() calls cartRequestLocalHubCopy() before it opens the cart, and cartNew() makes the
copy for it.
That has to be a property of the program rather than of the request, and the copy has to
be made at cart open, above hubConnectLoadHubs. Copying the file gives the hub a new id,
the hubs are loaded during cart open, and trackList carries that id in every track name,
so a copy made any later leaves printTrackDbListToHub looking for the collection under
an id trackList does not have and the hub file comes out with a header and no tracks. A
condition that instead tries to work out whether this particular request will write
cannot be correct either: "cmd" is not in hgTracks' excludeVars, so it persists in the
cart and hgCollection dispatches on the cart rather than on the CGI variable, a settings
file can carry it, and a command-line run has no SCRIPT_NAME to test.
Three other parts. copyLocalHubs skips a hub the cart already owns, so repeated edits
reuse one file and one hub id rather than renumbering the hub on every drag; the test
requires a plain file, because saveTrackFile leaves the trash name behind as a symbolic
link to the durable copy and writing through that link would rewrite the saved session's
own file. copyLocalHubs also screens the path with isServerUserFilePath before opening
it, the way getHubName does, and selects on customComposite-<db> with the dash so that
the names it acts on are the ones fileNameCartVarPrefixes screens. saveTrackFile copies
when the source is a local hub outside trash that is not under this session's own
directory, so loading one session and saving it under a new name gives the new session
its own file instead of a reference into the first one's directory. pathIsUnderDir is no
longer static in trashDir.c.
All of it is behind the hg.conf gate collectionHubCopyOnWrite, default off, which
reproduces the old behavior exactly. Retiring the gate is not uniform: the body of
cartCopyLocalHubsOnSessionLoad is the old behavior and that function and its five
callers go away with the gate, while the other three tests lose only the gate term.
Measured on hgcentraltest with one binary. With the gate off, five loads of a session
carrying a collection add five hubStatus rows and three later edits add none; with it
on, the five loads add none and the first edit adds one. Saving a loaded session costs
two rows off and one on. The resulting hub file is byte identical either way apart from
the hub id, and the file the saved session names is unchanged by md5 and mtime,
including after an edit made straight after a save, and including when the cart names a
trash symbolic link into session storage.
refs #38273
- lines changed 108, context: html, text, full: html, text
fc8de100a3437b9fc33bdeb0f459ca2a93e2f318 Wed Sep 9 08:14:32 2026 -0700
hgSession: address the code review of the new Sessions page
Rename and unshare now keep the public listing's thumbnail with the session it
belongs to. The picture's file name is built from the encoded session name, so
renaming a listed session left the listing pointing at nothing and the old file
behind, and dropping a session from the listing to a plain shared link kept the
picture. The classic page had the same problem in a subtler form: it removed the
thumbnail after the row had already been renamed, so the old file survived.
Saving under a name that is already in use asks before it replaces that session,
using the failIfExists reply that the top-right Share a link menu already relies
on. The description and "only I can load it" steps that follow a save now report
a failure instead of reloading in silence, and what thumbnailAdd has to say when
it cannot build a picture reaches the user instead of being freed unread.
A session description no longer travels through a title attribute. The tooltip
machinery in utils.js inserts its text with innerHTML and an attribute is decoded
on the way, so a description containing angle brackets was interpreted as markup
rather than shown as typed. It is attached, escaped, after each table draw, which
also gives the rows DataTables renders later the same styled mouseovers as the
rest of the page.
Also: the AJAX endpoints say so when there is no session by that name, instead of
reporting a no-op as a success; the new page always offers its way back to the
classic page, since the cart variable that got the user there sticks; and four
unused CSS rules, a dead element lookup and a dead local are gone. hgConfCatalog
cited the wrong ticket for the two sessionNewPage flags.
refs #38180, refs #38157
- lines changed 51, context: html, text, full: html, text
5a249cd50f592a3b7598110eec792cfc942fab3f Wed Sep 9 08:52:06 2026 -0700
Address the v504 code review
hgSession: an anonymous share name that arrives with the request is saved only
when it is not already in the table. Every anonymous link sits under the one
reserved user "l", so a name already there stays as it is and the caller is told
so. The top-right Share dialog is unaffected, since it passes a name it has
just reserved and such a name does not exist yet. Snapshot names are now left
out of both My Sessions listings, which is what their "__" prefix has claimed
all along.
Share dialog: "Create link & copy" reports the copy instead of promising it.
copyToClipboard says whether the text reached the clipboard, the dialog passes
that on when a browser refuses, and it tries the asynchronous clipboard API
before giving up. The preview is built with the same encoding the server uses,
so a name holding a hyphen or a slash previews as the link that really gets
made. Cancelling out of the name editor no longer copies a second time, and a
reply with no link in it says so rather than showing "undefined".
hgBlat: a second click on the share button while the first request is still out
no longer mints a second snapshot session, and a box dismissed during the wait
stays closed.
Also: a snapshot moves a cart value into durable storage only when it is a
trash path, the way sessionData's own callers check; sqlAddressMatch keeps to
its own documented precondition when handed an empty address; alphaGenomeToWig
compares its output with its input rather than with itself, rejects a position
that is not all digits and skips an empty score; and hgc's default iframe width
reaches the browser as one percent sign.
refs #38294
- lines changed 7, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- lines changed 9, context: html, text, full: html, text
b50a995e326f37cb2248e9a04628c15fb1b58215 Mon Sep 14 05:40:04 2026 -0700
My Sessions no longer hides a user's own "__" sessions, refs #38313
Both My Sessions listings decided a row was a share token by testing the
session name for the "__" prefix, so they also hid sessions a user had named
that way themselves - eleven of them on the RR, across four accounts, and
their owners could no longer rename, describe, reshare or delete them.
The prefix is a naming convention we follow, not a namespace we own. The
authoritative mark is the "snapshotType <type>" line saveSnapshotSession()
already writes into the settings column, so test that instead. Both queries
already select settings, so no query changes.
snapshotTypeFromSettings() walks the settings lines rather than calling
raFromString(): it runs once per listed row, and a hash there costs ~600ns and
three allocations for every session that has a description, against ~30ns and
none for the walk. Rows with empty settings, the common case, short-circuit
in both.
- src/hg/hgSession/makefile
- lines changed 2, context: html, text, full: html, text
05faf6b30a29e2260029a5d385a992c3481bcae1 Fri Sep 11 12:54:27 2026 -0700
hgSession: run the tests directory, refs #38340
hg/makefile's %.testAll rule runs "make test" in the app directory once that app
has a tests/makefile, and swallows the result with "|| true". hgSession had no
test target, so the backupParseTest added with the #38340 fix errored out and
was silently skipped. It passed only when run by hand.
Same three lines hubApi uses.
Note for anyone adding a tests directory under hg/: seventeen other apps there
have one and no test target, so whatever is in them is not running either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgSession/tests/backupParseTest.c
- lines changed 77, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgSession/tests/expected/backupParseTest
- lines changed 40, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgSession/tests/makefile
- lines changed 31, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgSuggest/hgSuggest.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgTables/genomeSpace.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgTables/makefile
- lines changed 25, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/hgText/hgText.c
- lines changed 5, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgText/hgWigText.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgTrackUi/hgTrackUi.c
- lines changed 18, context: html, text, full: html, text
5515006ee81e5de99ebd03581644a8a96f33599e Mon Sep 7 07:36:11 2026 -0700
tiny text change on trackUi page, no ticket
- lines changed 4, context: html, text, full: html, text
c0e8fa6df3a0bd406c4188d49ee00f20aef203e5 Mon Sep 7 12:07:18 2026 -0700
Substitute trackDb variables in hub track description pages
A hub's description page comes straight off the hub's web server and has
never been through variable substitution, so a $db or $parentTrack in it
reached the reader as literal text. Native trackDb pages are fine, since
hgTrackDb substitutes them when it loads trackDb, but there was no
equivalent step for a hub.
hgc's getTrackHtml and hgTrackUi's trackUi both call hVarSubstTrackDbHtml
on a hub track's html. Only a short list of variables is recognized there and nothing is an
error, because a hub page written before this existed can easily contain
a dollar sign inside a shell example, and silently rewriting that would
be worse than not substituting at all.
Adds $parentTrack, the name of the container a track sits in, which is
what a subtrack description page needs to link back to its superTrack or
composite. Views are skipped, since a view has no page of its own, and
the hub_<id>_ prefix is kept so the name works as hgTrackUi's g=
parameter. Documents $track, $parentTrack and $hgsid in trackDb/README.
refs #37599
- lines changed 25, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- lines changed 7, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- lines changed 4, context: html, text, full: html, text
d033cea2063e9362949baf5b4d8b837597173a0d Thu Sep 10 05:16:05 2026 -0700
Address the code review of the Sep 9 commits
Faceted composite: text that comes from a hub - a metadata column's
description, the name and title of a data type, and the values quoted back in
the "could not load the metadata" row - is put on the page as text rather than
as markup. The three places built their markup from template strings, so a
value carrying angle brackets or a quote was read as HTML: the column
description now goes through the shared htmlEncode() once where the header is
parsed, and the other two build their elements as nodes. The error row reads
better for it as well, since a value with brackets in it used to disappear from
the message that was meant to show it.
The saved UI state keys on the assembly as well as the metadata id. localStorage
is per-origin, so two assemblies whose tracks share a name were sharing one
entry, and a row order dragged on one came back on the other over a different
set of samples. hgTrackUi passes the database down for it. State saved under the
old key is dropped, which costs a facet selection or a page length.
Imprinting: the five subtrack description pages link back to the container as
hgTrackUi?db=$db&g=$parentTrack, without the hgsid. Native trackDb html is
substituted by hgTrackDb as it loads the table, where there is no cart, so
${hgsid} came out empty and the link read 'hgsid=&g=...'. Matches what the
Fiber-seq pages already do. The makeDoc note that described the old form is
updated with the reason.
UniProt otto: README.txt lists all eight things that reach runLog.txt. It had
four, and was missing LOCKED, along with PREFLIGHT-FAIL, END and INTERRUPTED.
refs #36210
refs #37599
refs #38300
- lines changed 8, context: html, text, full: html, text
543c9ee045ba1832faaa9b75c6dc1e369dffce5a Thu Sep 10 05:21:42 2026 -0700
Login and Sign out come back to a page that was reached by POST, refs #38192
Clicking a track name in the list below the browser image submits the
hgTracks form to hgTrackUi, so the request is a POST even though the
track name sits in the URL. The return URL builder threw the query
string away for anything that was not a GET, which left a returnto of
hgTrackUi?hgsid= alone, and hgTrackUi cannot draw a page from that
because the track name is deliberately not kept in the cart. Login and
Sign out therefore ended in an error instead of coming back.
The query string of a POST lives in the form's action URL, which is the
address the browser is showing, so returning to it is no different from
the visitor pressing reload. Only the form body is left behind, and the
cart already holds what mattered from it. hgTracks stays the exception:
its query string can hold a one-shot zoom or drag.
Also, hgTrackUi now says which parameter is missing when it is reached
without a track name, rather than failing on a bare hash lookup, and
hgCollection's own "you must be logged in" link brings the visitor back
to hgCollection instead of the sessions page.
- lines changed 3, context: html, text, full: html, text
0a5f9cb363719d91237e0ec3c4b8831ad447daca Thu Sep 10 06:45:33 2026 -0700
hgTrackUi: a missing track name is bad input, not a stack dump, refs #38192
hgTrackUi aborts when the URL carries no g= parameter, which happens with
a hand-edited or truncated address and with the crawlers that trim query
strings. errAbort routes that through the stack dump handler, so hg.conf
browser.dumpStack turns each one into a gdb backtrace in the error log
and about a third of a second of wait4. hUserAbort exists for errors that
come from user input: same message on the page, one line in the log.
- src/hg/hgTracks/bigBedTrack.c
- lines changed 45, context: html, text, full: html, text
544f7054d812254a0883a412b98a08dcd78f1b84 Fri Sep 4 11:39:41 2026 -0700
hgTracks: draw quickLifted psl and bigPsl tracks
pslTrack.c had no quickLift path at all. It now reads the alignments out of the
assembly the track came from through quickLiftSql and maps them onto the
reference. The chromFilter clause and the sort-and-filter tail moved into
helpers so the two loaders cannot drift apart.
bigBedTrack.c built the alignment out of the source interval but labelled it with
the reference sequence name and never lifted it, so a quickLifted bigPsl track
drew its items in the wrong place. It now takes the real source name out of the
bigBed, lifts, counts a failure the way the bed path does, and rewrites the
coordinate fields so a $chromStart in a mouseOver reports where the item is drawn.
cds.c asked the assembly on screen for the CDS and for the sequence the alignment
is to. Both belong to the assembly the alignment came from. This was already
the wrong answer for a quickLifted track on any target, and on a hub-backed
target it fails outright, because the name of such an assembly is not a database.
The table-exists answer is now remembered per assembly rather than once for the
process, since one page can hold both native and lifted alignment tracks.
The mRNA filter reads the same source assembly, for the same reason: the
accessions and the gbCdnaInfo ids are the source assembly's.
refs #38249
- lines changed 5, context: html, text, full: html, text
00811fcbcfd1c9a317b07477ff46a14dba046160 Wed Sep 9 12:39:44 2026 -0700
bigBed: use the file's own field count when the type line asks for more, refs #38310
A `type bigBed N` larger than the number of fields the file holds left the
track drawing nothing at all, and its details page reporting a disagreement
instead of the item. Fall back on the count in the file's header, which is
the count hubCheck already requires the type line to match.
The bound has to be the file's total field count and not its definedFieldCount.
Fifty-two tracks legitimately declare more bed fields than their header calls
defined, forty-eight of them the hs1 T2T_Encode_LOPeaks narrowPeak set, and
those are untouched.
Three tracks are in the over-declared state today, all `type bigBed 4` over a
three-field file: hg38 setDups, and the KAPA_HyperExome and
nexterarapidcapture subtracks of hg19 exomeProbesets. Their item boxes render
pixel-identically and their details pages, which failed before, now work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgTracks/cds.c
- lines changed 36, context: html, text, full: html, text
544f7054d812254a0883a412b98a08dcd78f1b84 Fri Sep 4 11:39:41 2026 -0700
hgTracks: draw quickLifted psl and bigPsl tracks
pslTrack.c had no quickLift path at all. It now reads the alignments out of the
assembly the track came from through quickLiftSql and maps them onto the
reference. The chromFilter clause and the sort-and-filter tail moved into
helpers so the two loaders cannot drift apart.
bigBedTrack.c built the alignment out of the source interval but labelled it with
the reference sequence name and never lifted it, so a quickLifted bigPsl track
drew its items in the wrong place. It now takes the real source name out of the
bigBed, lifts, counts a failure the way the bed path does, and rewrites the
coordinate fields so a $chromStart in a mouseOver reports where the item is drawn.
cds.c asked the assembly on screen for the CDS and for the sequence the alignment
is to. Both belong to the assembly the alignment came from. This was already
the wrong answer for a quickLifted track on any target, and on a hub-backed
target it fails outright, because the name of such an assembly is not a database.
The table-exists answer is now remembered per assembly rather than once for the
process, since one page can hold both native and lifted alignment tracks.
The mRNA filter reads the same source assembly, for the same reason: the
accessions and the gbCdnaInfo ids are the source assembly's.
refs #38249
- lines changed 5, context: html, text, full: html, text
02710f0107a6b154d9e5689165941ce724cb4c6a Fri Sep 4 13:38:54 2026 -0700
quickLift: fix the seams a second review pass found
quickLiftPslBackToProtein left two things wrong. pslTransMap can hand back
strand[0] == '-', because it reverse complements the input when the two
alignments disagree about the strand of the sequence they share, and forcing
strand[1] to '+' on top of that produced "-+". A protein psl is only ever "++"
or "+-", and pslShow reads strand[0] == '-' as "reverse complement the query", so
it would have reverse complemented a protein as though it were DNA. It now turns
the alignment over so the minus lands on the target side. qBaseInsert is in
nucleotides like everything else being divided, so it comes down too, and it
joins the divisibility guard: without it the result failed pslCheck and the
number was printed verbatim on the details page.
Adding that back-conversion made a comment in pslTrack.c false. The lift no
longer always returns an untranslated alignment, so the drawing code has to ask
rather than assume, the way bigBedTrack.c already did. A quickLifted protein psl
track was drawing every block at a third of its length. No such track exists on
hg19 or hg38 today, so this was latent.
The normalized score on the chain details page was read from the assembly on
screen. Where that assembly has no such table the page died; where it has a
table of the same name, which is the common case for a self or a well known
chain track, it silently returned some other assembly's chain and printed a blank
score. It now reads the assembly the chain came from, on a connection to it.
Two smaller things: htcBigPslAli guarded its connection with trackHubDatabase
alone, but a GenArk accession does not start with hub_, so it matches the guard
genericClickHandlerPlus already uses; and the table name tests in cds.c now skip
the hub prefix the way the ones in hgc.c were changed to, so a lifted refSeqAli
reaches its special case.
The chain item label took its start from the source chain and its strand
character from the lifted one. Both now come from the source chain.
refs #38249
- lines changed 226, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- lines changed 9, context: html, text, full: html, text
5bf1bd7150fc659d3a1bd9f34e4daad593728cb3 Fri Sep 11 17:55:11 2026 -0700
refGene never got the transcript codon number: the genbank CDS tables are in hgFixed, so their names are database-qualified and hTableExists could not see them. Use sqlTableExists, which can, refs #38298
- src/hg/hgTracks/cds.h
- lines changed 20, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- src/hg/hgTracks/chainTrack.c
- lines changed 146, context: html, text, full: html, text
fb55d9fe44a703a042d8ac6b586764819170f539 Fri Sep 4 12:48:13 2026 -0700
hgTracks: draw quickLifted chain and bigChain tracks
quickLiftChainLoadItems is reached from both chainLoadItems and
bigChainLoadItems. It reads the chains out of the assembly the track came from,
maps them onto the reference, and hangs the lifted blocks on the item directly,
so unlike the native loaders it does not leave the components for loadLinks to
fetch afterwards.
The link file comes from linkDataUrl when the track names one. Deriving it from
bigDataUrl, which is what bigChainGetLinkFile does, misses the tracks that do not
follow that naming: hg19's chainHs1 calls it hg19.chainHs1Link.bb, so nothing
loaded and the track came up empty.
The item label reports where the whole chain starts on the other species, taken
before the lift, since the lifted alignment only covers the window.
refs #38249
- lines changed 2, context: html, text, full: html, text
cec5ead054791f2a6601f308a56d280c08bd8ef7 Fri Sep 4 13:32:36 2026 -0700
quickLift: do not take the lift path on half a pair of settings
A hub can set quickLiftDb without setting quickLiftUrl, and nothing filters hub
trackDb settings. The alignment loaders gated on quickLiftDb alone, so such a
stanza took the lift path with no chain file and hgTracks died in
endsWith(NULL, ".bb") from bigChainGetLinkFile, by way of quickLiftLoadChains.
Verified: SIGSEGV in strlen from common.c:1653, page truncated mid-HTML. This is
reachable on a production browser now that bigChain and bigMaf are liftable,
because those carry their own bigDataUrl and so need no trustTrackDb.
quickLiftIsLifted requires both halves, and every gate now uses it, which also
settles the two different predicates that were in use for the same question.
quickLiftLoadChains returns an empty list for a NULL file as well, so the older
bed and genePred callers are covered whatever their gate does.
quickLiftSql now checks that a row has at least as many columns as the loader is
going to read. The native psl loader has always checked this, and the quickLift
path replacing it did not, so a table of the wrong type walked off the end of the
row; the psl caller now states the 21 columns it needs.
quickLiftMafs held a maf component name in a fixed buffer through safecpy, which
aborts rather than truncates, so a long name from a hub took hgTracks down. It
clones instead.
htcBigPslAliInWindow used a trackDb pointer its lookup can leave NULL, which its
sibling htcBigPslAli already checked for. And aliTrackParam formats a URL
parameter into a fixed buffer with safef, which aborts on a long one.
refs #38249
- lines changed 2, context: html, text, full: html, text
02710f0107a6b154d9e5689165941ce724cb4c6a Fri Sep 4 13:38:54 2026 -0700
quickLift: fix the seams a second review pass found
quickLiftPslBackToProtein left two things wrong. pslTransMap can hand back
strand[0] == '-', because it reverse complements the input when the two
alignments disagree about the strand of the sequence they share, and forcing
strand[1] to '+' on top of that produced "-+". A protein psl is only ever "++"
or "+-", and pslShow reads strand[0] == '-' as "reverse complement the query", so
it would have reverse complemented a protein as though it were DNA. It now turns
the alignment over so the minus lands on the target side. qBaseInsert is in
nucleotides like everything else being divided, so it comes down too, and it
joins the divisibility guard: without it the result failed pslCheck and the
number was printed verbatim on the details page.
Adding that back-conversion made a comment in pslTrack.c false. The lift no
longer always returns an untranslated alignment, so the drawing code has to ask
rather than assume, the way bigBedTrack.c already did. A quickLifted protein psl
track was drawing every block at a third of its length. No such track exists on
hg19 or hg38 today, so this was latent.
The normalized score on the chain details page was read from the assembly on
screen. Where that assembly has no such table the page died; where it has a
table of the same name, which is the common case for a self or a well known
chain track, it silently returned some other assembly's chain and printed a blank
score. It now reads the assembly the chain came from, on a connection to it.
Two smaller things: htcBigPslAli guarded its connection with trackHubDatabase
alone, but a GenArk accession does not start with hub_, so it matches the guard
genericClickHandlerPlus already uses; and the table name tests in cds.c now skip
the hub prefix the way the ones in hgc.c were changed to, so a lifted refSeqAli
reaches its special case.
The chain item label took its start from the source chain and its strand
character from the lifted one. Both now come from the source chain.
refs #38249
- src/hg/hgTracks/encode.c
- lines changed 10, context: html, text, full: html, text
96a903979dbe723e4f20a700dfc12881063b2c46 Tue Sep 8 00:26:18 2026 -0700
hgTracks: support mouseOver on bigNarrowPeak tracks, and register the peak filter tags
bigNarrowPeakLoadItems() had its own load loop and never looked at the
mouseOver setting, so a bigNarrowPeak track silently ignored it. It now uses
the mouseOverSetupForBbi() / mouseOverGetBbiText() helpers in mouseOver.c, which
also gets mouseOverField support for free. As with every other bigBed-like
track, the text only shows in pack or full, since dense makes no per-item map
boxes.
tagTypes.tab did not list bigNarrowPeak for mouseOver, scoreFilter,
scoreFilterLimits, scoreMin, scoreMax, signalFilter or signalFilterLimits, so
tdbQuery -strict rejected all of them even though the code reads them.
pValueFilter and qValueFilter, with their Limits, were not registered for any
type at all, although encodePeakCfgUi() in hui.c has always drawn them and
bigNarrowPeakLoadItems() has always applied them. Added.
refs #36210
- src/hg/hgTracks/hgTracks.c
- lines changed 38, context: html, text, full: html, text
d83026b7d8919bc8ad96679a98234aeb1308675e Mon Sep 7 10:55:13 2026 -0700
hgTracks: say in the track label why a track is showing item density
The density-mode labels existed but were only reachable through
labelTrackAsFilteredNumber(), which every caller guards with if (filtered),
so the note only appeared when a filter had also dropped features and the
automatic cases never said anything.
labelTrackAsDensityIfActive() now picks a message by cause: one for density
the user asked for on the configuration page, one for a window wider than
maxWindowCoverage, and one for the three paths that set limitWiggle because
there are too many features to draw. It is called once from makeActiveImage()
after every path into density mode has settled, over the tracks that will
actually be drawn, so hidden tracks keep their plain label.
refs #38279
- lines changed 1, context: html, text, full: html, text
e955e2f314ad9ca7fddad3fddc1f3c068fddd45e Tue Sep 8 00:20:03 2026 -0700
hgTracks: fix punctuation inconsistency in density-mode label message
Use a comma instead of a dash before 'zoom in' to match the style of
the neighboring too-many-items density label message.
- lines changed 8, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- lines changed 1, context: html, text, full: html, text
c69d3e1d9dedbcf44849aa6b25cf1a785e153bf1 Thu Sep 10 11:13:54 2026 -0700
hgTracks: turn on chromAlias names by default, refs #29201
The alias icon has been on for the RR since v498 through showAliases=on in
hg.conf on hgwdev, hgwbeta and the RR. The code default was still FALSE, so
mirrors and the GBiB never saw it. Flip the default so they do. A mirror can
still set showAliases=off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgTracks/hgTracks.h
- lines changed 9, context: html, text, full: html, text
d83026b7d8919bc8ad96679a98234aeb1308675e Mon Sep 7 10:55:13 2026 -0700
hgTracks: say in the track label why a track is showing item density
The density-mode labels existed but were only reachable through
labelTrackAsFilteredNumber(), which every caller guards with if (filtered),
so the note only appeared when a filter had also dropped features and the
automatic cases never said anything.
labelTrackAsDensityIfActive() now picks a message by cause: one for density
the user asked for on the configuration page, one for a window wider than
maxWindowCoverage, and one for the three paths that set limitWiggle because
there are too many features to draw. It is called once from makeActiveImage()
after every path into density mode has settled, over the tracks that will
actually be drawn, so hidden tracks keep their plain label.
refs #38279
- lines changed 5, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- src/hg/hgTracks/mafTrack.h
- lines changed 7, context: html, text, full: html, text
f2e67daf98177a40eea3e1d52364b52c4abf4728 Fri Sep 4 13:08:29 2026 -0700
hgTracks: draw quickLifted bigMaf and wigMaf tracks
quickLiftLoadMafs is reached from both places a maf track loads its blocks. It
reads them over the ranges in the other assembly through the existing readers and
maps them onto the reference.
A lifted maf cannot use its summary. The summary names a table or a file in the
assembly the track came from, so it cannot be queried with reference coordinates,
and for a hub track the setting comes back rewritten as a path under the hub
besides, which is how it was failing: a range query against
../trash/quickLift/NNN/multiz100waySummary. inSummaryMode now says no for a
lifted track and the real blocks are read instead.
That costs time at a wide window, since reading the summary is exactly the work a
maf track avoids that way. Measured on multiz100way, hg19 lifted onto hg38: at
900 kb, where both read real blocks, native is 1.52s and lifted is 1.09s, so
nothing about the lift is slow. At 2.5 Mb, where native reads the summary,
native is 1.05s and lifted is 7.0s. Lifting the summary is the fix and wants its
own pass: a summary row is a coordinate range and a score, so mapping one is
liftOverRemapRange, but it is read in the drawing functions rather than in a
loader.
refs #38249
- lines changed 5, context: html, text, full: html, text
cec5ead054791f2a6601f308a56d280c08bd8ef7 Fri Sep 4 13:32:36 2026 -0700
quickLift: do not take the lift path on half a pair of settings
A hub can set quickLiftDb without setting quickLiftUrl, and nothing filters hub
trackDb settings. The alignment loaders gated on quickLiftDb alone, so such a
stanza took the lift path with no chain file and hgTracks died in
endsWith(NULL, ".bb") from bigChainGetLinkFile, by way of quickLiftLoadChains.
Verified: SIGSEGV in strlen from common.c:1653, page truncated mid-HTML. This is
reachable on a production browser now that bigChain and bigMaf are liftable,
because those carry their own bigDataUrl and so need no trustTrackDb.
quickLiftIsLifted requires both halves, and every gate now uses it, which also
settles the two different predicates that were in use for the same question.
quickLiftLoadChains returns an empty list for a NULL file as well, so the older
bed and genePred callers are covered whatever their gate does.
quickLiftSql now checks that a row has at least as many columns as the loader is
going to read. The native psl loader has always checked this, and the quickLift
path replacing it did not, so a table of the wrong type walked off the end of the
row; the psl caller now states the 21 columns it needs.
quickLiftMafs held a maf component name in a fixed buffer through safecpy, which
aborts rather than truncates, so a long name from a hub took hgTracks down. It
clones instead.
htcBigPslAliInWindow used a trackDb pointer its lookup can leave NULL, which its
sibling htcBigPslAli already checked for. And aliTrackParam formats a URL
parameter into a fixed buffer with safef, which aborts on a long one.
refs #38249
- src/hg/hgTracks/mainMain.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- lines changed 1, context: html, text, full: html, text
cf9f4cb7f55c7beb8ad5f11118656a60770a71a5 Thu Sep 10 01:02:36 2026 -0700
Move the extra-HTTP-header list into cheapcgi, and write the header only once
Follow-on to the cgiPrintContentType() refactor.
cart.c owned the mechanism for adding headers ahead of the content type: a
global slPair list plus addHttpHeaders() to print it. That put it in hg/lib,
out of reach of the CGIs and library code that do not use a cart, even though
nothing about it is cart-specific. It now lives next to cgiPrintContentType()
in lib/cheapcgi.c, behind cgiAddHttpHeader(name, value) instead of a bare
global, and cgiPrintContentType() writes the queued headers itself. The one
caller, hgTracks/mainMain.c, reads the same but no longer reaches into cart.h
for it. cspWriteResponseHeader() stays in hg/lib where it belongs, since it
needs hg.conf; cartWriteHeaderAndCont() calls it directly now, the way the
other ten callers already do.
cgiPrintContentType() also writes at most once per process now. A second
content type cannot reach the browser as a header - it lands in the page body
as text - so the later caller is always the mistaken one. cart.c had a private
cartDidContentType flag for exactly this, covering only the flows that went
through the cart; the guard is now in the one function every flow shares, and
cartDidContentType is gone. Its public equivalent, cgiDidContentType(), is
what cartWriteHeaderAndCont() checks so it does not write a second cookie.
Verified: make libs, make cgi and the lib test suite are clean, hgTracks still
emits Cache-Control: no-store, and hgTracks, hgc and hgTables each emit exactly
one Content-Type on both their html and their text paths.
- src/hg/hgTracks/makefile
- lines changed 2, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- lines changed 7, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/hgTracks/makefile.hgRenderTracks
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/hgTracks/menu.c
- lines changed 3, context: html, text, full: html, text
57490a621bd10c9c7810bdf474b7c801027d237d Thu Sep 10 06:56:54 2026 -0700
Use one blue for the menu bar on every page, refs #38206
The bar came in three shades: #2636d1 from nice_menu.css on hgTables,
hgBlat, hgc and the other cart CGIs, #00457c on hgTracks through its own
set of ids, and #003a72 on the gateway and the static pages, which
override nice_menu.css afterwards. They are all #003a72 now, the house
deep navy already used for the gateway banner, the footer and the
buttons.
hgTracks no longer rewrites main-menu-whole and home-link into its own
ids, so its bar and its house icon come from the same rules as everyone
else's, and hgTracksHomeIconSprite.png goes away - the shared sprite is
white on a transparent background and sits on whatever blue the bar has.
gbStatic.css keeps its own copy of the color on purpose; the comment
there says why.
- src/hg/hgTracks/myVariantsTrack.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgTracks/netTrack.c
- lines changed 11, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 4, context: html, text, full: html, text
f29b65452a1cca8a5bc62f007100b313e589d036 Sun Sep 6 15:11:36 2026 -0700
quickLift: lift a bigNet track, refs #20824
A net is not an ordinary track to lift. It is an alignment of two assemblies,
so only its target side moves to the new reference; the query side names a third
assembly and is carried across untouched. And the browser draws a net by
recursion, so what really has to survive the lift is the tree: a row's level only
means anything relative to the row above it.
chainNetLoadRangeQuickLift() maps each row's target range with
quickLiftIntervalsToBedClip, which is the same code every other quickLift track
uses, so a net row lands where a bed of the same span would. The surviving rows
then go to the same helpToNet() the unlifted path uses, so there is one tree
builder and not two. The row collection either path does is now cnlHelperNew and
cnlHelperAddBigNet.
validateOneTdb lets bigNet into a quickLift hub. Its chain track is not offered,
so a lifted net's details page has no chain to follow and says so rather than
looking for a track this assembly does not have. bigNetLoadOne lifts the same
way for that page, unclipped, so it reports the item's whole extent.
bigNetFromInterval passes -1 as the cached chromId. bbiCachedChromLookup leaves
the buffer alone when the id matches the one before it, so a cache that outlives
the buffer hands back stale bytes.
Measured against the standalone liftOver tool, on hg19 chr22's mouse net lifted
to hg38: 36 of 36 source rows in the sampled window land on the same coordinates,
counting the one liftOver will not take whole, whose two ends it does place
exactly where the browser puts them. Against hg38's own mouse net, computed
independently, 1.02% of the drawn pixels differ.
- src/hg/hgTracks/pslTrack.c
- lines changed 114, context: html, text, full: html, text
544f7054d812254a0883a412b98a08dcd78f1b84 Fri Sep 4 11:39:41 2026 -0700
hgTracks: draw quickLifted psl and bigPsl tracks
pslTrack.c had no quickLift path at all. It now reads the alignments out of the
assembly the track came from through quickLiftSql and maps them onto the
reference. The chromFilter clause and the sort-and-filter tail moved into
helpers so the two loaders cannot drift apart.
bigBedTrack.c built the alignment out of the source interval but labelled it with
the reference sequence name and never lifted it, so a quickLifted bigPsl track
drew its items in the wrong place. It now takes the real source name out of the
bigBed, lifts, counts a failure the way the bed path does, and rewrites the
coordinate fields so a $chromStart in a mouseOver reports where the item is drawn.
cds.c asked the assembly on screen for the CDS and for the sequence the alignment
is to. Both belong to the assembly the alignment came from. This was already
the wrong answer for a quickLifted track on any target, and on a hub-backed
target it fails outright, because the name of such an assembly is not a database.
The table-exists answer is now remembered per assembly rather than once for the
process, since one page can hold both native and lifted alignment tracks.
The mRNA filter reads the same source assembly, for the same reason: the
accessions and the gbCdnaInfo ids are the source assembly's.
refs #38249
- lines changed 2, context: html, text, full: html, text
cec5ead054791f2a6601f308a56d280c08bd8ef7 Fri Sep 4 13:32:36 2026 -0700
quickLift: do not take the lift path on half a pair of settings
A hub can set quickLiftDb without setting quickLiftUrl, and nothing filters hub
trackDb settings. The alignment loaders gated on quickLiftDb alone, so such a
stanza took the lift path with no chain file and hgTracks died in
endsWith(NULL, ".bb") from bigChainGetLinkFile, by way of quickLiftLoadChains.
Verified: SIGSEGV in strlen from common.c:1653, page truncated mid-HTML. This is
reachable on a production browser now that bigChain and bigMaf are liftable,
because those carry their own bigDataUrl and so need no trustTrackDb.
quickLiftIsLifted requires both halves, and every gate now uses it, which also
settles the two different predicates that were in use for the same question.
quickLiftLoadChains returns an empty list for a NULL file as well, so the older
bed and genePred callers are covered whatever their gate does.
quickLiftSql now checks that a row has at least as many columns as the loader is
going to read. The native psl loader has always checked this, and the quickLift
path replacing it did not, so a table of the wrong type walked off the end of the
row; the psl caller now states the 21 columns it needs.
quickLiftMafs held a maf component name in a fixed buffer through safecpy, which
aborts rather than truncates, so a long name from a hub took hgTracks down. It
clones instead.
htcBigPslAliInWindow used a trackDb pointer its lookup can leave NULL, which its
sibling htcBigPslAli already checked for. And aliTrackParam formats a URL
parameter into a fixed buffer with safef, which aborts on a long one.
refs #38249
- lines changed 4, context: html, text, full: html, text
02710f0107a6b154d9e5689165941ce724cb4c6a Fri Sep 4 13:38:54 2026 -0700
quickLift: fix the seams a second review pass found
quickLiftPslBackToProtein left two things wrong. pslTransMap can hand back
strand[0] == '-', because it reverse complements the input when the two
alignments disagree about the strand of the sequence they share, and forcing
strand[1] to '+' on top of that produced "-+". A protein psl is only ever "++"
or "+-", and pslShow reads strand[0] == '-' as "reverse complement the query", so
it would have reverse complemented a protein as though it were DNA. It now turns
the alignment over so the minus lands on the target side. qBaseInsert is in
nucleotides like everything else being divided, so it comes down too, and it
joins the divisibility guard: without it the result failed pslCheck and the
number was printed verbatim on the details page.
Adding that back-conversion made a comment in pslTrack.c false. The lift no
longer always returns an untranslated alignment, so the drawing code has to ask
rather than assume, the way bigBedTrack.c already did. A quickLifted protein psl
track was drawing every block at a third of its length. No such track exists on
hg19 or hg38 today, so this was latent.
The normalized score on the chain details page was read from the assembly on
screen. Where that assembly has no such table the page died; where it has a
table of the same name, which is the common case for a self or a well known
chain track, it silently returned some other assembly's chain and printed a blank
score. It now reads the assembly the chain came from, on a connection to it.
Two smaller things: htcBigPslAli guarded its connection with trackHubDatabase
alone, but a GenArk accession does not start with hub_, so it matches the guard
genericClickHandlerPlus already uses; and the table name tests in cds.c now skip
the hub prefix the way the ones in hgc.c were changed to, so a lifted refSeqAli
reaches its special case.
The chain item label took its start from the source chain and its strand
character from the lifted one. Both now come from the source chain.
refs #38249
- src/hg/hgTracks/simpleTracks.c
- lines changed 5, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 76, context: html, text, full: html, text
c3322bc2d3f94a59721989762c58db1734ec7740 Mon Sep 7 20:04:43 2026 -0700
hgTracks: show the cDNA and codon range of an exon in its mouseover instead of "Zoom in to show cDNA position"
At gene-level zoom the exon popup only said "Codons: Zoom in to show cDNA
position", so the only way to find a c. or p. position was to zoom into one exon
after another. The popup now gives the exon's HGVS c. range and the codons it
spans, e.g. "Codons: c.1364-1482 (p.455-494)", plus the c.-N / c.*N range of any
UTR part of the exon. The numbers agree with the per-codon popups shown when
zoomed in.
Non-coding transcripts already got an n. range but only at codon-level zoom,
where the number is of little use; that gate is gone, so they are labelled at
every zoom too. Chain and LRG tracks have no cDNA coordinates and are left
alone.
refs #38278
- lines changed 34, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- lines changed 4, context: html, text, full: html, text
e2d86e5b1db3e603f503f58db38009fa31840dd5 Wed Sep 9 12:37:09 2026 -0700
hgTracks: don't read past the end of exonFrames on a transcript's last exon
The exon mouseover works out the codon phase at each end of an exon. The end
phase is the frame of the next exon along the transcript. On the last exon of
a forward-strand transcript there is no next exon, and the index has reached
the number of exons, so the read was one element past the end of the array.
The reverse-strand branch already guarded the same case at its own end of the
transcript.
Nothing the reader sees changes. makeExonFrameText prints an end phase only
when the exon is not the last one, so the value read here was always thrown
away. Measured on two builds from this tree, patched and not: 591 codon-phase
tooltips at chr12:459,900-462,400 are byte for byte the same, while valgrind
reports the invalid read in the unpatched build and none in this one.
refs #38309
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgTracks/wigMafTrack.c
- lines changed 50, context: html, text, full: html, text
f2e67daf98177a40eea3e1d52364b52c4abf4728 Fri Sep 4 13:08:29 2026 -0700
hgTracks: draw quickLifted bigMaf and wigMaf tracks
quickLiftLoadMafs is reached from both places a maf track loads its blocks. It
reads them over the ranges in the other assembly through the existing readers and
maps them onto the reference.
A lifted maf cannot use its summary. The summary names a table or a file in the
assembly the track came from, so it cannot be queried with reference coordinates,
and for a hub track the setting comes back rewritten as a path under the hub
besides, which is how it was failing: a range query against
../trash/quickLift/NNN/multiz100waySummary. inSummaryMode now says no for a
lifted track and the real blocks are read instead.
That costs time at a wide window, since reading the summary is exactly the work a
maf track avoids that way. Measured on multiz100way, hg19 lifted onto hg38: at
900 kb, where both read real blocks, native is 1.52s and lifted is 1.09s, so
nothing about the lift is slow. At 2.5 Mb, where native reads the summary,
native is 1.05s and lifted is 7.0s. Lifting the summary is the fix and wants its
own pass: a summary row is a coordinate range and a score, so mapping one is
liftOverRemapRange, but it is read in the drawing functions rather than in a
loader.
refs #38249
- lines changed 2, context: html, text, full: html, text
cec5ead054791f2a6601f308a56d280c08bd8ef7 Fri Sep 4 13:32:36 2026 -0700
quickLift: do not take the lift path on half a pair of settings
A hub can set quickLiftDb without setting quickLiftUrl, and nothing filters hub
trackDb settings. The alignment loaders gated on quickLiftDb alone, so such a
stanza took the lift path with no chain file and hgTracks died in
endsWith(NULL, ".bb") from bigChainGetLinkFile, by way of quickLiftLoadChains.
Verified: SIGSEGV in strlen from common.c:1653, page truncated mid-HTML. This is
reachable on a production browser now that bigChain and bigMaf are liftable,
because those carry their own bigDataUrl and so need no trustTrackDb.
quickLiftIsLifted requires both halves, and every gate now uses it, which also
settles the two different predicates that were in use for the same question.
quickLiftLoadChains returns an empty list for a NULL file as well, so the older
bed and genePred callers are covered whatever their gate does.
quickLiftSql now checks that a row has at least as many columns as the loader is
going to read. The native psl loader has always checked this, and the quickLift
path replacing it did not, so a table of the wrong type walked off the end of the
row; the psl caller now states the 21 columns it needs.
quickLiftMafs held a maf component name in a fixed buffer through safecpy, which
aborts rather than truncates, so a long name from a hub took hgTracks down. It
clones instead.
htcBigPslAliInWindow used a trackDb pointer its lookup can leave NULL, which its
sibling htcBigPslAli already checked for. And aliTrackParam formats a URL
parameter into a fixed buffer with safef, which aborts on a long one.
refs #38249
- src/hg/hgc/bigBedClick.c
- lines changed 74, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- lines changed 26, context: html, text, full: html, text
905b9cb05eeaca7f2dcda42fc6abdb95a2d2da7f Wed Sep 9 05:29:28 2026 -0700
no captcha for a command-line CGI run, and version the detailsScript module URL
Two small fixes to things noticed while adding the scatterPlot plot type.
A CGI run from the command line got the Cloudflare Turnstile challenge page
instead of the output the caller asked for, which makes "./hgc db=hg38 g=x" -
the quickest way to see what a CGI emits - useless without a hand-made hg.conf.
There is no browser to solve a captcha in that situation. printCaptcha() now
returns early when cgiWasSpoofed(). That flag cannot be set from an HTTP
request: cgiFromCommandLine() returns early and leaves it FALSE whenever the
web server has set REQUEST_METHOD. Checked that a plain argument-style run is
now clean, that a run which fakes the web environment with QUERY_STRING still
gets the captcha, and that an HTTP request behaves exactly as the unmodified
binary does.
The detailsScript module was loaded from a hardcoded import('../js/hgc.X.js'),
bypassing webTimeStampedLinkToResource(), so it was the one script on the page
with no ?v=<mtime>. That is the mechanism that flushes a browser's cache when
the CGI version changes and that keeps a mirror from pairing an old static file
with new CGIs, and without it a cached module could be handed newer bedDetails
JSON than it was written for. Now built through the helper, which also fixes the
already-shipped histogram type. The helper errAborts on a missing file and the
plot type comes from a hub, so a plot type with no module installed falls back to
the plain path: a silent failed import as before, rather than one bad hub setting
taking down the whole details page.
refs #35415
- lines changed 5, context: html, text, full: html, text
00811fcbcfd1c9a317b07477ff46a14dba046160 Wed Sep 9 12:39:44 2026 -0700
bigBed: use the file's own field count when the type line asks for more, refs #38310
A `type bigBed N` larger than the number of fields the file holds left the
track drawing nothing at all, and its details page reporting a disagreement
instead of the item. Fall back on the count in the file's header, which is
the count hubCheck already requires the type line to match.
The bound has to be the file's total field count and not its definedFieldCount.
Fifty-two tracks legitimately declare more bed fields than their header calls
defined, forty-eight of them the hs1 T2T_Encode_LOPeaks narrowPeak set, and
those are untouched.
Three tracks are in the over-declared state today, all `type bigBed 4` over a
three-field file: hg38 setDups, and the KAPA_HyperExome and
nexterarapidcapture subtracks of hg19 exomeProbesets. Their item boxes render
pixel-identically and their details pages, which failed before, now work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgc/hgc.c
- lines changed 320, context: html, text, full: html, text
ee190bc09015bf1d10254691824b0a38f59e1aff Fri Sep 4 11:39:56 2026 -0700
hgc: show the lifted alignment on a quickLifted psl or bigPsl details page
The details page and its alignment views all read the assembly on screen, so a
quickLifted alignment track either reported the position it had before the lift
or, for the base alignment views, could not find its item at all.
Finding the track was the first problem. aliTable is the table name from the
assembly the alignments came from, and that name usually belongs to a real table
on the assembly being viewed as well, so it cannot tell the two apart. The
alignment links now carry aliTrack, the track hgc was called on, built the same
way hgcAnchorSomewhere builds its table parameter. A hub track's trackDb only
reaches trackHash if its hub is attached in cartDoMiddle, which was already being
done for the two htcBigPsl commands and now covers the cdna and protein ones too.
quickLiftAliInfo resolves the trackDb, assembly, table and chain file behind an
aliTable and aliTrack pair, custom tracks included. quickLiftFindPsl finds the
alignment a link points at: only the reference position is known there and the
lift does not run backwards, so it reads every alignment of the accession out of
the other assembly, lifts them, and keeps the one that lands on that position.
htcCdnaAli and htcCdnaAliInWindow use that, and read the CDS and the query
sequence from the source assembly. htcBigPslAli and htcBigPslAliInWindow read
the source intervals through quickLiftGetIntervals and match on the lifted
position rather than on the raw interval. Several name tests that decide what
kind of alignment this is now skip the hub prefix, so the guard against a
translated alignment in a window is not bypassed on a lifted xeno track.
Separately, and not specific to quickLift: genericBigPslClick read through an
empty alignment list, which is what happens whenever nothing in the window
matches the item asked for.
refs #38249
- lines changed 65, context: html, text, full: html, text
8fb78633bc427f68f8d2c6b1a75d8093883a0b80 Fri Sep 4 12:48:23 2026 -0700
hgc: show the lifted chain on a quickLifted chain track's details page
chainLoadItemInRange is the one place all four chain detail consumers get their
chain, so lifting there covers the details page, the base by base alignment, its
translated variant, and quickLift's own difference page at once.
quickLiftChainInRange loads every chain over the source ranges and matches on id
rather than asking for the one id. The chain's sequence name in the other
assembly is not known at that point, and both by-id loaders abort when the id is
not inside the range they were handed.
The page used to end by saying the fields above describe the entire chain rather
than the part in the window. For a lifted chain that is not true, since only the
part around the window is ever worked out, so it now says that instead. Native
chain tracks keep the original sentence.
refs #38249
- lines changed 5, context: html, text, full: html, text
cec5ead054791f2a6601f308a56d280c08bd8ef7 Fri Sep 4 13:32:36 2026 -0700
quickLift: do not take the lift path on half a pair of settings
A hub can set quickLiftDb without setting quickLiftUrl, and nothing filters hub
trackDb settings. The alignment loaders gated on quickLiftDb alone, so such a
stanza took the lift path with no chain file and hgTracks died in
endsWith(NULL, ".bb") from bigChainGetLinkFile, by way of quickLiftLoadChains.
Verified: SIGSEGV in strlen from common.c:1653, page truncated mid-HTML. This is
reachable on a production browser now that bigChain and bigMaf are liftable,
because those carry their own bigDataUrl and so need no trustTrackDb.
quickLiftIsLifted requires both halves, and every gate now uses it, which also
settles the two different predicates that were in use for the same question.
quickLiftLoadChains returns an empty list for a NULL file as well, so the older
bed and genePred callers are covered whatever their gate does.
quickLiftSql now checks that a row has at least as many columns as the loader is
going to read. The native psl loader has always checked this, and the quickLift
path replacing it did not, so a table of the wrong type walked off the end of the
row; the psl caller now states the 21 columns it needs.
quickLiftMafs held a maf component name in a fixed buffer through safecpy, which
aborts rather than truncates, so a long name from a hub took hgTracks down. It
clones instead.
htcBigPslAliInWindow used a trackDb pointer its lookup can leave NULL, which its
sibling htcBigPslAli already checked for. And aliTrackParam formats a URL
parameter into a fixed buffer with safef, which aborts on a long one.
refs #38249
- lines changed 21, context: html, text, full: html, text
02710f0107a6b154d9e5689165941ce724cb4c6a Fri Sep 4 13:38:54 2026 -0700
quickLift: fix the seams a second review pass found
quickLiftPslBackToProtein left two things wrong. pslTransMap can hand back
strand[0] == '-', because it reverse complements the input when the two
alignments disagree about the strand of the sequence they share, and forcing
strand[1] to '+' on top of that produced "-+". A protein psl is only ever "++"
or "+-", and pslShow reads strand[0] == '-' as "reverse complement the query", so
it would have reverse complemented a protein as though it were DNA. It now turns
the alignment over so the minus lands on the target side. qBaseInsert is in
nucleotides like everything else being divided, so it comes down too, and it
joins the divisibility guard: without it the result failed pslCheck and the
number was printed verbatim on the details page.
Adding that back-conversion made a comment in pslTrack.c false. The lift no
longer always returns an untranslated alignment, so the drawing code has to ask
rather than assume, the way bigBedTrack.c already did. A quickLifted protein psl
track was drawing every block at a third of its length. No such track exists on
hg19 or hg38 today, so this was latent.
The normalized score on the chain details page was read from the assembly on
screen. Where that assembly has no such table the page died; where it has a
table of the same name, which is the common case for a self or a well known
chain track, it silently returned some other assembly's chain and printed a blank
score. It now reads the assembly the chain came from, on a connection to it.
Two smaller things: htcBigPslAli guarded its connection with trackHubDatabase
alone, but a GenArk accession does not start with hub_, so it matches the guard
genericClickHandlerPlus already uses; and the table name tests in cds.c now skip
the hub prefix the way the ones in hgc.c were changed to, so a lifted refSeqAli
reaches its special case.
The chain item label took its start from the source chain and its strand
character from the lifted one. Both now come from the source chain.
refs #38249
- lines changed 92, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 55, context: html, text, full: html, text
f29b65452a1cca8a5bc62f007100b313e589d036 Sun Sep 6 15:11:36 2026 -0700
quickLift: lift a bigNet track, refs #20824
A net is not an ordinary track to lift. It is an alignment of two assemblies,
so only its target side moves to the new reference; the query side names a third
assembly and is carried across untouched. And the browser draws a net by
recursion, so what really has to survive the lift is the tree: a row's level only
means anything relative to the row above it.
chainNetLoadRangeQuickLift() maps each row's target range with
quickLiftIntervalsToBedClip, which is the same code every other quickLift track
uses, so a net row lands where a bed of the same span would. The surviving rows
then go to the same helpToNet() the unlifted path uses, so there is one tree
builder and not two. The row collection either path does is now cnlHelperNew and
cnlHelperAddBigNet.
validateOneTdb lets bigNet into a quickLift hub. Its chain track is not offered,
so a lifted net's details page has no chain to follow and says so rather than
looking for a track this assembly does not have. bigNetLoadOne lifts the same
way for that page, unclipped, so it reports the item's whole extent.
bigNetFromInterval passes -1 as the cached chromId. bbiCachedChromLookup leaves
the buffer alone when the id matches the one before it, so a cache that outlives
the buffer hands back stale bytes.
Measured against the standalone liftOver tool, on hg19 chr22's mouse net lifted
to hg38: 36 of 36 source rows in the sampled window land on the same coordinates,
counting the one liftOver will not take whole, whose two ends it does place
exactly where the browser puts them. Against hg38's own mouse net, computed
independently, 1.02% of the drawn pixels differ.
- lines changed 35, context: html, text, full: html, text
357d4dbeca6b3bbb59f60185f6b833d74fd74fbc Sun Sep 6 15:51:56 2026 -0700
bigNet: four fixes from the code review, refs #20824
A quickLifted net's details page lifted the row unclipped so it could report
the whole extent, but the image lifts clipped. An item too big for the chains
loaded in the window is drawn clipped and was then unfindable on click, which
puts a box on screen that says it is not there. Try the unclipped lift, fall
back to the clipped one, and say plainly when the numbers describe only the
part that could be placed.
quickLiftGetIntervals can return one source row twice, through two chains whose
padded query ranges overlap. helpToNet cannot tell two identical parents apart:
the second inherits no children and then draws as one solid box over the first
one's gaps. A level, a target range and a chain id name a row in a net, so that
is enough to recognize the repeat and drop it. Preventive -- no duplicate was
observed in the window measured.
The sentence explaining why a lifted net has no alignment to show printed
quickLiftDb twice, and a hub can set quickLiftUrl and leave quickLiftDb unset,
so it could be handed a null. One printf, and it reads correctly either way.
Free the per-row bed in both lift loops. It is about ninety thousand of them on
a whole chromosome, which is more than a CGI should be asked to shrug off.
Rendering is unchanged: the unlifted net still draws pixel for pixel like the
native netAlign track at three widths, every lifted figure but the details page
is pixel-identical to the one built before these fixes, and the 36 of 36
agreement with the standalone liftOver tool is unchanged.
- lines changed 2, context: html, text, full: html, text
e9ed25747e3d8ab10963306b826f7cedc5e71897 Mon Sep 7 11:35:30 2026 -0700
Merge branch 'quickLiftAlign38249' -- alignment tracks in quickLift, refs #38249
# Conflicts:
# src/hg/lib/trackHub.c
- lines changed 6, context: html, text, full: html, text
c0e8fa6df3a0bd406c4188d49ee00f20aef203e5 Mon Sep 7 12:07:18 2026 -0700
Substitute trackDb variables in hub track description pages
A hub's description page comes straight off the hub's web server and has
never been through variable substitution, so a $db or $parentTrack in it
reached the reader as literal text. Native trackDb pages are fine, since
hgTrackDb substitutes them when it loads trackDb, but there was no
equivalent step for a hub.
hgc's getTrackHtml and hgTrackUi's trackUi both call hVarSubstTrackDbHtml
on a hub track's html. Only a short list of variables is recognized there and nothing is an
error, because a hub page written before this existed can easily contain
a dollar sign inside a shell example, and silently rewriting that would
be worse than not substituting at all.
Adds $parentTrack, the name of the container a track sits in, which is
what a subtrack description page needs to link back to its superTrack or
composite. Views are skipped, since a view has no page of its own, and
the hub_<id>_ prefix is kept so the name works as hgTrackUi's g=
parameter. Documents $track, $parentTrack and $hgsid in trackDb/README.
refs #37599
- lines changed 2, context: html, text, full: html, text
c1ac48b0feb9a8ff0f0bb16d7aff15d417d6772b Tue Sep 8 12:38:42 2026 -0700
Share dialog: scope the snapshot-lifetime wording to the BLAT alignment share only. Feedback from CR. refs #37996
The softer durability text (link valid for years, save into a Session for permanence) keyed
on the generic url mode, so hgTrackUi's page-share link and the hgc item-popup share, both
plain non-expiring URLs, showed misleading copy. The wording is now behind an explicit
snapshot flag that only the BLAT alignment page's share passes; every other caller keeps
"Links never time out". Also removes a stray blank line in hgc.c from the same review.
- lines changed 6, context: html, text, full: html, text
5a249cd50f592a3b7598110eec792cfc942fab3f Wed Sep 9 08:52:06 2026 -0700
Address the v504 code review
hgSession: an anonymous share name that arrives with the request is saved only
when it is not already in the table. Every anonymous link sits under the one
reserved user "l", so a name already there stays as it is and the caller is told
so. The top-right Share dialog is unaffected, since it passes a name it has
just reserved and such a name does not exist yet. Snapshot names are now left
out of both My Sessions listings, which is what their "__" prefix has claimed
all along.
Share dialog: "Create link & copy" reports the copy instead of promising it.
copyToClipboard says whether the text reached the clipboard, the dialog passes
that on when a browser refuses, and it tries the asynchronous clipboard API
before giving up. The preview is built with the same encoding the server uses,
so a name holding a hyphen or a slash previews as the link that really gets
made. Cancelling out of the name editor no longer copies a second time, and a
reply with no link in it says so rather than showing "undefined".
hgBlat: a second click on the share button while the first request is still out
no longer mints a second snapshot session, and a box dismissed during the wait
stays closed.
Also: a snapshot moves a cart value into durable storage only when it is a
trash path, the way sessionData's own callers check; sqlAddressMatch keeps to
its own documented precondition when handed an empty address; alphaGenomeToWig
compares its output with its input rather than with itself, rejects a position
that is not all digits and skips an empty score; and hgc's default iframe width
reaches the browser as one percent sign.
refs #38294
- lines changed 4, context: html, text, full: html, text
a1c0911798f9e50f0af79db624b8de840af4854f Wed Sep 9 13:02:54 2026 -0700
hgc: zero the stack refLink in doKnownGene so the details page stops reading uninitialised memory
doKnownGene builds a refLink two ways. When the accession contains NM_ it
loads a real row. Otherwise it uses a plain struct on the stack and fills in
three of the eight fields. A struct declared that way is not zeroed, and two
of the remaining fields are read further down: prKnownGeneInfo tests omimId and
prints an OMIM link built from it, and geneShowPosAndLinksPal passes protAcc to
hGenBankHaveSeq as a string.
No knownGene name on hg38, hg19 or mm39 contains NM_, so every knownGene
details page on those assemblies takes the else branch. The OMIM link appears
and disappears between requests to the same URL and carries a meaningless
number when it does. The protAcc read is worse: run hgc from the command line
on such a page and it segfaults in strlen, reached through checkIfInTable.
Zero the struct before filling it in. The fields that are not set then read as
absent, which is what the else branch already means. Also guard the
hGenBankHaveSeq call on a NULL pepName, because a NULL string argument to
sqlSafef reaches sqlCheckError, whose default level is abort.
Note for anyone moving code between these two functions later: prRefGeneInfo
dereferences rl->product, and doKnownGene does not call it. A zeroed struct
would be a NULL dereference there.
Found with valgrind, refs #38316.
refs #38317
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hgc/hgc.h
- lines changed 3, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 3, context: html, text, full: html, text
f29b65452a1cca8a5bc62f007100b313e589d036 Sun Sep 6 15:11:36 2026 -0700
quickLift: lift a bigNet track, refs #20824
A net is not an ordinary track to lift. It is an alignment of two assemblies,
so only its target side moves to the new reference; the query side names a third
assembly and is carried across untouched. And the browser draws a net by
recursion, so what really has to survive the lift is the tree: a row's level only
means anything relative to the row above it.
chainNetLoadRangeQuickLift() maps each row's target range with
quickLiftIntervalsToBedClip, which is the same code every other quickLift track
uses, so a net row lands where a bed of the same span would. The surviving rows
then go to the same helpToNet() the unlifted path uses, so there is one tree
builder and not two. The row collection either path does is now cnlHelperNew and
cnlHelperAddBigNet.
validateOneTdb lets bigNet into a quickLift hub. Its chain track is not offered,
so a lifted net's details page has no chain to follow and says so rather than
looking for a track this assembly does not have. bigNetLoadOne lifts the same
way for that page, unclipped, so it reports the item's whole extent.
bigNetFromInterval passes -1 as the cached chromId. bbiCachedChromLookup leaves
the buffer alone when the id matches the one before it, so a cache that outlives
the buffer hands back stale bytes.
Measured against the standalone liftOver tool, on hg19 chr22's mouse net lifted
to hg38: 36 of 36 source rows in the sampled window land on the same coordinates,
counting the one liftOver will not take whole, whose two ends it does place
exactly where the browser puts them. Against hg38's own mouse net, computed
independently, 1.02% of the drawn pixels differ.
- lines changed 8, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- src/hg/hgc/mafClick.c
- lines changed 48, context: html, text, full: html, text
15e8651da02e09bbd37dbb404a966ed8c7d75167 Fri Sep 4 13:32:47 2026 -0700
hgc: show the lifted alignment on a quickLifted maf track's details page
The page read the track's own file with reference coordinates and then went
looking for a reference row named for the assembly on screen. Both kinds of maf
track failed, differently: bigMaf aborted with "Couldn't find hg38.chr7 in maf"
out of mafFindComponent, and wigMaf drew a page with a title and nothing under
it, which is the worse of the two.
quickLiftClickMafs reads the blocks over the ranges in the other assembly, on
connections to that assembly, and maps them onto the reference, the same way the
track loader does.
refs #38249
- src/hg/htdocs/FAQ/FAQformat.html
- lines changed 1, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 7, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/FAQ/FAQgenes.html
- lines changed 59, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- lines changed 1, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/FAQ/FAQlink.html
- lines changed 1, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/assemblySearch.html
- lines changed 1, context: html, text, full: html, text
562b1b24f9d7cf5157c799f733ba219e2e9f7da9 Wed Sep 9 09:06:30 2026 -0700
Assembly search page uses the shared copyToClipboard instead of its own copy
The page carried a copy of copyToClipboard marked "borrowed this code from
utils.js", and the two had already drifted apart: the fix that stops the button
claiming a copy that a browser refused went into one and not the other. The
page now loads utils.js, as nine other static pages already do, and its own
copy is gone. jquery is already loaded by the page header, so nothing else was
needed.
The page also declared a global named debug, which utils.js declares too. Both
start out false and the two uses in utils.js are in functions this page never
calls, so nothing was broken, but the page flag is now searchDebug. The debug
URL parameter and stateObject.debug keep their names.
refs #38294
- src/hg/htdocs/data/recTrackSets/hg38/Clinical_SNVs_hg38
- lines changed 60, context: html, text, full: html, text
dfcf312ddc1c89afe3dd518d8383ed6c2dc913b1 Tue Sep 8 14:08:19 2026 -0700
Updating the hg38 Clinical SNVs recommended track set to add the NMD Escape MANE track in pack mode. Shifts the imgOrd of the following tracks by one to place NMD escape near the top. Also picks up hide settings for tracks added since the last update and renamed track IDs (cCREs, wgEncodeReg4, GencodeV50). No RM.
- src/hg/htdocs/goldenPath/help/api.html
- lines changed 1, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 2, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/bam.html
- lines changed 4, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/bedgraph.html
- lines changed 4, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/bigNet.html
- lines changed 176, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/htdocs/goldenPath/help/customTrackText.html
- lines changed 5, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/docker.html
- lines changed 1, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/ftp.html
- lines changed 3, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/hgCodonColoring.html
- lines changed 18, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- src/hg/htdocs/goldenPath/help/hgTrackHubHelp.html
- lines changed 1, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/hgVcfTrackHelp.html
- lines changed 10, context: html, text, full: html, text
eebe031af9dfcecc8f5d0c1a5cd6a13b4d3e8865 Wed Sep 9 14:43:26 2026 -0700
VCF help: explain which display modes draw the haplotype view, and add the settings that were missing, refs #38010
Neither VCF help page said that the haplotype sorting display depends on the
track's display mode. vcfTrack.c reaches vcfHapClusterOverloadMethods only when
the visibility is pack or squish and the file has genotypes for more than one
sample; every other case falls through to vcfFileToPgSnp. So full draws one row
per variant, dense collapses them onto a single row, and in both of those the
"Enable Haplotype sorting display" checkbox and everything conditional on it do
nothing. Multi-region view and the density graph option disable it as well. Say
so on hgVcfTrackHelp.html, above the settings it governs, and on vcf.html next
to the visibility parameter.
vcf.html also listed only hapCluster{Enabled,ColorBy,TreeAngle,Height},
applyMinQual, minQual and minFreq. Add hapClusterMethod, sampleColorFile, minAc
and the four vcfDo* switches that hide filter controls, in a block of their own
since they are mostly used by hubs. sampleMetadataFile and showHardyWeinberg
are defined in vcfUi.h but nothing in the tree reads them, so they are left out.
Checked all four modes on the HGDP phased variants track, chr21:33,000,000-
33,010,000: dense 566 px, squish 617, pack 681, full 9574.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 6, context: html, text, full: html, text
6d02024d6f5b80437784b274ff2ccf6940dde976 Wed Sep 9 15:08:26 2026 -0700
VCF help: document geneTrack, the function coloring scheme, and vcfPhasedColorBy, refs #38010
The rest of the settings vcfUi.c reads from trackDb but neither help page
mentioned.
geneTrack (vcfUi.c:269 and :681) is the gate for the functional-effect coloring
in both the haplotype display and the trio display: the radio button is only
printed when the setting is non-empty. Nothing on either page said so, so the
scheme was undiscoverable and its absence looked like a bug.
hapClusterColorBy therefore has four values, not the three both pages listed --
hgVcfTrackHelp.html went as far as saying "There are three ways that reference
and alternate alleles can be colored" above three bullets. Add the fourth, in
the order vcfCfgHapClusterColor prints the buttons, and add function to the
value lists in vcf.html.
vcfPhasedColorBy (mendelDiff|deNovo|function|noColor) was documented nowhere at
all, not even in trackDbLibrary.shtml, though vcf.html already described what it
does in the alt text of the trio screenshot. Add it to the trio settings.
Both settings tables needed a wider value column to fit, so those rows are
repadded; no wording in them changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/hic.html
- lines changed 5, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/net.html
- lines changed 3, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/posters.html
- lines changed 1, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/query.html
- lines changed 23, context: html, text, full: html, text
fa5b31d305066e1938953374d80b3782ef87e239 Mon Sep 7 23:23:40 2026 -0700
Position box: accept a bare codon number, and a range of codon numbers
"KAT6A p.495_533" used to land on codon 495 and silently drop the end of the
range, and a bare codon number after a transcript accession was not understood
at all, so "ENST00000265713.8 p.495" fell through the HGVS code and ended up on
an unrelated locus. Nucleotide ranges already worked. The pseudo-HGVS layer now
takes an optional _end on a bare codon number, and accepts a bare codon number
or range after an NM_ or ENST accession as well as after a gene symbol, looking
up the reference amino acids that HGVS wants and the user did not type.
The accession forms require a literal "p", so "NM_006766.5 1483" keeps meaning
what it meant. A hyphen is still not a range separator: c.1483-1599 is the HGVS
intronic position and stays that way.
Also fixes a read past the end of the protein sequence when the codon number
was larger than the protein, and documents codon ranges in query.html.
refs #38285
- lines changed 20, context: html, text, full: html, text
0f23d17640ca30e2c9ee456c7e15966cabd3bc57 Mon Sep 7 23:32:24 2026 -0700
Position box: let a hyphen separate a range of codons, e.g. "BRCA1 100-200"
A bare number after a gene symbol has always meant a codon, and "KAT6A 495-533"
was already accepted -- it just landed on codon 495 and dropped the rest, the
same silent truncation that the underscore form had. A hyphen now separates a
range wherever the coordinates are protein: after a gene symbol with no prefix,
and after an explicit p. with a symbol or a transcript accession.
The hyphen stays out of c. and n. terms, where HGVS already uses it for an
intron offset. KAT6A c.1483-1599 is still the single base 1599 nt before
c.1483, not codons 1483 to 1599, and there are now regression tests pinning
both readings so the two do not drift into each other.
refs #38285
- src/hg/htdocs/goldenPath/help/quickLift.html
- lines changed 39, context: html, text, full: html, text
874a1eb980511c7e2273d8a6e4d29f00b81eae64 Fri Sep 11 20:22:55 2026 -0700
quickLift help: list the formats that lift instead of the ones that do not, refs #35536
The "Unsupported Track Formats" section was a denylist. It went stale as
soon as a format was added, and it was already wrong: it listed PSL and
bigChain as unsupported when both lift.
Replaced it with a "Supported Track Formats" allowlist, which matches how
the code decides. A short paragraph after the list names the common
formats that do not lift and repeats the message the Convert page shows.
The WIG to bigWig conversion note moved out of the list into its own
paragraph.
Kept an unsupportedTypes anchor on the section so old links still land.
- src/hg/htdocs/goldenPath/help/trackDb/changes.html
- lines changed 12, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 13, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- lines changed 1, context: html, text, full: html, text
160d10a30597d7435c30409878e2ba11c3e7c660 Wed Sep 9 14:19:40 2026 -0700
trackDb docs: stop the shared script from aborting on the pages without a search box, refs #38062
Two JavaScript faults on the trackDb doc pages, both of which trackDbHub.v3.html
had already avoided while the other pages drifted.
trackDbDoc.html, changes.html and trackDbHub.v0/v1/v2.html load /js/utils.js in
their head, and the hgMenubar include further down brings in the same file by a
different path, so it ran twice. utils.js declares mouseoverContainer with let
at top level, so the second run threw a redeclaration error. v3 does not load
utils.js in the head, and the menubar copy is in place well before anything
needs it, so drop the head copy from the other five.
documentLoad() then called addEventListener on the result of
getElementById("tdbSearch") with no null check, and only v3 has that input, so
the function threw partway through on the other four pages and never reached
its end. The search box and jump-to-top button do not exist on those pages, so
nothing visible was lost, but the exception aborted the rest of documentLoad
and would have swallowed anything added after it. Guard the two listeners.
All six pages now load with no console errors, checked in a browser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 11, context: html, text, full: html, text
9d9210bb7b34131505ab50b5e62bb88680dc6129 Wed Sep 9 15:14:09 2026 -0700
trackDb docs: add vcfPhasedColorBy, refs #38010
vcfPhasedColorBy has been read by vcfUi.c since the trio display went in, but it
was documented nowhere: not on the VCF help pages, not in trackDbLibrary, and
tdbQuery -check would have rejected it because tagTypes.tab did not list it
either. So a hub author had no way to find the setting and no way to use it
without tripping the checker.
Add the library blurb, the rows in trackDbDoc.html and trackDbHub.v3.html, a
changes.html entry, and the tagTypes.tab registration. The blurb spells out that
mendelDiff needs vcfParentSamples and that function is only offered when
geneTrack is set, since both conditions are enforced in vcfUi.c and neither is
obvious from the value name.
Companion to the two commits documenting the same settings on vcf.html and
hgVcfTrackHelp.html.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 11, context: html, text, full: html, text
37adfded75ec133c1fd2b4cb86e4757e76a54de5 Wed Sep 9 15:29:55 2026 -0700
trackDb docs: describe minAc and vcfDoMinAc, and fix two mislabelled vcfDoMaf rows, refs #38010
The last two VCF settings with no description anywhere. Both were absent from
trackDbLibrary, trackDbDoc and trackDbHub.v3, so registering them in
tagTypes.tab last commit made them legal but still undiscoverable.
minAc is a real filter, not just a UI default: minAcFail() in vcfTrack.c takes
the largest alternate allele count from the AC field of the INFO column and
drops the record when it is below the setting. Records whose INFO has no usable
AC are never dropped, which is worth saying since it is the surprising half.
vcfDoMinAc gates the matching control, like its three siblings.
Rows follow the placement the sibling settings already use: minAc beside
minFreq in the vcfTabix table, vcfDoMinAc beside vcfDoMaf in both the vcfTabix
and vcfPhasedTrio tables.
While adding those, trackDbDoc.html turned out to carry two rows with
class="vcfDoMaf" whose anchor and format line both read vcfDoQual, so the page
listed vcfDoQual twice and never showed vcfDoMaf's syntax. Copy-paste, present
in both the vcfTabix and vcfPhasedTrio tables. Corrected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/trackDb/trackDbDoc.html
- lines changed 1, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 1, context: html, text, full: html, text
160d10a30597d7435c30409878e2ba11c3e7c660 Wed Sep 9 14:19:40 2026 -0700
trackDb docs: stop the shared script from aborting on the pages without a search box, refs #38062
Two JavaScript faults on the trackDb doc pages, both of which trackDbHub.v3.html
had already avoided while the other pages drifted.
trackDbDoc.html, changes.html and trackDbHub.v0/v1/v2.html load /js/utils.js in
their head, and the hgMenubar include further down brings in the same file by a
different path, so it ran twice. utils.js declares mouseoverContainer with let
at top level, so the second run threw a redeclaration error. v3 does not load
utils.js in the head, and the menubar copy is in place well before anything
needs it, so drop the head copy from the other five.
documentLoad() then called addEventListener on the result of
getElementById("tdbSearch") with no null check, and only v3 has that input, so
the function threw partway through on the other four pages and never reached
its end. The search box and jump-to-top button do not exist on those pages, so
nothing visible was lost, but the exception aborted the rest of documentLoad
and would have swallowed anything added after it. Guard the two listeners.
All six pages now load with no console errors, checked in a browser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 3, context: html, text, full: html, text
9d9210bb7b34131505ab50b5e62bb88680dc6129 Wed Sep 9 15:14:09 2026 -0700
trackDb docs: add vcfPhasedColorBy, refs #38010
vcfPhasedColorBy has been read by vcfUi.c since the trio display went in, but it
was documented nowhere: not on the VCF help pages, not in trackDbLibrary, and
tdbQuery -check would have rejected it because tagTypes.tab did not list it
either. So a hub author had no way to find the setting and no way to use it
without tripping the checker.
Add the library blurb, the rows in trackDbDoc.html and trackDbHub.v3.html, a
changes.html entry, and the tagTypes.tab registration. The blurb spells out that
mendelDiff needs vcfParentSamples and that function is only offered when
geneTrack is set, since both conditions are enforced in vcfUi.c and neither is
obvious from the value name.
Companion to the two commits documenting the same settings on vcf.html and
hgVcfTrackHelp.html.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 13, context: html, text, full: html, text
37adfded75ec133c1fd2b4cb86e4757e76a54de5 Wed Sep 9 15:29:55 2026 -0700
trackDb docs: describe minAc and vcfDoMinAc, and fix two mislabelled vcfDoMaf rows, refs #38010
The last two VCF settings with no description anywhere. Both were absent from
trackDbLibrary, trackDbDoc and trackDbHub.v3, so registering them in
tagTypes.tab last commit made them legal but still undiscoverable.
minAc is a real filter, not just a UI default: minAcFail() in vcfTrack.c takes
the largest alternate allele count from the AC field of the INFO column and
drops the record when it is below the setting. Records whose INFO has no usable
AC are never dropped, which is worth saying since it is the surprising half.
vcfDoMinAc gates the matching control, like its three siblings.
Rows follow the placement the sibling settings already use: minAc beside
minFreq in the vcfTabix table, vcfDoMinAc beside vcfDoMaf in both the vcfTabix
and vcfPhasedTrio tables.
While adding those, trackDbDoc.html turned out to carry two rows with
class="vcfDoMaf" whose anchor and format line both read vcfDoQual, so the page
listed vcfDoQual twice and never showed vcfDoMaf's syntax. Copy-paste, present
in both the vcfTabix and vcfPhasedTrio tables. Corrected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/trackDb/trackDbDoc.js
- lines changed 19, context: html, text, full: html, text
bc030946a04b1f5c4c0a1501e0eb385747238b1b Wed Sep 9 11:43:51 2026 -0700
trackDb docs: send a library cross-link to the current spec when the host page has no such anchor, refs #38062
trackDbLibrary.shtml is shared by trackDbDoc.html and all four trackDbHub
version pages. Its setting blurbs cross-reference other settings with a bare
fragment, but the anchor is written by each host page's own table of contents,
so a blurb that mentions a setting newer than a frozen hub spec has no target
on that page. trackDbHub.v0, v1 and v2 therefore carry dangling links to
filterLabel, detailsDynamicTable, filterBy, faceted_composite, onlyVisibility
and detailsStaticTable, and the library is regenerated while those snapshots
stay put, so the set grows with every new setting.
Rather than edit the frozen snapshots or hard-code a page into the library,
check at the moment a blurb is moved out of the hidden library into the
document and repoint the link to trackDbHub.html, which always serves the
current spec, when the anchor is not on the page. Links whose target is
present are untouched, so this changes nothing on trackDbDoc.html or on the
current spec, and it keeps working as settings are added.
Only the visibility blurb's link to faceted_composite is reachable by a reader
today; the rest sit in blurbs those pages do not display. Verified in a browser
on all five pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 5, context: html, text, full: html, text
160d10a30597d7435c30409878e2ba11c3e7c660 Wed Sep 9 14:19:40 2026 -0700
trackDb docs: stop the shared script from aborting on the pages without a search box, refs #38062
Two JavaScript faults on the trackDb doc pages, both of which trackDbHub.v3.html
had already avoided while the other pages drifted.
trackDbDoc.html, changes.html and trackDbHub.v0/v1/v2.html load /js/utils.js in
their head, and the hgMenubar include further down brings in the same file by a
different path, so it ran twice. utils.js declares mouseoverContainer with let
at top level, so the second run threw a redeclaration error. v3 does not load
utils.js in the head, and the menubar copy is in place well before anything
needs it, so drop the head copy from the other five.
documentLoad() then called addEventListener on the result of
getElementById("tdbSearch") with no null check, and only v3 has that input, so
the function threw partway through on the other four pages and never reached
its end. The search box and jump-to-top button do not exist on those pages, so
nothing visible was lost, but the exception aborted the rest of documentLoad
and would have swallowed anything added after it. Guard the two listeners.
All six pages now load with no console errors, checked in a browser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/trackDb/trackDbHub.v0.html
- lines changed 1, context: html, text, full: html, text
160d10a30597d7435c30409878e2ba11c3e7c660 Wed Sep 9 14:19:40 2026 -0700
trackDb docs: stop the shared script from aborting on the pages without a search box, refs #38062
Two JavaScript faults on the trackDb doc pages, both of which trackDbHub.v3.html
had already avoided while the other pages drifted.
trackDbDoc.html, changes.html and trackDbHub.v0/v1/v2.html load /js/utils.js in
their head, and the hgMenubar include further down brings in the same file by a
different path, so it ran twice. utils.js declares mouseoverContainer with let
at top level, so the second run threw a redeclaration error. v3 does not load
utils.js in the head, and the menubar copy is in place well before anything
needs it, so drop the head copy from the other five.
documentLoad() then called addEventListener on the result of
getElementById("tdbSearch") with no null check, and only v3 has that input, so
the function threw partway through on the other four pages and never reached
its end. The search box and jump-to-top button do not exist on those pages, so
nothing visible was lost, but the exception aborted the rest of documentLoad
and would have swallowed anything added after it. Guard the two listeners.
All six pages now load with no console errors, checked in a browser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/trackDb/trackDbHub.v1.html
- lines changed 1, context: html, text, full: html, text
160d10a30597d7435c30409878e2ba11c3e7c660 Wed Sep 9 14:19:40 2026 -0700
trackDb docs: stop the shared script from aborting on the pages without a search box, refs #38062
Two JavaScript faults on the trackDb doc pages, both of which trackDbHub.v3.html
had already avoided while the other pages drifted.
trackDbDoc.html, changes.html and trackDbHub.v0/v1/v2.html load /js/utils.js in
their head, and the hgMenubar include further down brings in the same file by a
different path, so it ran twice. utils.js declares mouseoverContainer with let
at top level, so the second run threw a redeclaration error. v3 does not load
utils.js in the head, and the menubar copy is in place well before anything
needs it, so drop the head copy from the other five.
documentLoad() then called addEventListener on the result of
getElementById("tdbSearch") with no null check, and only v3 has that input, so
the function threw partway through on the other four pages and never reached
its end. The search box and jump-to-top button do not exist on those pages, so
nothing visible was lost, but the exception aborted the rest of documentLoad
and would have swallowed anything added after it. Guard the two listeners.
All six pages now load with no console errors, checked in a browser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/trackDb/trackDbHub.v2.html
- lines changed 1, context: html, text, full: html, text
160d10a30597d7435c30409878e2ba11c3e7c660 Wed Sep 9 14:19:40 2026 -0700
trackDb docs: stop the shared script from aborting on the pages without a search box, refs #38062
Two JavaScript faults on the trackDb doc pages, both of which trackDbHub.v3.html
had already avoided while the other pages drifted.
trackDbDoc.html, changes.html and trackDbHub.v0/v1/v2.html load /js/utils.js in
their head, and the hgMenubar include further down brings in the same file by a
different path, so it ran twice. utils.js declares mouseoverContainer with let
at top level, so the second run threw a redeclaration error. v3 does not load
utils.js in the head, and the menubar copy is in place well before anything
needs it, so drop the head copy from the other five.
documentLoad() then called addEventListener on the result of
getElementById("tdbSearch") with no null check, and only v3 has that input, so
the function threw partway through on the other four pages and never reached
its end. The search box and jump-to-top button do not exist on those pages, so
nothing visible was lost, but the exception aborted the rest of documentLoad
and would have swallowed anything added after it. Guard the two listeners.
All six pages now load with no console errors, checked in a browser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/trackDb/trackDbHub.v3.html
- lines changed 20, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 1, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 3, context: html, text, full: html, text
9d9210bb7b34131505ab50b5e62bb88680dc6129 Wed Sep 9 15:14:09 2026 -0700
trackDb docs: add vcfPhasedColorBy, refs #38010
vcfPhasedColorBy has been read by vcfUi.c since the trio display went in, but it
was documented nowhere: not on the VCF help pages, not in trackDbLibrary, and
tdbQuery -check would have rejected it because tagTypes.tab did not list it
either. So a hub author had no way to find the setting and no way to use it
without tripping the checker.
Add the library blurb, the rows in trackDbDoc.html and trackDbHub.v3.html, a
changes.html entry, and the tagTypes.tab registration. The blurb spells out that
mendelDiff needs vcfParentSamples and that function is only offered when
geneTrack is set, since both conditions are enforced in vcfUi.c and neither is
obvious from the value name.
Companion to the two commits documenting the same settings on vcf.html and
hgVcfTrackHelp.html.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 9, context: html, text, full: html, text
37adfded75ec133c1fd2b4cb86e4757e76a54de5 Wed Sep 9 15:29:55 2026 -0700
trackDb docs: describe minAc and vcfDoMinAc, and fix two mislabelled vcfDoMaf rows, refs #38010
The last two VCF settings with no description anywhere. Both were absent from
trackDbLibrary, trackDbDoc and trackDbHub.v3, so registering them in
tagTypes.tab last commit made them legal but still undiscoverable.
minAc is a real filter, not just a UI default: minAcFail() in vcfTrack.c takes
the largest alternate allele count from the AC field of the INFO column and
drops the record when it is below the setting. Records whose INFO has no usable
AC are never dropped, which is worth saying since it is the surprising half.
vcfDoMinAc gates the matching control, like its three siblings.
Rows follow the placement the sibling settings already use: minAc beside
minFreq in the vcfTabix table, vcfDoMinAc beside vcfDoMaf in both the vcfTabix
and vcfPhasedTrio tables.
While adding those, trackDbDoc.html turned out to carry two rows with
class="vcfDoMaf" whose anchor and format line both read vcfDoQual, so the page
listed vcfDoQual twice and never showed vcfDoMaf's syntax. Copy-paste, present
in both the vcfTabix and vcfPhasedTrio tables. Corrected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/trackDb/trackDbLibrary.shtml
- lines changed 12, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 83, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- lines changed 14, context: html, text, full: html, text
9d9210bb7b34131505ab50b5e62bb88680dc6129 Wed Sep 9 15:14:09 2026 -0700
trackDb docs: add vcfPhasedColorBy, refs #38010
vcfPhasedColorBy has been read by vcfUi.c since the trio display went in, but it
was documented nowhere: not on the VCF help pages, not in trackDbLibrary, and
tdbQuery -check would have rejected it because tagTypes.tab did not list it
either. So a hub author had no way to find the setting and no way to use it
without tripping the checker.
Add the library blurb, the rows in trackDbDoc.html and trackDbHub.v3.html, a
changes.html entry, and the tagTypes.tab registration. The blurb spells out that
mendelDiff needs vcfParentSamples and that function is only offered when
geneTrack is set, since both conditions are enforced in vcfUi.c and neither is
obvious from the value name.
Companion to the two commits documenting the same settings on vcf.html and
hgVcfTrackHelp.html.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 15, context: html, text, full: html, text
37adfded75ec133c1fd2b4cb86e4757e76a54de5 Wed Sep 9 15:29:55 2026 -0700
trackDb docs: describe minAc and vcfDoMinAc, and fix two mislabelled vcfDoMaf rows, refs #38010
The last two VCF settings with no description anywhere. Both were absent from
trackDbLibrary, trackDbDoc and trackDbHub.v3, so registering them in
tagTypes.tab last commit made them legal but still undiscoverable.
minAc is a real filter, not just a UI default: minAcFail() in vcfTrack.c takes
the largest alternate allele count from the AC field of the INFO column and
drops the record when it is below the setting. Records whose INFO has no usable
AC are never dropped, which is worth saying since it is the surprising half.
vcfDoMinAc gates the matching control, like its three siblings.
Rows follow the placement the sibling settings already use: minAc beside
minFreq in the vcfTabix table, vcfDoMinAc beside vcfDoMaf in both the vcfTabix
and vcfPhasedTrio tables.
While adding those, trackDbDoc.html turned out to carry two rows with
class="vcfDoMaf" whose anchor and format line both read vcfDoQual, so the page
listed vcfDoQual twice and never showed vcfDoMaf's syntax. Copy-paste, present
in both the vcfTabix and vcfPhasedTrio tables. Corrected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 8, context: html, text, full: html, text
964cacf58dec20af72e4c5ed64a33eaf8596c96f Fri Sep 11 15:22:00 2026 -0700
trackDb docs: add the missing bigNet_intro blurb, refs #20824
The bigNet section in trackDbHub.v3.html declares a DIV with ID
bigNet_intro, but the library had no blurb with that class. The page
logged "Missing document blurb for ID: bigNet_intro" and the bigNet
section rendered with no introduction.
Added the blurb next to bigChain_intro, worded like the other format
intros and pointing at the bigNet help page.
Every other *_intro ID in trackDbHub.v3.html and trackDbDoc.html already
has a blurb; bigNet was the only one missing.
- src/hg/htdocs/goldenPath/help/trackDb/trackDbSettings.json
- lines changed 16, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 2, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- src/hg/htdocs/goldenPath/help/trackDb/trackDbSettings.yaml
- lines changed 22, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 1349, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- src/hg/htdocs/goldenPath/help/trackDb/trackDbSettingsGen.py
- lines changed 23, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- src/hg/htdocs/goldenPath/help/trackDbIndexBb.html
- lines changed 3, context: html, text, full: html, text
e5759993329b6e609e9ab825991a90283e714088 Wed Sep 9 11:17:57 2026 -0700
Help pages: fix broken internal links and restore anchors people still cite, refs #38062
Three groups of anchor problems on the help and FAQ pages.
Broken internal links, five pages: posters.html listed a 2022 section that
does not exist (no 2022 posters), api.html listed REST and JSON separately
after the two sections were merged, docker.html pointed at a #UsrAcct section
that is not on that page, FAQgenes.html had a capitalized #ncbiRefSeq where
the anchor is #ncbiRefseq, and the genomes.txt settings rows in
trackDbHub.v3.html carried no anchors so its own "genome" link missed.
Retired anchors that are still cited in twenty years of answers on the genome
list. Content moved to its own page and the old anchor was deleted rather than
left behind, so the citations land at the top of the page. Reattached seven
numeric FAQformat anchors to the Topics entry linking to each format's page.
BED, PSL, GFF and GTF were removed from the custom track page in 2012 and
never added back to its list of supported formats; added them with the old
anchors, which fixes customTrack.html and hgTracksHelp.html together since
both include customTrackText.html. Also restored #lines there, and #Session
on hgTrackHubHelp.html and #link4 on FAQlink.html.
Section anchors on six pages that had none, so a support answer can link to
one part of them: bam.html, hic.html, bedgraph.html, ftp.html, net.html and
trackDbIndexBb.html. Skipped quickLiftChain.html, oligoMatch.html and
cutters.html, which are track description fragments included into the details
page rather than standalone pages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/help/vcf.html
- lines changed 19, context: html, text, full: html, text
eebe031af9dfcecc8f5d0c1a5cd6a13b4d3e8865 Wed Sep 9 14:43:26 2026 -0700
VCF help: explain which display modes draw the haplotype view, and add the settings that were missing, refs #38010
Neither VCF help page said that the haplotype sorting display depends on the
track's display mode. vcfTrack.c reaches vcfHapClusterOverloadMethods only when
the visibility is pack or squish and the file has genotypes for more than one
sample; every other case falls through to vcfFileToPgSnp. So full draws one row
per variant, dense collapses them onto a single row, and in both of those the
"Enable Haplotype sorting display" checkbox and everything conditional on it do
nothing. Multi-region view and the density graph option disable it as well. Say
so on hgVcfTrackHelp.html, above the settings it governs, and on vcf.html next
to the visibility parameter.
vcf.html also listed only hapCluster{Enabled,ColorBy,TreeAngle,Height},
applyMinQual, minQual and minFreq. Add hapClusterMethod, sampleColorFile, minAc
and the four vcfDo* switches that hide filter controls, in a block of their own
since they are mostly used by hubs. sampleMetadataFile and showHardyWeinberg
are defined in vcfUi.h but nothing in the tree reads them, so they are left out.
Checked all four modes on the HGDP phased variants track, chr21:33,000,000-
33,010,000: dense 566 px, squish 617, pack 681, full 9574.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 5, context: html, text, full: html, text
6d02024d6f5b80437784b274ff2ccf6940dde976 Wed Sep 9 15:08:26 2026 -0700
VCF help: document geneTrack, the function coloring scheme, and vcfPhasedColorBy, refs #38010
The rest of the settings vcfUi.c reads from trackDb but neither help page
mentioned.
geneTrack (vcfUi.c:269 and :681) is the gate for the functional-effect coloring
in both the haplotype display and the trio display: the radio button is only
printed when the setting is non-empty. Nothing on either page said so, so the
scheme was undiscoverable and its absence looked like a bug.
hapClusterColorBy therefore has four values, not the three both pages listed --
hgVcfTrackHelp.html went as far as saying "There are three ways that reference
and alternate alleles can be colored" above three bullets. Add the fourth, in
the order vcfCfgHapClusterColor prints the buttons, and add function to the
value lists in vcf.html.
vcfPhasedColorBy (mendelDiff|deNovo|function|noColor) was documented nowhere at
all, not even in trackDbLibrary.shtml, though vcf.html already described what it
does in the alt text of the trio screenshot. Add it to the trio settings.
Both settings tables needed a wider value column to fit, so those rows are
repadded; no wording in them changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/htdocs/goldenPath/newsarch.html
- lines changed 54, context: html, text, full: html, text
808c426eb279a17a5539731811f1deda6e783e1e Wed Sep 9 17:20:31 2026 -0700
Announcing the AlphaGenome Variant Impact (AVI) score track for hg38. refs #38261
Adds the Sept. 10 news archive entry with a figure of the AVI scores across the TERT
locus, and adds the item to the front page news list, dropping the Jul. 22 entry to
keep the list at six.
- src/hg/htdocs/images/hgTracksHomeIconSprite.png
- lines changed 0, context: html, text, full: html, text
57490a621bd10c9c7810bdf474b7c801027d237d Thu Sep 10 06:56:54 2026 -0700
Use one blue for the menu bar on every page, refs #38206
The bar came in three shades: #2636d1 from nice_menu.css on hgTables,
hgBlat, hgc and the other cart CGIs, #00457c on hgTracks through its own
set of ids, and #003a72 on the gateway and the static pages, which
override nice_menu.css afterwards. They are all #003a72 now, the house
deep navy already used for the gateway banner, the footer and the
buttons.
hgTracks no longer rewrites main-menu-whole and home-link into its own
ids, so its bar and its house icon come from the same rules as everyone
else's, and hgTracksHomeIconSprite.png goes away - the shared sprite is
white on a transparent background and sits on whatever blue the bar has.
gbStatic.css keeps its own copy of the color on purpose; the comment
there says why.
- src/hg/htdocs/images/newsArchImages/alphaGenome.png
- lines changed 0, context: html, text, full: html, text
808c426eb279a17a5539731811f1deda6e783e1e Wed Sep 9 17:20:31 2026 -0700
Announcing the AlphaGenome Variant Impact (AVI) score track for hg38. refs #38261
Adds the Sept. 10 news archive entry with a figure of the AVI scores across the TERT
locus, and adds the item to the front page news list, dropping the Jul. 22 entry to
keep the list at six.
- src/hg/htdocs/indexNews.html
- lines changed 12, context: html, text, full: html, text
808c426eb279a17a5539731811f1deda6e783e1e Wed Sep 9 17:20:31 2026 -0700
Announcing the AlphaGenome Variant Impact (AVI) score track for hg38. refs #38261
Adds the Sept. 10 news archive entry with a figure of the AVI scores across the TERT
locus, and adds the item to the front page news list, dropping the Jul. 22 entry to
keep the list at six.
- src/hg/htdocs/staticStyle/gbStatic.css
- lines changed 1, context: html, text, full: html, text
2941692d3e4198ba220b16c9b791c9090aa69fbc Wed Sep 9 17:20:31 2026 -0700
Reduce the font size of image captions on static pages so they read as distinct from
body text. No RM.
p.gbsCaption had no font-size, so captions rendered at the same size as body copy.
Note this is the copy of gbStatic.css under staticStyle/, which is the one the
gbPageStart includes load; the copy under style/ is not referenced by any page.
- lines changed 4, context: html, text, full: html, text
57490a621bd10c9c7810bdf474b7c801027d237d Thu Sep 10 06:56:54 2026 -0700
Use one blue for the menu bar on every page, refs #38206
The bar came in three shades: #2636d1 from nice_menu.css on hgTables,
hgBlat, hgc and the other cart CGIs, #00457c on hgTracks through its own
set of ids, and #003a72 on the gateway and the static pages, which
override nice_menu.css afterwards. They are all #003a72 now, the house
deep navy already used for the gateway banner, the footer and the
buttons.
hgTracks no longer rewrites main-menu-whole and home-link into its own
ids, so its bar and its house icon come from the same rules as everyone
else's, and hgTracksHomeIconSprite.png goes away - the shared sprite is
white on a transparent background and sits on whatever blue the bar has.
gbStatic.css keeps its own copy of the color on purpose; the comment
there says why.
- src/hg/htdocs/style/dataTables.rowReorder-1.5.1.min.css
- lines changed 1, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- src/hg/htdocs/style/facetedComposite.css
- lines changed 88, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- src/hg/htdocs/style/gbAfterMenu.css
- lines changed 6, context: html, text, full: html, text
57490a621bd10c9c7810bdf474b7c801027d237d Thu Sep 10 06:56:54 2026 -0700
Use one blue for the menu bar on every page, refs #38206
The bar came in three shades: #2636d1 from nice_menu.css on hgTables,
hgBlat, hgc and the other cart CGIs, #00457c on hgTracks through its own
set of ids, and #003a72 on the gateway and the static pages, which
override nice_menu.css afterwards. They are all #003a72 now, the house
deep navy already used for the gateway banner, the footer and the
buttons.
hgTracks no longer rewrites main-menu-whole and home-link into its own
ids, so its bar and its house icon come from the same rules as everyone
else's, and hgTracksHomeIconSprite.png goes away - the shared sprite is
white on a transparent background and sits on whatever blue the bar has.
gbStatic.css keeps its own copy of the color on purpose; the comment
there says why.
- src/hg/htdocs/style/gbStatic.css
- lines changed 4, context: html, text, full: html, text
57490a621bd10c9c7810bdf474b7c801027d237d Thu Sep 10 06:56:54 2026 -0700
Use one blue for the menu bar on every page, refs #38206
The bar came in three shades: #2636d1 from nice_menu.css on hgTables,
hgBlat, hgc and the other cart CGIs, #00457c on hgTracks through its own
set of ids, and #003a72 on the gateway and the static pages, which
override nice_menu.css afterwards. They are all #003a72 now, the house
deep navy already used for the gateway banner, the footer and the
buttons.
hgTracks no longer rewrites main-menu-whole and home-link into its own
ids, so its bar and its house icon come from the same rules as everyone
else's, and hgTracksHomeIconSprite.png goes away - the shared sprite is
white on a transparent background and sits on whatever blue the bar has.
gbStatic.css keeps its own copy of the color on purpose; the comment
there says why.
- src/hg/htdocs/style/hgSession.css
- lines changed 6, context: html, text, full: html, text
fc8de100a3437b9fc33bdeb0f459ca2a93e2f318 Wed Sep 9 08:14:32 2026 -0700
hgSession: address the code review of the new Sessions page
Rename and unshare now keep the public listing's thumbnail with the session it
belongs to. The picture's file name is built from the encoded session name, so
renaming a listed session left the listing pointing at nothing and the old file
behind, and dropping a session from the listing to a plain shared link kept the
picture. The classic page had the same problem in a subtler form: it removed the
thumbnail after the row had already been renamed, so the old file survived.
Saving under a name that is already in use asks before it replaces that session,
using the failIfExists reply that the top-right Share a link menu already relies
on. The description and "only I can load it" steps that follow a save now report
a failure instead of reloading in silence, and what thumbnailAdd has to say when
it cannot build a picture reaches the user instead of being freed unread.
A session description no longer travels through a title attribute. The tooltip
machinery in utils.js inserts its text with innerHTML and an attribute is decoded
on the way, so a description containing angle brackets was interpreted as markup
rather than shown as typed. It is attached, escaped, after each table draw, which
also gives the rows DataTables renders later the same styled mouseovers as the
rest of the page.
Also: the AJAX endpoints say so when there is no session by that name, instead of
reporting a no-op as a success; the new page always offers its way back to the
classic page, since the cart variable that got the user there sticks; and four
unused CSS rules, a dead element lookup and a dead local are gone. hgConfCatalog
cited the wrong ticket for the two sessionNewPage flags.
refs #38180, refs #38157
- src/hg/htdocs/style/jWest.afterNiceMenu.css
- lines changed 6, context: html, text, full: html, text
57490a621bd10c9c7810bdf474b7c801027d237d Thu Sep 10 06:56:54 2026 -0700
Use one blue for the menu bar on every page, refs #38206
The bar came in three shades: #2636d1 from nice_menu.css on hgTables,
hgBlat, hgc and the other cart CGIs, #00457c on hgTracks through its own
set of ids, and #003a72 on the gateway and the static pages, which
override nice_menu.css afterwards. They are all #003a72 now, the house
deep navy already used for the gateway banner, the footer and the
buttons.
hgTracks no longer rewrites main-menu-whole and home-link into its own
ids, so its bar and its house icon come from the same rules as everyone
else's, and hgTracksHomeIconSprite.png goes away - the shared sprite is
white on a transparent background and sits on whatever blue the bar has.
gbStatic.css keeps its own copy of the color on purpose; the comment
there says why.
- src/hg/htdocs/style/makefile
- lines changed 2, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- src/hg/htdocs/style/nice_menu.css
- lines changed 35, context: html, text, full: html, text
57490a621bd10c9c7810bdf474b7c801027d237d Thu Sep 10 06:56:54 2026 -0700
Use one blue for the menu bar on every page, refs #38206
The bar came in three shades: #2636d1 from nice_menu.css on hgTables,
hgBlat, hgc and the other cart CGIs, #00457c on hgTracks through its own
set of ids, and #003a72 on the gateway and the static pages, which
override nice_menu.css afterwards. They are all #003a72 now, the house
deep navy already used for the gateway banner, the footer and the
buttons.
hgTracks no longer rewrites main-menu-whole and home-link into its own
ids, so its bar and its house icon come from the same rules as everyone
else's, and hgTracksHomeIconSprite.png goes away - the shared sprite is
white on a transparent background and sits on whatever blue the bar has.
gbStatic.css keeps its own copy of the color on purpose; the comment
there says why.
- src/hg/hubApi/apiUtils.c
- lines changed 1, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 3, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hubApi/blat.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hubApi/findGenome.c
- lines changed 44, context: html, text, full: html, text
441ec7ab39569a301912b2b08981747d98fdb752 Fri Sep 11 15:11:40 2026 -0700
the exactNameSearch() was the wrong way to go, back that out and do this correctly from claude code review refs #38290
- src/hg/hubApi/hubApi.c
- lines changed 5, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 3, context: html, text, full: html, text
d11a21c48cdc2cebe0a97779b510df8317b980cd Mon Sep 7 06:29:43 2026 -0700
bigNet: gate the track type behind an hg.conf flag, refs #20824
Add the boolean hg.conf setting bigNet, default FALSE, so the type ships
dark and a machine turns it on with bigNet=on.
trackHubBigNetEnabled() in hg/lib/trackHub.c is the one read; the four
places that accept or advertise the type ask it. validateOneTrack drops
bigNet from the hub track type allowlist, validateOneTdb drops it from the
types quickLift will lift, hubCheck drops it from VALID_TRACK_TYPES and
from the message listing the valid types, and hubApi does not add it to
supportedTypes. netToBigNet, netTrack.c, chainNetDbLoad.c and the hgc
details code are unchanged, since they cannot be reached once a bigNet
track will not load.
With the flag off hgTracks does not abort the hub. It draws the track row
as a bigWarn bar reading "Unsupported type 'bigNet ...'" and the rest of
the hub loads normally.
Register the flag in hgConfCatalog.py with role="gate" so the sunset
report tracks it.
- lines changed 4, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/hubApi/tests/findGenome.sh
- lines changed 7, context: html, text, full: html, text
d522ecef8f0426e8323009094dee2fa04798ffc1 Fri Sep 11 15:12:36 2026 -0700
add a regression test that was uncovered with v504 preview 1 build refs #38290
- src/hg/inc/bigNet.h
- lines changed 91, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/inc/cart.h
- lines changed 12, context: html, text, full: html, text
59e2bbf2d3a99e5f18436a756e1e5c8b64308e33 Tue Sep 8 09:21:02 2026 -0700
Copy a track collection's hub file when the program that writes it asks for a copy, instead of on every session load.
cartCopyLocalHubs ran on every session load, in every CGI and in four more places in
hgSession. For each customComposite-<db> cart variable it copied the collection's hub
file to a fresh trash name and registered the copy in hgcentral.hubStatus to get a hub
id. Nothing removed the old row. 81% of the 3.1 million rows in hubStatus on the RR are
these dead registrations, and each load also took the central_hubStatus advisory lock
and wrote to a MyISAM table, which serializes every concurrent load of a shared session
that carries a collection.
The copy itself is needed. A saved session's hub file lives under sessionDataDir and
every load of that session names it, so hgCollection must not write it in place. It was
just being made for every load, and almost no load is followed by an edit. hgCollection
is the only program that writes one of these files, so it now asks for its own copy:
main() calls cartRequestLocalHubCopy() before it opens the cart, and cartNew() makes the
copy for it.
That has to be a property of the program rather than of the request, and the copy has to
be made at cart open, above hubConnectLoadHubs. Copying the file gives the hub a new id,
the hubs are loaded during cart open, and trackList carries that id in every track name,
so a copy made any later leaves printTrackDbListToHub looking for the collection under
an id trackList does not have and the hub file comes out with a header and no tracks. A
condition that instead tries to work out whether this particular request will write
cannot be correct either: "cmd" is not in hgTracks' excludeVars, so it persists in the
cart and hgCollection dispatches on the cart rather than on the CGI variable, a settings
file can carry it, and a command-line run has no SCRIPT_NAME to test.
Three other parts. copyLocalHubs skips a hub the cart already owns, so repeated edits
reuse one file and one hub id rather than renumbering the hub on every drag; the test
requires a plain file, because saveTrackFile leaves the trash name behind as a symbolic
link to the durable copy and writing through that link would rewrite the saved session's
own file. copyLocalHubs also screens the path with isServerUserFilePath before opening
it, the way getHubName does, and selects on customComposite-<db> with the dash so that
the names it acts on are the ones fileNameCartVarPrefixes screens. saveTrackFile copies
when the source is a local hub outside trash that is not under this session's own
directory, so loading one session and saving it under a new name gives the new session
its own file instead of a reference into the first one's directory. pathIsUnderDir is no
longer static in trashDir.c.
All of it is behind the hg.conf gate collectionHubCopyOnWrite, default off, which
reproduces the old behavior exactly. Retiring the gate is not uniform: the body of
cartCopyLocalHubsOnSessionLoad is the old behavior and that function and its five
callers go away with the gate, while the other three tests lose only the gate term.
Measured on hgcentraltest with one binary. With the gate off, five loads of a session
carrying a collection add five hubStatus rows and three later edits add none; with it
on, the five loads add none and the first edit adds one. Saving a loaded session costs
two rows off and one on. The resulting hub file is byte identical either way apart from
the hub id, and the file the saved session names is unchanged by md5 and mtime,
including after an edit made straight after a save, and including when the cart names a
trash symbolic link into session storage.
refs #38273
- lines changed 3, context: html, text, full: html, text
cf9f4cb7f55c7beb8ad5f11118656a60770a71a5 Thu Sep 10 01:02:36 2026 -0700
Move the extra-HTTP-header list into cheapcgi, and write the header only once
Follow-on to the cgiPrintContentType() refactor.
cart.c owned the mechanism for adding headers ahead of the content type: a
global slPair list plus addHttpHeaders() to print it. That put it in hg/lib,
out of reach of the CGIs and library code that do not use a cart, even though
nothing about it is cart-specific. It now lives next to cgiPrintContentType()
in lib/cheapcgi.c, behind cgiAddHttpHeader(name, value) instead of a bare
global, and cgiPrintContentType() writes the queued headers itself. The one
caller, hgTracks/mainMain.c, reads the same but no longer reaches into cart.h
for it. cspWriteResponseHeader() stays in hg/lib where it belongs, since it
needs hg.conf; cartWriteHeaderAndCont() calls it directly now, the way the
other ten callers already do.
cgiPrintContentType() also writes at most once per process now. A second
content type cannot reach the browser as a header - it lands in the page body
as text - so the later caller is always the mistaken one. cart.c had a private
cartDidContentType flag for exactly this, covering only the flows that went
through the cart; the guard is now in the one function every flow shares, and
cartDidContentType is gone. Its public equivalent, cgiDidContentType(), is
what cartWriteHeaderAndCont() checks so it does not write a second cookie.
Verified: make libs, make cgi and the lib test suite are clean, hgTracks still
emits Cache-Control: no-store, and hgTracks, hgc and hgTables each emit exactly
one Content-Type on both their html and their text paths.
- src/hg/inc/chainNetDbLoad.h
- lines changed 5, context: html, text, full: html, text
42cfb8cd4a22209d8183ffcad97e5ca5e9ae070b Fri Sep 4 12:47:47 2026 -0700
chainNetDbLoad: add chainLoadRange, every chain in a range with its blocks
chainLoadIdRange loads one chain by id, and the bigChain reader takes -1 to mean
every chain in the range, but there was no SQL equivalent of the latter. This is
it: one query for the headers, one for the links, and each link handed to the
chain it belongs to.
The blocks are sorted rather than just reversed. A bin indexed range query does
not return rows in position order, and anything that walks a chain's blocks
expects them ascending, which is why chainLinkAddResult sorts them too.
refs #38249
- lines changed 4, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 16, context: html, text, full: html, text
f29b65452a1cca8a5bc62f007100b313e589d036 Sun Sep 6 15:11:36 2026 -0700
quickLift: lift a bigNet track, refs #20824
A net is not an ordinary track to lift. It is an alignment of two assemblies,
so only its target side moves to the new reference; the query side names a third
assembly and is carried across untouched. And the browser draws a net by
recursion, so what really has to survive the lift is the tree: a row's level only
means anything relative to the row above it.
chainNetLoadRangeQuickLift() maps each row's target range with
quickLiftIntervalsToBedClip, which is the same code every other quickLift track
uses, so a net row lands where a bed of the same span would. The surviving rows
then go to the same helpToNet() the unlifted path uses, so there is one tree
builder and not two. The row collection either path does is now cnlHelperNew and
cnlHelperAddBigNet.
validateOneTdb lets bigNet into a quickLift hub. Its chain track is not offered,
so a lifted net's details page has no chain to follow and says so rather than
looking for a track this assembly does not have. bigNetLoadOne lifts the same
way for that page, unclipped, so it reports the item's whole extent.
bigNetFromInterval passes -1 as the cached chromId. bbiCachedChromLookup leaves
the buffer alone when the id matches the one before it, so a cache that outlives
the buffer hands back stale bytes.
Measured against the standalone liftOver tool, on hg19 chr22's mouse net lifted
to hg38: 36 of 36 source rows in the sampled window land on the same coordinates,
counting the one liftOver will not take whole, whose two ends it does place
exactly where the browser puts them. Against hg38's own mouse net, computed
independently, 1.02% of the drawn pixels differ.
- lines changed 4, context: html, text, full: html, text
b18f2f9facbf4fede2ba522909535f133726c400 Mon Sep 7 11:20:43 2026 -0700
bigNet: declare the bigBed types chainNetDbLoad.h uses, refs #20824
bigNetFromInterval takes a struct bbiFile and a struct bigBedInterval, and
the header named them without including bigBed.h. Any file that reached
chainNetDbLoad.h without bigBed.h ahead of it failed -Werror with "declared
inside parameter list", which is where hg/lib/liftOver.c ended up once the
generated header dependencies started rebuilding it.
- lines changed 29, context: html, text, full: html, text
e9ed25747e3d8ab10963306b826f7cedc5e71897 Mon Sep 7 11:35:30 2026 -0700
Merge branch 'quickLiftAlign38249' -- alignment tracks in quickLift, refs #38249
# Conflicts:
# src/hg/lib/trackHub.c
- src/hg/inc/genark.h
- lines changed 2, context: html, text, full: html, text
f219460db8db952415b5201ba01df99ce4999004 Thu Sep 10 13:46:42 2026 -0700
genark: pass the liftOver accession list as an slName list, refs #38328
genarkLiftOverDbs() took a pre-quoted SQL fragment that its callers
assembled. It now takes a struct slName list and builds the query
itself with sqlDyStringCreate, so no caller writes SQL text.
Accessions that do not start with GC are skipped, since nothing else
can match the table. hdb.c and hgConvert.c updated for the new
signature.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/inc/hVarSubst.h
- lines changed 9, context: html, text, full: html, text
c0e8fa6df3a0bd406c4188d49ee00f20aef203e5 Mon Sep 7 12:07:18 2026 -0700
Substitute trackDb variables in hub track description pages
A hub's description page comes straight off the hub's web server and has
never been through variable substitution, so a $db or $parentTrack in it
reached the reader as literal text. Native trackDb pages are fine, since
hgTrackDb substitutes them when it loads trackDb, but there was no
equivalent step for a hub.
hgc's getTrackHtml and hgTrackUi's trackUi both call hVarSubstTrackDbHtml
on a hub track's html. Only a short list of variables is recognized there and nothing is an
error, because a hub page written before this existed can easily contain
a dollar sign inside a shell example, and silently rewriting that would
be worse than not substituting at all.
Adds $parentTrack, the name of the container a track sits in, which is
what a subtrack description page needs to link back to its superTrack or
composite. Views are skipped, since a view has no page of its own, and
the hub_<id>_ prefix is kept so the name works as hgTrackUi's g=
parameter. Documents $track, $parentTrack and $hgsid in trackDb/README.
refs #37599
- src/hg/inc/quickLift.h
- lines changed 18, context: html, text, full: html, text
d0ed51d077718213c0b843a89fd6f36ac32f1804 Fri Sep 4 11:39:28 2026 -0700
quickLift: lift alignments, and let psl and bigPsl tracks into the hub
Every type quickLift handled so far is one set of coordinates on one genome, so
the lift is a call to remapBlockedBed. An alignment carries coordinates on both
sides at once, with a block start on each, so moving the genome side means
splitting and trimming blocks while keeping the other side lined up with them.
quickLiftPsl does that with pieces that were already in the tree: pick the chain
with liftOverChainForRange, the same one the bed path picks, turn it into a
mapping alignment with chainToPsl plus pslSwap, then pslTransMap. The mapping
alignment is kept per chain, since rebuilding it for every item costs nothing at
gene zoom and a great deal zoomed out.
Two things pslTransMap does had to be undone. It recounts match and mismatch off
the blocks, which would read every lifted alignment as a perfect match and draw
every item at full shade, so quickLiftPslCounts puts the original counts back,
scaled by how much of the alignment survived. And it leaves a protein alignment
in nucleotide space, which the base alignment view rejects, so
quickLiftPslBackToProtein returns the query side to protein units when the lift
did not split a codon.
quickLiftChainHash hands out the chains for a reference range for callers that
collected their items some other way. validateOneTdb now accepts psl and bigPsl,
and bigPsl joins bigBed and bigWig in the bigDataUrl fill-in.
refs #38249
- lines changed 28, context: html, text, full: html, text
e70c7f68f61a3477631bd364c31f99c157528fc8 Fri Sep 4 12:48:02 2026 -0700
quickLift: lift chains, and let chain and bigChain tracks into the hub
A chain is an alignment between the assembly the track came from and some other
species, so lifting one composes two alignments and leaves the user with chains
between the assembly on screen and that species.
quickLiftChain does it by the same route the psl work uses: chainToPsl, then
quickLiftPsl, then chainFromPsl, which is the inverse chainToPsl never had. It
does not modify the chain handed to it. That matters more than it sounds: every
chain loader leaves the header describing the whole chain while loading only the
blocks in range, so the header has to be corrected for the conversion, but the
callers still want the original. The details page reports it, and the track
sorts on it, which is what puts a lifted chain in the same row as the native one.
quickLiftSourceRanges hands back the ranges in the other assembly that the window
maps to, for callers whose items cannot be had from a query quickLiftSql knows
how to make. It works those ranges out by intersecting the window with each
chain block rather than taking the whole block, which matters here because one
block can be enormous: hg19 and hg38 run identical for 12.8Mb on chr7, and
asking for all of it turned a 39 chain window into a 1892 chain one.
quickLiftIsOwnChainTrack keeps quickLift's own chain track out of all this. That
stanza carries quickLiftUrl and quickLiftDb like any lifted track and is loaded
by bigChainLoadItems like any bigChain, but its data is already in reference
coordinates. The giveaway is that its bigDataUrl is the quickLift chain file.
validateOneTdb now accepts chain and bigChain. netAlign is still refused: a net
is a hierarchy of gaps rather than a plain alignment and wants its own thought.
refs #38249
- lines changed 8, context: html, text, full: html, text
5fc426954da9ceb7dc42b9760858bbd1189760e1 Fri Sep 4 13:08:14 2026 -0700
quickLift: lift MAF blocks, and let bigMaf and wigMaf tracks into the hub
Most of this was already written. mafSubset does the part that looked hard,
which is recomputing every row's start and size when columns are taken away, so
what was left was deciding where to cut.
quickLiftMafs cuts a block at every chain block boundary. Inside one chain block
the two assemblies run in step, so the columns carry over untouched and only the
first row's coordinates change. Across a boundary the reference either loses
bases or gains them, and either way the block can no longer be one contiguous run
on the reference, which is the one thing a MAF block has to be.
The reference row is named with the assembly name minus any hub prefix, since
that is the name the maf drawing code builds when it goes looking for it.
A minus strand chain turns the block over, so every row is turned over with it and
the forward start comes from the far end of the run. That path is written but has
not been exercised: no minus strand quickLift chain wins in a window I could
find.
validateOneTdb accepts bigMaf and wigMaf. Plain maf is left out on purpose:
those tracks are drawn by mafTrack.c, which has no quickLift path, so offering
them would hand back a track read from the wrong assembly.
refs #38249
- lines changed 5, context: html, text, full: html, text
cec5ead054791f2a6601f308a56d280c08bd8ef7 Fri Sep 4 13:32:36 2026 -0700
quickLift: do not take the lift path on half a pair of settings
A hub can set quickLiftDb without setting quickLiftUrl, and nothing filters hub
trackDb settings. The alignment loaders gated on quickLiftDb alone, so such a
stanza took the lift path with no chain file and hgTracks died in
endsWith(NULL, ".bb") from bigChainGetLinkFile, by way of quickLiftLoadChains.
Verified: SIGSEGV in strlen from common.c:1653, page truncated mid-HTML. This is
reachable on a production browser now that bigChain and bigMaf are liftable,
because those carry their own bigDataUrl and so need no trustTrackDb.
quickLiftIsLifted requires both halves, and every gate now uses it, which also
settles the two different predicates that were in use for the same question.
quickLiftLoadChains returns an empty list for a NULL file as well, so the older
bed and genePred callers are covered whatever their gate does.
quickLiftSql now checks that a row has at least as many columns as the loader is
going to read. The native psl loader has always checked this, and the quickLift
path replacing it did not, so a table of the wrong type walked off the end of the
row; the psl caller now states the 21 columns it needs.
quickLiftMafs held a maf component name in a fixed buffer through safecpy, which
aborts rather than truncates, so a long name from a hub took hgTracks down. It
clones instead.
htcBigPslAliInWindow used a trackDb pointer its lookup can leave NULL, which its
sibling htcBigPslAli already checked for. And aliTrackParam formats a URL
parameter into a fixed buffer with safef, which aborts on a long one.
refs #38249
- lines changed 5, context: html, text, full: html, text
992aeef92fea7be25a2acd916578898649212a32 Mon Sep 7 11:31:07 2026 -0700
quickLift: gate the alignment lift behind an hg.conf flag, refs #38249
Add browser.quickLiftAlignments, default FALSE, so the alignment lift ships
dark and a machine turns it on with browser.quickLiftAlignments=on. It sits
beside browser.quickLift, the gate on the rest of the feature.
quickLiftAlignmentsEnabled() in hg/lib/quickLift.c is the one read, and
validateOneTdb in hg/lib/trackHub.c is the one place that asks it, before an
alignment track may enter a quickLift hub. That is the only door:
quickLiftUrl and quickLiftDb, the pair every lift path keys off, are written
by the quickLift hub writer and by nothing else, so with the flag off an
alignment track never gets them and the lifting, drawing and details code
behind them cannot be reached. pslTrack.c, chainTrack.c, wigMafTrack.c,
bigBedTrack.c and hgc.c are unchanged.
With the flag off hgConvert lists psl, bigPsl, chain, bigChain, maf, bigMaf
and wigMaf tracks in its "type is not supported by QuickLift" table, which is
what it did before this work. A hub built while the flag was on keeps working
after it is turned off, since its stanzas are already in the hub file in
trash, so this holds the feature back from people who have not used it rather
than switching off a session that has.
Read the hg.conf half with a literal cfgOptionBooleanDefault rather than
cartOrCfgOption so harvestHgConf.py can see it; a cart variable of the same
name still overrides it. Register the flag in hgConfCatalog.py with
role="gate" so the sunset report tracks it, and turn it on in
confs/hgwdev.hg.conf.
- src/hg/inc/sessionData.h
- lines changed 16, context: html, text, full: html, text
11d5b26d9798079ec5adedc103216ce818e2da5e Thu Sep 10 05:19:27 2026 -0700
Widen the session data directory hash from 8 to 10 hex characters
sessionDirFromNames() named a session's durable data directory with 8 hex
characters of md5(sessionName). 32 bits was fine while a directory only had
to be unique among one user's sessions, but every anonymous "Share a link"
session belongs to the single reserved user "l", which makes it a birthday
problem across all of them: two unrelated sessions land in the same directory
more likely than not at around 77,000 anonymous sessions, and at 500,000 we
would expect about 29 such pairs. Two sessions sharing a directory means
cleaning up one takes the other's custom track files with it.
sessionDirHashLen is now 10 (40 bits), which moves the even-odds point past a
million sessions. The two fan-out levels snapshotSessionDir() added for user
"l" do not help here, since they are a prefix of the same hash: they spread
the entries over 65536 directories but leave the number of distinct leaf names
unchanged.
Both directory layouts change name as a result, so snapshotCleaner would have
walked past anything written earlier and orphaned its files. Both dir-naming
functions grew a hashLen argument, sessionDirHashLenLegacy records the old
value, and snapshotCleanAnon() now tries the old spelling of both layouts as
well as the new one.
Existing sessions keep working either way: the cart stores the absolute path
of each durable file, so nothing looks a session's directory up by name except
the cleaner.
refs #10138
- src/hg/inc/snapshotSession.h
- lines changed 32, context: html, text, full: html, text
b50a995e326f37cb2248e9a04628c15fb1b58215 Mon Sep 14 05:40:04 2026 -0700
My Sessions no longer hides a user's own "__" sessions, refs #38313
Both My Sessions listings decided a row was a share token by testing the
session name for the "__" prefix, so they also hid sessions a user had named
that way themselves - eleven of them on the RR, across four accounts, and
their owners could no longer rename, describe, reshare or delete them.
The prefix is a naming convention we follow, not a namespace we own. The
authoritative mark is the "snapshotType <type>" line saveSnapshotSession()
already writes into the settings column, so test that instead. Both queries
already select settings, so no query changes.
snapshotTypeFromSettings() walks the settings lines rather than calling
raFromString(): it runs once per listed row, and a hash there costs ~600ns and
three allocations for every session that has a description, against ~30ns and
none for the walk. Rows with empty settings, the common case, short-circuit
in both.
- src/hg/inc/trackHub.h
- lines changed 4, context: html, text, full: html, text
d11a21c48cdc2cebe0a97779b510df8317b980cd Mon Sep 7 06:29:43 2026 -0700
bigNet: gate the track type behind an hg.conf flag, refs #20824
Add the boolean hg.conf setting bigNet, default FALSE, so the type ships
dark and a machine turns it on with bigNet=on.
trackHubBigNetEnabled() in hg/lib/trackHub.c is the one read; the four
places that accept or advertise the type ask it. validateOneTrack drops
bigNet from the hub track type allowlist, validateOneTdb drops it from the
types quickLift will lift, hubCheck drops it from VALID_TRACK_TYPES and
from the message listing the valid types, and hubApi does not add it to
supportedTypes. netToBigNet, netTrack.c, chainNetDbLoad.c and the hgc
details code are unchanged, since they cannot be reached once a bigNet
track will not load.
With the flag off hgTracks does not abort the hub. It draws the track row
as a bigWarn bar reading "Unsupported type 'bigNet ...'" and the rest of
the hub loads normally.
Register the flag in hgConfCatalog.py with role="gate" so the sunset
report tracks it.
- src/hg/inc/trashDir.h
- lines changed 5, context: html, text, full: html, text
59e2bbf2d3a99e5f18436a756e1e5c8b64308e33 Tue Sep 8 09:21:02 2026 -0700
Copy a track collection's hub file when the program that writes it asks for a copy, instead of on every session load.
cartCopyLocalHubs ran on every session load, in every CGI and in four more places in
hgSession. For each customComposite-<db> cart variable it copied the collection's hub
file to a fresh trash name and registered the copy in hgcentral.hubStatus to get a hub
id. Nothing removed the old row. 81% of the 3.1 million rows in hubStatus on the RR are
these dead registrations, and each load also took the central_hubStatus advisory lock
and wrote to a MyISAM table, which serializes every concurrent load of a shared session
that carries a collection.
The copy itself is needed. A saved session's hub file lives under sessionDataDir and
every load of that session names it, so hgCollection must not write it in place. It was
just being made for every load, and almost no load is followed by an edit. hgCollection
is the only program that writes one of these files, so it now asks for its own copy:
main() calls cartRequestLocalHubCopy() before it opens the cart, and cartNew() makes the
copy for it.
That has to be a property of the program rather than of the request, and the copy has to
be made at cart open, above hubConnectLoadHubs. Copying the file gives the hub a new id,
the hubs are loaded during cart open, and trackList carries that id in every track name,
so a copy made any later leaves printTrackDbListToHub looking for the collection under
an id trackList does not have and the hub file comes out with a header and no tracks. A
condition that instead tries to work out whether this particular request will write
cannot be correct either: "cmd" is not in hgTracks' excludeVars, so it persists in the
cart and hgCollection dispatches on the cart rather than on the CGI variable, a settings
file can carry it, and a command-line run has no SCRIPT_NAME to test.
Three other parts. copyLocalHubs skips a hub the cart already owns, so repeated edits
reuse one file and one hub id rather than renumbering the hub on every drag; the test
requires a plain file, because saveTrackFile leaves the trash name behind as a symbolic
link to the durable copy and writing through that link would rewrite the saved session's
own file. copyLocalHubs also screens the path with isServerUserFilePath before opening
it, the way getHubName does, and selects on customComposite-<db> with the dash so that
the names it acts on are the ones fileNameCartVarPrefixes screens. saveTrackFile copies
when the source is a local hub outside trash that is not under this session's own
directory, so loading one session and saving it under a new name gives the new session
its own file instead of a reference into the first one's directory. pathIsUnderDir is no
longer static in trashDir.c.
All of it is behind the hg.conf gate collectionHubCopyOnWrite, default off, which
reproduces the old behavior exactly. Retiring the gate is not uniform: the body of
cartCopyLocalHubsOnSessionLoad is the old behavior and that function and its five
callers go away with the gate, while the other three tests lose only the gate term.
Measured on hgcentraltest with one binary. With the gate off, five loads of a session
carrying a collection add five hubStatus rows and three later edits add none; with it
on, the five loads add none and the first edit adds one. Saving a loaded session costs
two rows off and one on. The resulting hub file is byte identical either way apart from
the hub id, and the file the saved session names is unchanged by md5 and mtime,
including after an edit made straight after a save, and including when the cart names a
trash symbolic link into session storage.
refs #38273
- src/hg/js/assemblySearch.js
- lines changed 20, context: html, text, full: html, text
b761b0b0fc3cd9e830db0f11a536e0da29d7c9e5 Wed Sep 9 08:58:30 2026 -0700
A copy-to-clipboard button goes back to its own label after three seconds
Saying "Copied" and then keeping that as the label left no sign that the button
could be used again. It now says "Copied" for three seconds and then puts back
its own label and icon. A second copy while the message is up restarts the
three seconds rather than adopting "Copied" as the label to go back to.
The assembly search page carries its own borrowed copy of copyToClipboard and
does not load utils.js, so it gets the same treatment there.
refs #38294
- lines changed 60, context: html, text, full: html, text
562b1b24f9d7cf5157c799f733ba219e2e9f7da9 Wed Sep 9 09:06:30 2026 -0700
Assembly search page uses the shared copyToClipboard instead of its own copy
The page carried a copy of copyToClipboard marked "borrowed this code from
utils.js", and the two had already drifted apart: the fix that stops the button
claiming a copy that a browser refused went into one and not the other. The
page now loads utils.js, as nine other static pages already do, and its own
copy is gone. jquery is already loaded by the page header, so nothing else was
needed.
The page also declared a global named debug, which utils.js declares too. Both
start out false and the two uses in utils.js are in functions this page never
calls, so nothing was broken, but the page flag is now searchDebug. The debug
URL parameter and stateObject.debug keep their names.
refs #38294
- src/hg/js/external/dataTables.rowReorder-1.5.1.min.js
- lines changed 0, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- src/hg/js/external/makefile
- lines changed 1, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- src/hg/js/facetedComposite.js
- lines changed 11, context: html, text, full: html, text
b678946c7d246ae01d552242ac0296b8694d5fc3 Tue Sep 8 06:15:43 2026 -0700
hgTrackUi: faceted composite says "Samples" when data types are on, and stops paging a short table
The two filter tabs were hardcoded to "All Tracks" and "Active Tracks", which is
wrong for a composite that uses dataTypes: there a row is a sample, standing for
as many tracks as there are active data types. The file already had an itemLabel
that resolves to "samples" or "tracks" and was feeding the DataTables strings, so
the tabs now use the same variable. A composite without dataTypes reads exactly
as before.
Page length was a flat 25, so a table of 41 samples hid a third of itself behind
a pager for no good reason. Tables under 50 rows now start out showing
everything; the length menu still offers 10/25/50/100/All for bigger ones.
refs #36210
- lines changed 471, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- lines changed 7, context: html, text, full: html, text
64d40b9204a60274a49330c6b61d146e5cdc6d87 Wed Sep 9 05:54:34 2026 -0700
Faceted composite: a container's max display mode must clamp a pinned child, not drop it
Selecting only CpG methylation on the Fiber-seq compendium and hitting
submit drew nothing at all, with no message to say why.
Two things combined. The children of a faceted composite can be pinned
to one display mode with onlyVisibility, and five of the six Fiber-seq
data types are pinned to full because they are signal tracks. The
container's own "Maximum display mode" is a ceiling over those children,
and it was set to pack. tdbVisLimitedByAncestors() then took a pinned
child that sat above the ceiling and returned hide for it, so every
pinned-to-full track disappeared and only the peaks, pinned to dense,
came through. With every data type but peaks unchecked, that left an
empty image.
A ceiling should limit a child, not delete it, so use tvMin the same way
the unpinned case a line below already did. A bigWig draws the same at
pack as at full, minus the horizontal grid, so nothing is lost here.
The page was also asking for the wrong ceiling. On a faceted composite
the dropdown is a ceiling rather than a display mode, since each child
carries its own, so taking the container out of hide should ask for full,
the one value that clips nothing. Pack was chosen on the reasoning that
it suits a mix of signal and feature tracks, which is the right instinct
for a plain composite and the wrong one here.
refs #36210
- lines changed 40, context: html, text, full: html, text
d033cea2063e9362949baf5b4d8b837597173a0d Thu Sep 10 05:16:05 2026 -0700
Address the code review of the Sep 9 commits
Faceted composite: text that comes from a hub - a metadata column's
description, the name and title of a data type, and the values quoted back in
the "could not load the metadata" row - is put on the page as text rather than
as markup. The three places built their markup from template strings, so a
value carrying angle brackets or a quote was read as HTML: the column
description now goes through the shared htmlEncode() once where the header is
parsed, and the other two build their elements as nodes. The error row reads
better for it as well, since a value with brackets in it used to disappear from
the message that was meant to show it.
The saved UI state keys on the assembly as well as the metadata id. localStorage
is per-origin, so two assemblies whose tracks share a name were sharing one
entry, and a row order dragged on one came back on the other over a different
set of samples. hgTrackUi passes the database down for it. State saved under the
old key is dropped, which costs a facet selection or a page length.
Imprinting: the five subtrack description pages link back to the container as
hgTrackUi?db=$db&g=$parentTrack, without the hgsid. Native trackDb html is
substituted by hgTrackDb as it loads the table, where there is no cart, so
${hgsid} came out empty and the link read 'hgsid=&g=...'. Matches what the
Fiber-seq pages already do. The makeDoc note that described the old form is
updated with the reason.
UniProt otto: README.txt lists all eight things that reach runLog.txt. It had
four, and was missing LOCKED, along with PREFLIGHT-FAIL, END and INTERRUPTED.
refs #36210
refs #37599
refs #38300
- src/hg/js/hgBlat.js
- lines changed 14, context: html, text, full: html, text
595531f894a27e59e18b32436f6cb932fa4374a0 Tue Sep 8 13:03:44 2026 -0700
BLAT form character counter: per-type limits passed through from the C constants. refs #38293
The counter showed the 75,000 DNA limit for every query type; protein and translated queries
are capped at 10,000, so an oversized protein paste looked fine until the server rejected it.
The per-sequence limits are now named constants in hgBlat.c, emitted into hgBlatFormData and
read by the counter, which keys on the Query type dropdown and recounts when it changes -
so the numbers cannot drift apart again. BLAT's guess counts against the DNA limit.
- lines changed 20, context: html, text, full: html, text
5a249cd50f592a3b7598110eec792cfc942fab3f Wed Sep 9 08:52:06 2026 -0700
Address the v504 code review
hgSession: an anonymous share name that arrives with the request is saved only
when it is not already in the table. Every anonymous link sits under the one
reserved user "l", so a name already there stays as it is and the caller is told
so. The top-right Share dialog is unaffected, since it passes a name it has
just reserved and such a name does not exist yet. Snapshot names are now left
out of both My Sessions listings, which is what their "__" prefix has claimed
all along.
Share dialog: "Create link & copy" reports the copy instead of promising it.
copyToClipboard says whether the text reached the clipboard, the dialog passes
that on when a browser refuses, and it tries the asynchronous clipboard API
before giving up. The preview is built with the same encoding the server uses,
so a name holding a hyphen or a slash previews as the link that really gets
made. Cancelling out of the name editor no longer copies a second time, and a
reply with no link in it says so rather than showing "undefined".
hgBlat: a second click on the share button while the first request is still out
no longer mints a second snapshot session, and a box dismissed during the wait
stays closed.
Also: a snapshot moves a cart value into durable storage only when it is a
trash path, the way sessionData's own callers check; sqlAddressMatch keeps to
its own documented precondition when handed an empty address; alphaGenomeToWig
compares its output with its input rather than with itself, rejects a position
that is not all digits and skips an empty score; and hgc's default iframe width
reaches the browser as one percent sign.
refs #38294
- lines changed 1, context: html, text, full: html, text
c0a6706dea55f019923332c5f51dfe64fc420dc8 Wed Sep 9 12:15:27 2026 -0700
Comment touch-ups from CR: reattach delayFraction's continuation comment, drop an imprecise history aside in the counter comment. refs #38293
- lines changed 2, context: html, text, full: html, text
e6eaaaa2b64a0ac14f17f5011f27f294d5287a23 Wed Sep 9 16:30:48 2026 -0700
Escape the ampersands in the BLAT share box and rename dialog hgSession hrefs. Feedback from CR. refs #38292
- src/hg/js/hgMyData.js
- lines changed 28, context: html, text, full: html, text
ac3191c7e768e8c0c8f6c376526f029ad39f4e1b Thu Sep 10 05:57:18 2026 -0700
hgHubConnect: on mirrors, show the Hub Upload tab as instructions, not a dialog
On a site that is not the login host the tab used to render the full upload
UI with a warning dialog layered over it. Show only a message instead: where
to upload, a direct link to the Hub Upload tab on the US site, and how to get
an uploaded hub onto this site via Connected Hubs or a hub-connecting URL.
refs #38323
- src/hg/js/hgSession.js
- lines changed 100, context: html, text, full: html, text
fc8de100a3437b9fc33bdeb0f459ca2a93e2f318 Wed Sep 9 08:14:32 2026 -0700
hgSession: address the code review of the new Sessions page
Rename and unshare now keep the public listing's thumbnail with the session it
belongs to. The picture's file name is built from the encoded session name, so
renaming a listed session left the listing pointing at nothing and the old file
behind, and dropping a session from the listing to a plain shared link kept the
picture. The classic page had the same problem in a subtler form: it removed the
thumbnail after the row had already been renamed, so the old file survived.
Saving under a name that is already in use asks before it replaces that session,
using the failIfExists reply that the top-right Share a link menu already relies
on. The description and "only I can load it" steps that follow a save now report
a failure instead of reloading in silence, and what thumbnailAdd has to say when
it cannot build a picture reaches the user instead of being freed unread.
A session description no longer travels through a title attribute. The tooltip
machinery in utils.js inserts its text with innerHTML and an attribute is decoded
on the way, so a description containing angle brackets was interpreted as markup
rather than shown as typed. It is attached, escaped, after each table draw, which
also gives the rows DataTables renders later the same styled mouseovers as the
rest of the page.
Also: the AJAX endpoints say so when there is no session by that name, instead of
reporting a no-op as a success; the new page always offers its way back to the
classic page, since the cart variable that got the user there sticks; and four
unused CSS rules, a dead element lookup and a dead local are gone. hgConfCatalog
cited the wrong ticket for the two sessionNewPage flags.
refs #38180, refs #38157
- lines changed 16, context: html, text, full: html, text
0cdd15681f10789132dc9e88fcf4be23bd47ee4b Wed Sep 9 09:14:37 2026 -0700
New Sessions page: Replace on the save card keeps the session's sharing level
Confirming Replace re-saved through the save endpoint, which always writes a
session as shared by link. Replacing a session that was in the public listing
took it off the list, and replacing a private one made it loadable by anyone
with the link. Replace now goes through the overwrite endpoint, which reads the
row's sharing level and keeps it - the same endpoint the floppy button on each
table row already uses. The description and the "only I can load it" box on
the save card are still applied afterwards, refs #38311
- src/hg/js/hgc.scatterPlot.js
- lines changed 567, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- src/hg/js/hui.js
- lines changed 7, context: html, text, full: html, text
c94e417525c00e37443c212e0ded342ef7809e56 Mon Sep 7 12:07:18 2026 -0700
hgTrackUi: add "Hide all tracks" and "Show all tracks" buttons to the superTrack page, and drop "Apply to all visible tracks". The two new buttons cover the two things people actually do on a container page. "Show all tracks" asks for pack and falls back to full for tracks that have no pack, i.e. signal tracks, so a container of bigWigs comes up in full. The old "Apply to all visible tracks" button needed two sentences of help text to distinguish it from "Apply to all tracks" and is gone; the dropdown plus "Apply to all tracks" still covers dense and squish. refs #38281
- src/hg/js/makefile
- lines changed 1, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- src/hg/js/topLinks.js
- lines changed 6, context: html, text, full: html, text
c1ac48b0feb9a8ff0f0bb16d7aff15d417d6772b Tue Sep 8 12:38:42 2026 -0700
Share dialog: scope the snapshot-lifetime wording to the BLAT alignment share only. Feedback from CR. refs #37996
The softer durability text (link valid for years, save into a Session for permanence) keyed
on the generic url mode, so hgTrackUi's page-share link and the hgc item-popup share, both
plain non-expiring URLs, showed misleading copy. The wording is now behind an explicit
snapshot flag that only the BLAT alignment page's share passes; every other caller keeps
"Links never time out". Also removes a stray blank line in hgc.c from the same review.
- lines changed 60, context: html, text, full: html, text
5a249cd50f592a3b7598110eec792cfc942fab3f Wed Sep 9 08:52:06 2026 -0700
Address the v504 code review
hgSession: an anonymous share name that arrives with the request is saved only
when it is not already in the table. Every anonymous link sits under the one
reserved user "l", so a name already there stays as it is and the caller is told
so. The top-right Share dialog is unaffected, since it passes a name it has
just reserved and such a name does not exist yet. Snapshot names are now left
out of both My Sessions listings, which is what their "__" prefix has claimed
all along.
Share dialog: "Create link & copy" reports the copy instead of promising it.
copyToClipboard says whether the text reached the clipboard, the dialog passes
that on when a browser refuses, and it tries the asynchronous clipboard API
before giving up. The preview is built with the same encoding the server uses,
so a name holding a hyphen or a slash previews as the link that really gets
made. Cancelling out of the name editor no longer copies a second time, and a
reply with no link in it says so rather than showing "undefined".
hgBlat: a second click on the share button while the first request is still out
no longer mints a second snapshot session, and a box dismissed during the wait
stays closed.
Also: a snapshot moves a cart value into durable storage only when it is a
trash path, the way sessionData's own callers check; sqlAddressMatch keeps to
its own documented precondition when handed an empty address; alphaGenomeToWig
compares its output with its input rather than with itself, rejects a position
that is not all digits and skips an empty score; and hgc's default iframe width
reaches the browser as one percent sign.
refs #38294
- lines changed 3, context: html, text, full: html, text
b761b0b0fc3cd9e830db0f11a536e0da29d7c9e5 Wed Sep 9 08:58:30 2026 -0700
A copy-to-clipboard button goes back to its own label after three seconds
Saying "Copied" and then keeping that as the label left no sign that the button
could be used again. It now says "Copied" for three seconds and then puts back
its own label and icon. A second copy while the message is up restarts the
three seconds rather than adopting "Copied" as the label to go back to.
The assembly search page carries its own borrowed copy of copyToClipboard and
does not load utils.js, so it gets the same treatment there.
refs #38294
- src/hg/js/utils.js
- lines changed 10, context: html, text, full: html, text
5a249cd50f592a3b7598110eec792cfc942fab3f Wed Sep 9 08:52:06 2026 -0700
Address the v504 code review
hgSession: an anonymous share name that arrives with the request is saved only
when it is not already in the table. Every anonymous link sits under the one
reserved user "l", so a name already there stays as it is and the caller is told
so. The top-right Share dialog is unaffected, since it passes a name it has
just reserved and such a name does not exist yet. Snapshot names are now left
out of both My Sessions listings, which is what their "__" prefix has claimed
all along.
Share dialog: "Create link & copy" reports the copy instead of promising it.
copyToClipboard says whether the text reached the clipboard, the dialog passes
that on when a browser refuses, and it tries the asynchronous clipboard API
before giving up. The preview is built with the same encoding the server uses,
so a name holding a hyphen or a slash previews as the link that really gets
made. Cancelling out of the name editor no longer copies a second time, and a
reply with no link in it says so rather than showing "undefined".
hgBlat: a second click on the share button while the first request is still out
no longer mints a second snapshot session, and a box dismissed during the wait
stays closed.
Also: a snapshot moves a cart value into durable storage only when it is a
trash path, the way sessionData's own callers check; sqlAddressMatch keeps to
its own documented precondition when handed an empty address; alphaGenomeToWig
compares its output with its input rather than with itself, rejects a position
that is not all digits and skips an empty score; and hgc's default iframe width
reaches the browser as one percent sign.
refs #38294
- lines changed 16, context: html, text, full: html, text
b761b0b0fc3cd9e830db0f11a536e0da29d7c9e5 Wed Sep 9 08:58:30 2026 -0700
A copy-to-clipboard button goes back to its own label after three seconds
Saying "Copied" and then keeping that as the label left no sign that the button
could be used again. It now says "Copied" for three seconds and then puts back
its own label and icon. A second copy while the message is up restarts the
three seconds rather than adopting "Copied" as the label to go back to.
The assembly search page carries its own borrowed copy of copyToClipboard and
does not load utils.js, so it gets the same treatment there.
refs #38294
- src/hg/lib/bigNet.as
- lines changed 31, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/lib/bigNet.c
- lines changed 264, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/lib/botDelay.c
- lines changed 13, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/lib/cart.c
- lines changed 87, context: html, text, full: html, text
59e2bbf2d3a99e5f18436a756e1e5c8b64308e33 Tue Sep 8 09:21:02 2026 -0700
Copy a track collection's hub file when the program that writes it asks for a copy, instead of on every session load.
cartCopyLocalHubs ran on every session load, in every CGI and in four more places in
hgSession. For each customComposite-<db> cart variable it copied the collection's hub
file to a fresh trash name and registered the copy in hgcentral.hubStatus to get a hub
id. Nothing removed the old row. 81% of the 3.1 million rows in hubStatus on the RR are
these dead registrations, and each load also took the central_hubStatus advisory lock
and wrote to a MyISAM table, which serializes every concurrent load of a shared session
that carries a collection.
The copy itself is needed. A saved session's hub file lives under sessionDataDir and
every load of that session names it, so hgCollection must not write it in place. It was
just being made for every load, and almost no load is followed by an edit. hgCollection
is the only program that writes one of these files, so it now asks for its own copy:
main() calls cartRequestLocalHubCopy() before it opens the cart, and cartNew() makes the
copy for it.
That has to be a property of the program rather than of the request, and the copy has to
be made at cart open, above hubConnectLoadHubs. Copying the file gives the hub a new id,
the hubs are loaded during cart open, and trackList carries that id in every track name,
so a copy made any later leaves printTrackDbListToHub looking for the collection under
an id trackList does not have and the hub file comes out with a header and no tracks. A
condition that instead tries to work out whether this particular request will write
cannot be correct either: "cmd" is not in hgTracks' excludeVars, so it persists in the
cart and hgCollection dispatches on the cart rather than on the CGI variable, a settings
file can carry it, and a command-line run has no SCRIPT_NAME to test.
Three other parts. copyLocalHubs skips a hub the cart already owns, so repeated edits
reuse one file and one hub id rather than renumbering the hub on every drag; the test
requires a plain file, because saveTrackFile leaves the trash name behind as a symbolic
link to the durable copy and writing through that link would rewrite the saved session's
own file. copyLocalHubs also screens the path with isServerUserFilePath before opening
it, the way getHubName does, and selects on customComposite-<db> with the dash so that
the names it acts on are the ones fileNameCartVarPrefixes screens. saveTrackFile copies
when the source is a local hub outside trash that is not under this session's own
directory, so loading one session and saving it under a new name gives the new session
its own file instead of a reference into the first one's directory. pathIsUnderDir is no
longer static in trashDir.c.
All of it is behind the hg.conf gate collectionHubCopyOnWrite, default off, which
reproduces the old behavior exactly. Retiring the gate is not uniform: the body of
cartCopyLocalHubsOnSessionLoad is the old behavior and that function and its five
callers go away with the gate, while the other three tests lose only the gate term.
Measured on hgcentraltest with one binary. With the gate off, five loads of a session
carrying a collection add five hubStatus rows and three later edits add none; with it
on, the five loads add none and the first edit adds one. Saving a loaded session costs
two rows off and one on. The resulting hub file is byte identical either way apart from
the hub id, and the file the saved session names is unchanged by md5 and mtime,
including after an edit made straight after a save, and including when the cart names a
trash symbolic link into session storage.
refs #38273
- lines changed 8, context: html, text, full: html, text
905b9cb05eeaca7f2dcda42fc6abdb95a2d2da7f Wed Sep 9 05:29:28 2026 -0700
no captcha for a command-line CGI run, and version the detailsScript module URL
Two small fixes to things noticed while adding the scatterPlot plot type.
A CGI run from the command line got the Cloudflare Turnstile challenge page
instead of the output the caller asked for, which makes "./hgc db=hg38 g=x" -
the quickest way to see what a CGI emits - useless without a hand-made hg.conf.
There is no browser to solve a captcha in that situation. printCaptcha() now
returns early when cgiWasSpoofed(). That flag cannot be set from an HTTP
request: cgiFromCommandLine() returns early and leaves it FALSE whenever the
web server has set REQUEST_METHOD. Checked that a plain argument-style run is
now clean, that a run which fakes the web environment with QUERY_STRING still
gets the captcha, and that an HTTP request behaves exactly as the unmodified
binary does.
The detailsScript module was loaded from a hardcoded import('../js/hgc.X.js'),
bypassing webTimeStampedLinkToResource(), so it was the one script on the page
with no ?v=<mtime>. That is the mechanism that flushes a browser's cache when
the CGI version changes and that keeps a mirror from pairing an old static file
with new CGIs, and without it a cached module could be handed newer bedDetails
JSON than it was written for. Now built through the helper, which also fixes the
already-shipped histogram type. The helper errAborts on a missing file and the
plot type comes from a hub, so a plot type with no module installed falls back to
the plain path: a silent failed import as before, rather than one bad hub setting
taking down the whole details page.
refs #35415
- lines changed 5, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- lines changed 29, context: html, text, full: html, text
cf9f4cb7f55c7beb8ad5f11118656a60770a71a5 Thu Sep 10 01:02:36 2026 -0700
Move the extra-HTTP-header list into cheapcgi, and write the header only once
Follow-on to the cgiPrintContentType() refactor.
cart.c owned the mechanism for adding headers ahead of the content type: a
global slPair list plus addHttpHeaders() to print it. That put it in hg/lib,
out of reach of the CGIs and library code that do not use a cart, even though
nothing about it is cart-specific. It now lives next to cgiPrintContentType()
in lib/cheapcgi.c, behind cgiAddHttpHeader(name, value) instead of a bare
global, and cgiPrintContentType() writes the queued headers itself. The one
caller, hgTracks/mainMain.c, reads the same but no longer reaches into cart.h
for it. cspWriteResponseHeader() stays in hg/lib where it belongs, since it
needs hg.conf; cartWriteHeaderAndCont() calls it directly now, the way the
other ten callers already do.
cgiPrintContentType() also writes at most once per process now. A second
content type cannot reach the browser as a header - it lands in the page body
as text - so the later caller is always the mistaken one. cart.c had a private
cartDidContentType flag for exactly this, covering only the flows that went
through the cart; the guard is now in the one function every flow shares, and
cartDidContentType is gone. Its public equivalent, cgiDidContentType(), is
what cartWriteHeaderAndCont() checks so it does not write a second cookie.
Verified: make libs, make cgi and the lib test suite are clean, hgTracks still
emits Cache-Control: no-store, and hgTracks, hgc and hgTables each emit exactly
one Content-Type on both their html and their text paths.
- src/hg/lib/chainNetDbLoad.c
- lines changed 56, context: html, text, full: html, text
42cfb8cd4a22209d8183ffcad97e5ca5e9ae070b Fri Sep 4 12:47:47 2026 -0700
chainNetDbLoad: add chainLoadRange, every chain in a range with its blocks
chainLoadIdRange loads one chain by id, and the bigChain reader takes -1 to mean
every chain in the range, but there was no SQL equivalent of the latter. This is
it: one query for the headers, one for the links, and each link handed to the
chain it belongs to.
The blocks are sorted rather than just reversed. A bin indexed range query does
not return rows in position order, and anything that walks a chain's blocks
expects them ascending, which is why chainLinkAddResult sorts them too.
refs #38249
- lines changed 80, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 98, context: html, text, full: html, text
f29b65452a1cca8a5bc62f007100b313e589d036 Sun Sep 6 15:11:36 2026 -0700
quickLift: lift a bigNet track, refs #20824
A net is not an ordinary track to lift. It is an alignment of two assemblies,
so only its target side moves to the new reference; the query side names a third
assembly and is carried across untouched. And the browser draws a net by
recursion, so what really has to survive the lift is the tree: a row's level only
means anything relative to the row above it.
chainNetLoadRangeQuickLift() maps each row's target range with
quickLiftIntervalsToBedClip, which is the same code every other quickLift track
uses, so a net row lands where a bed of the same span would. The surviving rows
then go to the same helpToNet() the unlifted path uses, so there is one tree
builder and not two. The row collection either path does is now cnlHelperNew and
cnlHelperAddBigNet.
validateOneTdb lets bigNet into a quickLift hub. Its chain track is not offered,
so a lifted net's details page has no chain to follow and says so rather than
looking for a track this assembly does not have. bigNetLoadOne lifts the same
way for that page, unclipped, so it reports the item's whole extent.
bigNetFromInterval passes -1 as the cached chromId. bbiCachedChromLookup leaves
the buffer alone when the id matches the one before it, so a cache that outlives
the buffer hands back stale bytes.
Measured against the standalone liftOver tool, on hg19 chr22's mouse net lifted
to hg38: 36 of 36 source rows in the sampled window land on the same coordinates,
counting the one liftOver will not take whole, whose two ends it does place
exactly where the browser puts them. Against hg38's own mouse net, computed
independently, 1.02% of the drawn pixels differ.
- lines changed 26, context: html, text, full: html, text
357d4dbeca6b3bbb59f60185f6b833d74fd74fbc Sun Sep 6 15:51:56 2026 -0700
bigNet: four fixes from the code review, refs #20824
A quickLifted net's details page lifted the row unclipped so it could report
the whole extent, but the image lifts clipped. An item too big for the chains
loaded in the window is drawn clipped and was then unfindable on click, which
puts a box on screen that says it is not there. Try the unclipped lift, fall
back to the clipped one, and say plainly when the numbers describe only the
part that could be placed.
quickLiftGetIntervals can return one source row twice, through two chains whose
padded query ranges overlap. helpToNet cannot tell two identical parents apart:
the second inherits no children and then draws as one solid box over the first
one's gaps. A level, a target range and a chain id name a row in a net, so that
is enough to recognize the repeat and drop it. Preventive -- no duplicate was
observed in the window measured.
The sentence explaining why a lifted net has no alignment to show printed
quickLiftDb twice, and a hub can set quickLiftUrl and leave quickLiftDb unset,
so it could be handed a null. One printf, and it reads correctly either way.
Free the per-row bed in both lift loops. It is about ninety thousand of them on
a whole chromosome, which is more than a CGI should be asked to shrug off.
Rendering is unchanged: the unlifted net still draws pixel for pixel like the
native netAlign track at three widths, every lifted figure but the details page
is pixel-identical to the one built before these fixes, and the 36 of 36
agreement with the standalone liftOver tool is unchanged.
- src/hg/lib/genark.c
- lines changed 8, context: html, text, full: html, text
bc4639b87acf68232656130b2972293975e11933 Thu Sep 10 12:43:45 2026 -0700
genark: tolerate a missing or stale genarkOrg table, refs #38327
genarkGetOrgHash() aborted when the central database had no genarkOrg
table. A mirror's hgcentral has never had one: buildHgCentralSql.csh did
not list the table, so hgcentral.sql on hgdownload carries neither its rows
nor its schema. Add an sqlTableExists check, and add genarkOrg to the list
of tables that hgcentral.sql replaces entirely.
genarkMakeDbDb() defaulted genome to "Other" for an accession with no
genarkOrg row, but left organism NULL. hgConvert prints organism, so the
Convert page read "Genome: (null)". Default both. The copy of genarkOrg
on the RR is 20,754 rows behind hgwdev, so this is visible on
genome.ucsc.edu today for 23 of the 855 GenArk assemblies that appear in
liftOverChain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 39, context: html, text, full: html, text
f219460db8db952415b5201ba01df99ce4999004 Thu Sep 10 13:46:42 2026 -0700
genark: pass the liftOver accession list as an slName list, refs #38328
genarkLiftOverDbs() took a pre-quoted SQL fragment that its callers
assembled. It now takes a struct slName list and builds the query
itself with sqlDyStringCreate, so no caller writes SQL text.
Accessions that do not start with GC are skipped, since nothing else
can match the table. hdb.c and hgConvert.c updated for the new
signature.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/lib/hCommon.c
- lines changed 14, context: html, text, full: html, text
58104a98358604975dac0a9350ca91d2d25c501d Thu Sep 10 05:02:45 2026 -0700
hUserAbort shows its message to the user instead of turning into a 500
hUserAbort() reports an error caused by user input, so the message is written
to be read by the user. It was only reaching them when a CGI had already
pushed a warn handler of its own. The apiKey and bot checks call it from
main() before that happens, and the default handler then writes to stderr and
nothing else, unless hg.conf sets showEarlyErrors - off by default, and off on
the RR. Apache turns the empty response into a 500, which is what
hubApi/hubApi.c works around by pre-validating the apiKey itself.
hVaUserAbort() now turns doContentType on for the rest of the process when it
is running as a CGI, so the default handler emits the Content-Type line and
the message. It stays off for a program that never called cgiSpoof(), and it
is inert inside an errCatch, which pushes its own warn handler - so a caller
that catches the abort to write its own response (hubApi's JSON) is unchanged.
Fixes the va_list handling in defaultVaWarn() while in there. It read args
three times but only the second and third read from a va_copy: the first
vfprintf consumed args itself, so the two reads after it saw a spent va_list
and the copy sent to the browser lost every %s and %d. It printed
"Bad thing: [br]" where the message was "Bad thing: %s<br>". Every read now
takes its own copy, and the buffer is filled with vsnprintf rather than
vsprintf.
No XSS: on this path defaultVaWarn replaces < and > with [ and ] across the
whole formatted message, args included. The other handlers that can report an
hUserAbort - earlyWarningHandler and cartEarlyWarningHandler via
htmlVaEncodeErrorText, htmlVaWarn, webVaWarn - all run the arguments through
vaHtmlDyStringPrintf, which html-encodes & < > / " and '. No caller passes
user data as the format string.
- src/hg/lib/hVarSubst.c
- lines changed 104, context: html, text, full: html, text
c0e8fa6df3a0bd406c4188d49ee00f20aef203e5 Mon Sep 7 12:07:18 2026 -0700
Substitute trackDb variables in hub track description pages
A hub's description page comes straight off the hub's web server and has
never been through variable substitution, so a $db or $parentTrack in it
reached the reader as literal text. Native trackDb pages are fine, since
hgTrackDb substitutes them when it loads trackDb, but there was no
equivalent step for a hub.
hgc's getTrackHtml and hgTrackUi's trackUi both call hVarSubstTrackDbHtml
on a hub track's html. Only a short list of variables is recognized there and nothing is an
error, because a hub page written before this existed can easily contain
a dollar sign inside a shell example, and silently rewriting that would
be worse than not substituting at all.
Adds $parentTrack, the name of the container a track sits in, which is
what a subtrack description page needs to link back to its superTrack or
composite. Views are skipped, since a view has no page of its own, and
the hub_<id>_ prefix is kept so the name works as hgTrackUi's g=
parameter. Documents $track, $parentTrack and $hgsid in trackDb/README.
refs #37599
- src/hg/lib/hdb.c
- lines changed 12, context: html, text, full: html, text
f219460db8db952415b5201ba01df99ce4999004 Thu Sep 10 13:46:42 2026 -0700
genark: pass the liftOver accession list as an slName list, refs #38328
genarkLiftOverDbs() took a pre-quoted SQL fragment that its callers
assembled. It now takes a struct slName list and builds the query
itself with sqlDyStringCreate, so no caller writes SQL text.
Accessions that do not start with GC are skipped, since nothing else
can match the table. hdb.c and hgConvert.c updated for the new
signature.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/lib/hgConfig.c
- lines changed 5, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/lib/hgHgvs.c
- lines changed 102, context: html, text, full: html, text
fa5b31d305066e1938953374d80b3782ef87e239 Mon Sep 7 23:23:40 2026 -0700
Position box: accept a bare codon number, and a range of codon numbers
"KAT6A p.495_533" used to land on codon 495 and silently drop the end of the
range, and a bare codon number after a transcript accession was not understood
at all, so "ENST00000265713.8 p.495" fell through the HGVS code and ended up on
an unrelated locus. Nucleotide ranges already worked. The pseudo-HGVS layer now
takes an optional _end on a bare codon number, and accepts a bare codon number
or range after an NM_ or ENST accession as well as after a gene symbol, looking
up the reference amino acids that HGVS wants and the user did not type.
The accession forms require a literal "p", so "NM_006766.5 1483" keeps meaning
what it meant. A hyphen is still not a range separator: c.1483-1599 is the HGVS
intronic position and stays that way.
Also fixes a read past the end of the protein sequence when the codon number
was larger than the protein, and documents codon ranges in query.html.
refs #38285
- lines changed 6, context: html, text, full: html, text
0f23d17640ca30e2c9ee456c7e15966cabd3bc57 Mon Sep 7 23:32:24 2026 -0700
Position box: let a hyphen separate a range of codons, e.g. "BRCA1 100-200"
A bare number after a gene symbol has always meant a codon, and "KAT6A 495-533"
was already accepted -- it just landed on codon 495 and dropped the rest, the
same silent truncation that the underscore form had. A hyphen now separates a
range wherever the coordinates are protein: after a gene symbol with no prefix,
and after an explicit p. with a symbol or a transcript accession.
The hyphen stays out of c. and n. terms, where HGVS already uses it for an
intron offset. KAT6A c.1483-1599 is still the single base 1599 nt before
c.1483, not codons 1483 to 1599, and there are now regression tests pinning
both readings so the two do not drift into each other.
refs #38285
- src/hg/lib/hui.c
- lines changed 4, context: html, text, full: html, text
64d40b9204a60274a49330c6b61d146e5cdc6d87 Wed Sep 9 05:54:34 2026 -0700
Faceted composite: a container's max display mode must clamp a pinned child, not drop it
Selecting only CpG methylation on the Fiber-seq compendium and hitting
submit drew nothing at all, with no message to say why.
Two things combined. The children of a faceted composite can be pinned
to one display mode with onlyVisibility, and five of the six Fiber-seq
data types are pinned to full because they are signal tracks. The
container's own "Maximum display mode" is a ceiling over those children,
and it was set to pack. tdbVisLimitedByAncestors() then took a pinned
child that sat above the ceiling and returned hide for it, so every
pinned-to-full track disappeared and only the peaks, pinned to dense,
came through. With every data type but peaks unchecked, that left an
empty image.
A ceiling should limit a child, not delete it, so use tvMin the same way
the unpinned case a line below already did. A bigWig draws the same at
pack as at full, minus the horizontal grid, so nothing is lost here.
The page was also asking for the wrong ceiling. On a faceted composite
the dropdown is a ceiling rather than a display mode, since each child
carries its own, so taking the container out of hide should ask for full,
the one value that clips nothing. Pack was chosen on the reasoning that
it suits a mix of signal and feature tracks, which is the right instinct
for a plain composite and the wrong one here.
refs #36210
- lines changed 1, context: html, text, full: html, text
ba5aa085f499f17ea3b88621e40a34c9c1ccb3d1 Thu Sep 10 11:03:15 2026 -0700
hui.c: turn the track color picker on by default, refs #20460
The showColorPicker gate has been in the tree since v496 and defaulted to
FALSE, so the color picker was only visible on hgwdev and genome-test. The
default is now TRUE. The flag stays in place, so a mirror or hgwbeta can
still set showColorPicker=off without a code change.
Also record the new default in the hg.conf catalog.
- src/hg/lib/makefile
- lines changed 1, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 7, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/lib/quickLift.c
- lines changed 168, context: html, text, full: html, text
d0ed51d077718213c0b843a89fd6f36ac32f1804 Fri Sep 4 11:39:28 2026 -0700
quickLift: lift alignments, and let psl and bigPsl tracks into the hub
Every type quickLift handled so far is one set of coordinates on one genome, so
the lift is a call to remapBlockedBed. An alignment carries coordinates on both
sides at once, with a block start on each, so moving the genome side means
splitting and trimming blocks while keeping the other side lined up with them.
quickLiftPsl does that with pieces that were already in the tree: pick the chain
with liftOverChainForRange, the same one the bed path picks, turn it into a
mapping alignment with chainToPsl plus pslSwap, then pslTransMap. The mapping
alignment is kept per chain, since rebuilding it for every item costs nothing at
gene zoom and a great deal zoomed out.
Two things pslTransMap does had to be undone. It recounts match and mismatch off
the blocks, which would read every lifted alignment as a perfect match and draw
every item at full shade, so quickLiftPslCounts puts the original counts back,
scaled by how much of the alignment survived. And it leaves a protein alignment
in nucleotide space, which the base alignment view rejects, so
quickLiftPslBackToProtein returns the query side to protein units when the lift
did not split a codon.
quickLiftChainHash hands out the chains for a reference range for callers that
collected their items some other way. validateOneTdb now accepts psl and bigPsl,
and bigPsl joins bigBed and bigWig in the bigDataUrl fill-in.
refs #38249
- lines changed 195, context: html, text, full: html, text
e70c7f68f61a3477631bd364c31f99c157528fc8 Fri Sep 4 12:48:02 2026 -0700
quickLift: lift chains, and let chain and bigChain tracks into the hub
A chain is an alignment between the assembly the track came from and some other
species, so lifting one composes two alignments and leaves the user with chains
between the assembly on screen and that species.
quickLiftChain does it by the same route the psl work uses: chainToPsl, then
quickLiftPsl, then chainFromPsl, which is the inverse chainToPsl never had. It
does not modify the chain handed to it. That matters more than it sounds: every
chain loader leaves the header describing the whole chain while loading only the
blocks in range, so the header has to be corrected for the conversion, but the
callers still want the original. The details page reports it, and the track
sorts on it, which is what puts a lifted chain in the same row as the native one.
quickLiftSourceRanges hands back the ranges in the other assembly that the window
maps to, for callers whose items cannot be had from a query quickLiftSql knows
how to make. It works those ranges out by intersecting the window with each
chain block rather than taking the whole block, which matters here because one
block can be enormous: hg19 and hg38 run identical for 12.8Mb on chr7, and
asking for all of it turned a 39 chain window into a 1892 chain one.
quickLiftIsOwnChainTrack keeps quickLift's own chain track out of all this. That
stanza carries quickLiftUrl and quickLiftDb like any lifted track and is loaded
by bigChainLoadItems like any bigChain, but its data is already in reference
coordinates. The giveaway is that its bigDataUrl is the quickLift chain file.
validateOneTdb now accepts chain and bigChain. netAlign is still refused: a net
is a hierarchy of gaps rather than a plain alignment and wants its own thought.
refs #38249
- lines changed 81, context: html, text, full: html, text
5fc426954da9ceb7dc42b9760858bbd1189760e1 Fri Sep 4 13:08:14 2026 -0700
quickLift: lift MAF blocks, and let bigMaf and wigMaf tracks into the hub
Most of this was already written. mafSubset does the part that looked hard,
which is recomputing every row's start and size when columns are taken away, so
what was left was deciding where to cut.
quickLiftMafs cuts a block at every chain block boundary. Inside one chain block
the two assemblies run in step, so the columns carry over untouched and only the
first row's coordinates change. Across a boundary the reference either loses
bases or gains them, and either way the block can no longer be one contiguous run
on the reference, which is the one thing a MAF block has to be.
The reference row is named with the assembly name minus any hub prefix, since
that is the name the maf drawing code builds when it goes looking for it.
A minus strand chain turns the block over, so every row is turned over with it and
the forward start comes from the far end of the run. That path is written but has
not been exercised: no minus strand quickLift chain wins in a window I could
find.
validateOneTdb accepts bigMaf and wigMaf. Plain maf is left out on purpose:
those tracks are drawn by mafTrack.c, which has no quickLift path, so offering
them would hand back a track read from the wrong assembly.
refs #38249
- lines changed 29, context: html, text, full: html, text
cec5ead054791f2a6601f308a56d280c08bd8ef7 Fri Sep 4 13:32:36 2026 -0700
quickLift: do not take the lift path on half a pair of settings
A hub can set quickLiftDb without setting quickLiftUrl, and nothing filters hub
trackDb settings. The alignment loaders gated on quickLiftDb alone, so such a
stanza took the lift path with no chain file and hgTracks died in
endsWith(NULL, ".bb") from bigChainGetLinkFile, by way of quickLiftLoadChains.
Verified: SIGSEGV in strlen from common.c:1653, page truncated mid-HTML. This is
reachable on a production browser now that bigChain and bigMaf are liftable,
because those carry their own bigDataUrl and so need no trustTrackDb.
quickLiftIsLifted requires both halves, and every gate now uses it, which also
settles the two different predicates that were in use for the same question.
quickLiftLoadChains returns an empty list for a NULL file as well, so the older
bed and genePred callers are covered whatever their gate does.
quickLiftSql now checks that a row has at least as many columns as the loader is
going to read. The native psl loader has always checked this, and the quickLift
path replacing it did not, so a table of the wrong type walked off the end of the
row; the psl caller now states the 21 columns it needs.
quickLiftMafs held a maf component name in a fixed buffer through safecpy, which
aborts rather than truncates, so a long name from a hub took hgTracks down. It
clones instead.
htcBigPslAliInWindow used a trackDb pointer its lookup can leave NULL, which its
sibling htcBigPslAli already checked for. And aliTrackParam formats a URL
parameter into a fixed buffer with safef, which aborts on a long one.
refs #38249
- lines changed 11, context: html, text, full: html, text
02710f0107a6b154d9e5689165941ce724cb4c6a Fri Sep 4 13:38:54 2026 -0700
quickLift: fix the seams a second review pass found
quickLiftPslBackToProtein left two things wrong. pslTransMap can hand back
strand[0] == '-', because it reverse complements the input when the two
alignments disagree about the strand of the sequence they share, and forcing
strand[1] to '+' on top of that produced "-+". A protein psl is only ever "++"
or "+-", and pslShow reads strand[0] == '-' as "reverse complement the query", so
it would have reverse complemented a protein as though it were DNA. It now turns
the alignment over so the minus lands on the target side. qBaseInsert is in
nucleotides like everything else being divided, so it comes down too, and it
joins the divisibility guard: without it the result failed pslCheck and the
number was printed verbatim on the details page.
Adding that back-conversion made a comment in pslTrack.c false. The lift no
longer always returns an untranslated alignment, so the drawing code has to ask
rather than assume, the way bigBedTrack.c already did. A quickLifted protein psl
track was drawing every block at a third of its length. No such track exists on
hg19 or hg38 today, so this was latent.
The normalized score on the chain details page was read from the assembly on
screen. Where that assembly has no such table the page died; where it has a
table of the same name, which is the common case for a self or a well known
chain track, it silently returned some other assembly's chain and printed a blank
score. It now reads the assembly the chain came from, on a connection to it.
Two smaller things: htcBigPslAli guarded its connection with trackHubDatabase
alone, but a GenArk accession does not start with hub_, so it matches the guard
genericClickHandlerPlus already uses; and the table name tests in cds.c now skip
the hub prefix the way the ones in hgc.c were changed to, so a lifted refSeqAli
reaches its special case.
The chain item label took its start from the source chain and its strand
character from the lifted one. Both now come from the source chain.
refs #38249
- lines changed 17, context: html, text, full: html, text
992aeef92fea7be25a2acd916578898649212a32 Mon Sep 7 11:31:07 2026 -0700
quickLift: gate the alignment lift behind an hg.conf flag, refs #38249
Add browser.quickLiftAlignments, default FALSE, so the alignment lift ships
dark and a machine turns it on with browser.quickLiftAlignments=on. It sits
beside browser.quickLift, the gate on the rest of the feature.
quickLiftAlignmentsEnabled() in hg/lib/quickLift.c is the one read, and
validateOneTdb in hg/lib/trackHub.c is the one place that asks it, before an
alignment track may enter a quickLift hub. That is the only door:
quickLiftUrl and quickLiftDb, the pair every lift path keys off, are written
by the quickLift hub writer and by nothing else, so with the flag off an
alignment track never gets them and the lifting, drawing and details code
behind them cannot be reached. pslTrack.c, chainTrack.c, wigMafTrack.c,
bigBedTrack.c and hgc.c are unchanged.
With the flag off hgConvert lists psl, bigPsl, chain, bigChain, maf, bigMaf
and wigMaf tracks in its "type is not supported by QuickLift" table, which is
what it did before this work. A hub built while the flag was on keeps working
after it is turned off, since its stanzas are already in the hub file in
trash, so this holds the feature back from people who have not used it rather
than switching off a session that has.
Read the hg.conf half with a literal cfgOptionBooleanDefault rather than
cartOrCfgOption so harvestHgConf.py can see it; a cart variable of the same
name still overrides it. Register the flag in hgConfCatalog.py with
role="gate" so the sunset report tracks it, and turn it on in
confs/hgwdev.hg.conf.
- src/hg/lib/sessionData.c
- lines changed 39, context: html, text, full: html, text
59e2bbf2d3a99e5f18436a756e1e5c8b64308e33 Tue Sep 8 09:21:02 2026 -0700
Copy a track collection's hub file when the program that writes it asks for a copy, instead of on every session load.
cartCopyLocalHubs ran on every session load, in every CGI and in four more places in
hgSession. For each customComposite-<db> cart variable it copied the collection's hub
file to a fresh trash name and registered the copy in hgcentral.hubStatus to get a hub
id. Nothing removed the old row. 81% of the 3.1 million rows in hubStatus on the RR are
these dead registrations, and each load also took the central_hubStatus advisory lock
and wrote to a MyISAM table, which serializes every concurrent load of a shared session
that carries a collection.
The copy itself is needed. A saved session's hub file lives under sessionDataDir and
every load of that session names it, so hgCollection must not write it in place. It was
just being made for every load, and almost no load is followed by an edit. hgCollection
is the only program that writes one of these files, so it now asks for its own copy:
main() calls cartRequestLocalHubCopy() before it opens the cart, and cartNew() makes the
copy for it.
That has to be a property of the program rather than of the request, and the copy has to
be made at cart open, above hubConnectLoadHubs. Copying the file gives the hub a new id,
the hubs are loaded during cart open, and trackList carries that id in every track name,
so a copy made any later leaves printTrackDbListToHub looking for the collection under
an id trackList does not have and the hub file comes out with a header and no tracks. A
condition that instead tries to work out whether this particular request will write
cannot be correct either: "cmd" is not in hgTracks' excludeVars, so it persists in the
cart and hgCollection dispatches on the cart rather than on the CGI variable, a settings
file can carry it, and a command-line run has no SCRIPT_NAME to test.
Three other parts. copyLocalHubs skips a hub the cart already owns, so repeated edits
reuse one file and one hub id rather than renumbering the hub on every drag; the test
requires a plain file, because saveTrackFile leaves the trash name behind as a symbolic
link to the durable copy and writing through that link would rewrite the saved session's
own file. copyLocalHubs also screens the path with isServerUserFilePath before opening
it, the way getHubName does, and selects on customComposite-<db> with the dash so that
the names it acts on are the ones fileNameCartVarPrefixes screens. saveTrackFile copies
when the source is a local hub outside trash that is not under this session's own
directory, so loading one session and saving it under a new name gives the new session
its own file instead of a reference into the first one's directory. pathIsUnderDir is no
longer static in trashDir.c.
All of it is behind the hg.conf gate collectionHubCopyOnWrite, default off, which
reproduces the old behavior exactly. Retiring the gate is not uniform: the body of
cartCopyLocalHubsOnSessionLoad is the old behavior and that function and its five
callers go away with the gate, while the other three tests lose only the gate term.
Measured on hgcentraltest with one binary. With the gate off, five loads of a session
carrying a collection add five hubStatus rows and three later edits add none; with it
on, the five loads add none and the first edit adds one. Saving a loaded session costs
two rows off and one on. The resulting hub file is byte identical either way apart from
the hub id, and the file the saved session names is unchanged by md5 and mtime,
including after an edit made straight after a save, and including when the cart names a
trash symbolic link into session storage.
refs #38273
- lines changed 7, context: html, text, full: html, text
133df533c4df3b1ebfcf3bbf14ab070f7d5c2ff2 Wed Sep 9 12:52:00 2026 -0700
sessionData: return kent-allocated memory from sessionDataSaveTrashFile
sessionDataSaveTrashFile() returned the result of realpath(path, NULL) when the
trash file it saves is already a relative symlink, which is what the trashCleaner
scripts leave behind. That pointer comes from the system malloc, but all three
callers release it with freeMem() or freez(), which dispatch through kent's own
handler stack. Under pushCarefulMemHandler() the free reads a block header that
was never written and subtracts a garbage size from the running total, so the
next allocation dies with "carefulAlloc: Allocated too much memory".
Nothing reaches it today. hgSession, hgPhyloPlace and snapshotSession are the
callers and none of them installs that handler, so the bug was latent. The fix
is a PATH_MAX stack buffer and a cloneString, the same shape already used by
pathIsUnderDirOrItsTarget() in trashDir.c. PATH_MAX needs an explicit include
of limits.h.
That same branch also dropped the readlink buffer on the floor, because it
replaced newPath with the resolved path instead of with the link target. It is
freed now.
Checked with a probe linking jkhgap under pushCarefulMemHandler: the patched
function resolves the symlink, frees, allocates again and passes
carefulCheckHeap, while the old pattern exits with a negative total.
refs #38318
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 17, context: html, text, full: html, text
11d5b26d9798079ec5adedc103216ce818e2da5e Thu Sep 10 05:19:27 2026 -0700
Widen the session data directory hash from 8 to 10 hex characters
sessionDirFromNames() named a session's durable data directory with 8 hex
characters of md5(sessionName). 32 bits was fine while a directory only had
to be unique among one user's sessions, but every anonymous "Share a link"
session belongs to the single reserved user "l", which makes it a birthday
problem across all of them: two unrelated sessions land in the same directory
more likely than not at around 77,000 anonymous sessions, and at 500,000 we
would expect about 29 such pairs. Two sessions sharing a directory means
cleaning up one takes the other's custom track files with it.
sessionDirHashLen is now 10 (40 bits), which moves the even-odds point past a
million sessions. The two fan-out levels snapshotSessionDir() added for user
"l" do not help here, since they are a prefix of the same hash: they spread
the entries over 65536 directories but leave the number of distinct leaf names
unchanged.
Both directory layouts change name as a result, so snapshotCleaner would have
walked past anything written earlier and orphaned its files. Both dir-naming
functions grew a hashLen argument, sessionDirHashLenLegacy records the old
value, and snapshotCleanAnon() now tries the old spelling of both layouts as
well as the new one.
Existing sessions keep working either way: the cart stores the absolute path
of each durable file, so nothing looks a session's directory up by name except
the cleaner.
refs #10138
- src/hg/lib/snapshotSession.c
- lines changed 6, context: html, text, full: html, text
5a249cd50f592a3b7598110eec792cfc942fab3f Wed Sep 9 08:52:06 2026 -0700
Address the v504 code review
hgSession: an anonymous share name that arrives with the request is saved only
when it is not already in the table. Every anonymous link sits under the one
reserved user "l", so a name already there stays as it is and the caller is told
so. The top-right Share dialog is unaffected, since it passes a name it has
just reserved and such a name does not exist yet. Snapshot names are now left
out of both My Sessions listings, which is what their "__" prefix has claimed
all along.
Share dialog: "Create link & copy" reports the copy instead of promising it.
copyToClipboard says whether the text reached the clipboard, the dialog passes
that on when a browser refuses, and it tries the asynchronous clipboard API
before giving up. The preview is built with the same encoding the server uses,
so a name holding a hyphen or a slash previews as the link that really gets
made. Cancelling out of the name editor no longer copies a second time, and a
reply with no link in it says so rather than showing "undefined".
hgBlat: a second click on the share button while the first request is still out
no longer mints a second snapshot session, and a box dismissed during the wait
stays closed.
Also: a snapshot moves a cart value into durable storage only when it is a
trash path, the way sessionData's own callers check; sqlAddressMatch keeps to
its own documented precondition when handed an empty address; alphaGenomeToWig
compares its output with its input rather than with itself, rejects a position
that is not all digits and skips an empty score; and hgc's default iframe width
reaches the browser as one percent sign.
refs #38294
- lines changed 34, context: html, text, full: html, text
11d5b26d9798079ec5adedc103216ce818e2da5e Thu Sep 10 05:19:27 2026 -0700
Widen the session data directory hash from 8 to 10 hex characters
sessionDirFromNames() named a session's durable data directory with 8 hex
characters of md5(sessionName). 32 bits was fine while a directory only had
to be unique among one user's sessions, but every anonymous "Share a link"
session belongs to the single reserved user "l", which makes it a birthday
problem across all of them: two unrelated sessions land in the same directory
more likely than not at around 77,000 anonymous sessions, and at 500,000 we
would expect about 29 such pairs. Two sessions sharing a directory means
cleaning up one takes the other's custom track files with it.
sessionDirHashLen is now 10 (40 bits), which moves the even-odds point past a
million sessions. The two fan-out levels snapshotSessionDir() added for user
"l" do not help here, since they are a prefix of the same hash: they spread
the entries over 65536 directories but leave the number of distinct leaf names
unchanged.
Both directory layouts change name as a result, so snapshotCleaner would have
walked past anything written earlier and orphaned its files. Both dir-naming
functions grew a hashLen argument, sessionDirHashLenLegacy records the old
value, and snapshotCleanAnon() now tries the old spelling of both layouts as
well as the new one.
Existing sessions keep working either way: the cart stores the absolute path
of each durable file, so nothing looks a session's directory up by name except
the cleaner.
refs #10138
- lines changed 46, context: html, text, full: html, text
b50a995e326f37cb2248e9a04628c15fb1b58215 Mon Sep 14 05:40:04 2026 -0700
My Sessions no longer hides a user's own "__" sessions, refs #38313
Both My Sessions listings decided a row was a share token by testing the
session name for the "__" prefix, so they also hid sessions a user had named
that way themselves - eleven of them on the RR, across four accounts, and
their owners could no longer rename, describe, reshare or delete them.
The prefix is a naming convention we follow, not a namespace we own. The
authoritative mark is the "snapshotType <type>" line saveSnapshotSession()
already writes into the settings column, so test that instead. Both queries
already select settings, so no query changes.
snapshotTypeFromSettings() walks the settings lines rather than calling
raFromString(): it runs once per listed row, and a hash there costs ~600ns and
three allocations for every session that has a description, against ~30ns and
none for the walk. Rows with empty settings, the common case, short-circuit
in both.
- src/hg/lib/tests/expected/hgvs/validTerms.txt
- lines changed 10, context: html, text, full: html, text
fa5b31d305066e1938953374d80b3782ef87e239 Mon Sep 7 23:23:40 2026 -0700
Position box: accept a bare codon number, and a range of codon numbers
"KAT6A p.495_533" used to land on codon 495 and silently drop the end of the
range, and a bare codon number after a transcript accession was not understood
at all, so "ENST00000265713.8 p.495" fell through the HGVS code and ended up on
an unrelated locus. Nucleotide ranges already worked. The pseudo-HGVS layer now
takes an optional _end on a bare codon number, and accepts a bare codon number
or range after an NM_ or ENST accession as well as after a gene symbol, looking
up the reference amino acids that HGVS wants and the user did not type.
The accession forms require a literal "p", so "NM_006766.5 1483" keeps meaning
what it meant. A hyphen is still not a range separator: c.1483-1599 is the HGVS
intronic position and stays that way.
Also fixes a read past the end of the protein sequence when the codon number
was larger than the protein, and documents codon ranges in query.html.
refs #38285
- lines changed 11, context: html, text, full: html, text
0f23d17640ca30e2c9ee456c7e15966cabd3bc57 Mon Sep 7 23:32:24 2026 -0700
Position box: let a hyphen separate a range of codons, e.g. "BRCA1 100-200"
A bare number after a gene symbol has always meant a codon, and "KAT6A 495-533"
was already accepted -- it just landed on codon 495 and dropped the rest, the
same silent truncation that the underscore form had. A hyphen now separates a
range wherever the coordinates are protein: after a gene symbol with no prefix,
and after an explicit p. with a symbol or a transcript accession.
The hyphen stays out of c. and n. terms, where HGVS already uses it for an
intron offset. KAT6A c.1483-1599 is still the single base 1599 nt before
c.1483, not codons 1483 to 1599, and there are now regression tests pinning
both readings so the two do not drift into each other.
refs #38285
- src/hg/lib/tests/input/hgvs/validTerms.txt
- lines changed 10, context: html, text, full: html, text
fa5b31d305066e1938953374d80b3782ef87e239 Mon Sep 7 23:23:40 2026 -0700
Position box: accept a bare codon number, and a range of codon numbers
"KAT6A p.495_533" used to land on codon 495 and silently drop the end of the
range, and a bare codon number after a transcript accession was not understood
at all, so "ENST00000265713.8 p.495" fell through the HGVS code and ended up on
an unrelated locus. Nucleotide ranges already worked. The pseudo-HGVS layer now
takes an optional _end on a bare codon number, and accepts a bare codon number
or range after an NM_ or ENST accession as well as after a gene symbol, looking
up the reference amino acids that HGVS wants and the user did not type.
The accession forms require a literal "p", so "NM_006766.5 1483" keeps meaning
what it meant. A hyphen is still not a range separator: c.1483-1599 is the HGVS
intronic position and stays that way.
Also fixes a read past the end of the protein sequence when the codon number
was larger than the protein, and documents codon ranges in query.html.
refs #38285
- lines changed 11, context: html, text, full: html, text
0f23d17640ca30e2c9ee456c7e15966cabd3bc57 Mon Sep 7 23:32:24 2026 -0700
Position box: let a hyphen separate a range of codons, e.g. "BRCA1 100-200"
A bare number after a gene symbol has always meant a codon, and "KAT6A 495-533"
was already accepted -- it just landed on codon 495 and dropped the rest, the
same silent truncation that the underscore form had. A hyphen now separates a
range wherever the coordinates are protein: after a gene symbol with no prefix,
and after an explicit p. with a symbol or a transcript accession.
The hyphen stays out of c. and n. terms, where HGVS already uses it for an
intron offset. KAT6A c.1483-1599 is still the single base 1599 nt before
c.1483, not codons 1483 to 1599, and there are now regression tests pinning
both readings so the two do not drift into each other.
refs #38285
- src/hg/lib/trackDbCustom.c
- lines changed 1, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/lib/trackHub.c
- lines changed 4, context: html, text, full: html, text
d0ed51d077718213c0b843a89fd6f36ac32f1804 Fri Sep 4 11:39:28 2026 -0700
quickLift: lift alignments, and let psl and bigPsl tracks into the hub
Every type quickLift handled so far is one set of coordinates on one genome, so
the lift is a call to remapBlockedBed. An alignment carries coordinates on both
sides at once, with a block start on each, so moving the genome side means
splitting and trimming blocks while keeping the other side lined up with them.
quickLiftPsl does that with pieces that were already in the tree: pick the chain
with liftOverChainForRange, the same one the bed path picks, turn it into a
mapping alignment with chainToPsl plus pslSwap, then pslTransMap. The mapping
alignment is kept per chain, since rebuilding it for every item costs nothing at
gene zoom and a great deal zoomed out.
Two things pslTransMap does had to be undone. It recounts match and mismatch off
the blocks, which would read every lifted alignment as a perfect match and draw
every item at full shade, so quickLiftPslCounts puts the original counts back,
scaled by how much of the alignment survived. And it leaves a protein alignment
in nucleotide space, which the base alignment view rejects, so
quickLiftPslBackToProtein returns the query side to protein units when the lift
did not split a codon.
quickLiftChainHash hands out the chains for a reference range for callers that
collected their items some other way. validateOneTdb now accepts psl and bigPsl,
and bigPsl joins bigBed and bigWig in the bigDataUrl fill-in.
refs #38249
- lines changed 4, context: html, text, full: html, text
e70c7f68f61a3477631bd364c31f99c157528fc8 Fri Sep 4 12:48:02 2026 -0700
quickLift: lift chains, and let chain and bigChain tracks into the hub
A chain is an alignment between the assembly the track came from and some other
species, so lifting one composes two alignments and leaves the user with chains
between the assembly on screen and that species.
quickLiftChain does it by the same route the psl work uses: chainToPsl, then
quickLiftPsl, then chainFromPsl, which is the inverse chainToPsl never had. It
does not modify the chain handed to it. That matters more than it sounds: every
chain loader leaves the header describing the whole chain while loading only the
blocks in range, so the header has to be corrected for the conversion, but the
callers still want the original. The details page reports it, and the track
sorts on it, which is what puts a lifted chain in the same row as the native one.
quickLiftSourceRanges hands back the ranges in the other assembly that the window
maps to, for callers whose items cannot be had from a query quickLiftSql knows
how to make. It works those ranges out by intersecting the window with each
chain block rather than taking the whole block, which matters here because one
block can be enormous: hg19 and hg38 run identical for 12.8Mb on chr7, and
asking for all of it turned a 39 chain window into a 1892 chain one.
quickLiftIsOwnChainTrack keeps quickLift's own chain track out of all this. That
stanza carries quickLiftUrl and quickLiftDb like any lifted track and is loaded
by bigChainLoadItems like any bigChain, but its data is already in reference
coordinates. The giveaway is that its bigDataUrl is the quickLift chain file.
validateOneTdb now accepts chain and bigChain. netAlign is still refused: a net
is a hierarchy of gaps rather than a plain alignment and wants its own thought.
refs #38249
- lines changed 3, context: html, text, full: html, text
5fc426954da9ceb7dc42b9760858bbd1189760e1 Fri Sep 4 13:08:14 2026 -0700
quickLift: lift MAF blocks, and let bigMaf and wigMaf tracks into the hub
Most of this was already written. mafSubset does the part that looked hard,
which is recomputing every row's start and size when columns are taken away, so
what was left was deciding where to cut.
quickLiftMafs cuts a block at every chain block boundary. Inside one chain block
the two assemblies run in step, so the columns carry over untouched and only the
first row's coordinates change. Across a boundary the reference either loses
bases or gains them, and either way the block can no longer be one contiguous run
on the reference, which is the one thing a MAF block has to be.
The reference row is named with the assembly name minus any hub prefix, since
that is the name the maf drawing code builds when it goes looking for it.
A minus strand chain turns the block over, so every row is turned over with it and
the forward start comes from the far end of the run. That path is written but has
not been exercised: no minus strand quickLift chain wins in a window I could
find.
validateOneTdb accepts bigMaf and wigMaf. Plain maf is left out on purpose:
those tracks are drawn by mafTrack.c, which has no quickLift path, so offering
them would hand back a track read from the wrong assembly.
refs #38249
- lines changed 3, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 2, context: html, text, full: html, text
f29b65452a1cca8a5bc62f007100b313e589d036 Sun Sep 6 15:11:36 2026 -0700
quickLift: lift a bigNet track, refs #20824
A net is not an ordinary track to lift. It is an alignment of two assemblies,
so only its target side moves to the new reference; the query side names a third
assembly and is carried across untouched. And the browser draws a net by
recursion, so what really has to survive the lift is the tree: a row's level only
means anything relative to the row above it.
chainNetLoadRangeQuickLift() maps each row's target range with
quickLiftIntervalsToBedClip, which is the same code every other quickLift track
uses, so a net row lands where a bed of the same span would. The surviving rows
then go to the same helpToNet() the unlifted path uses, so there is one tree
builder and not two. The row collection either path does is now cnlHelperNew and
cnlHelperAddBigNet.
validateOneTdb lets bigNet into a quickLift hub. Its chain track is not offered,
so a lifted net's details page has no chain to follow and says so rather than
looking for a track this assembly does not have. bigNetLoadOne lifts the same
way for that page, unclipped, so it reports the item's whole extent.
bigNetFromInterval passes -1 as the cached chromId. bbiCachedChromLookup leaves
the buffer alone when the id matches the one before it, so a cache that outlives
the buffer hands back stale bytes.
Measured against the standalone liftOver tool, on hg19 chr22's mouse net lifted
to hg38: 36 of 36 source rows in the sampled window land on the same coordinates,
counting the one liftOver will not take whole, whose two ends it does place
exactly where the browser puts them. Against hg38's own mouse net, computed
independently, 1.02% of the drawn pixels differ.
- lines changed 9, context: html, text, full: html, text
d11a21c48cdc2cebe0a97779b510df8317b980cd Mon Sep 7 06:29:43 2026 -0700
bigNet: gate the track type behind an hg.conf flag, refs #20824
Add the boolean hg.conf setting bigNet, default FALSE, so the type ships
dark and a machine turns it on with bigNet=on.
trackHubBigNetEnabled() in hg/lib/trackHub.c is the one read; the four
places that accept or advertise the type ask it. validateOneTrack drops
bigNet from the hub track type allowlist, validateOneTdb drops it from the
types quickLift will lift, hubCheck drops it from VALID_TRACK_TYPES and
from the message listing the valid types, and hubApi does not add it to
supportedTypes. netToBigNet, netTrack.c, chainNetDbLoad.c and the hgc
details code are unchanged, since they cannot be reached once a bigNet
track will not load.
With the flag off hgTracks does not abort the hub. It draws the track row
as a bigWarn bar reading "Unsupported type 'bigNet ...'" and the rest of
the hub loads normally.
Register the flag in hgConfCatalog.py with role="gate" so the sunset
report tracks it.
- lines changed 29, context: html, text, full: html, text
992aeef92fea7be25a2acd916578898649212a32 Mon Sep 7 11:31:07 2026 -0700
quickLift: gate the alignment lift behind an hg.conf flag, refs #38249
Add browser.quickLiftAlignments, default FALSE, so the alignment lift ships
dark and a machine turns it on with browser.quickLiftAlignments=on. It sits
beside browser.quickLift, the gate on the rest of the feature.
quickLiftAlignmentsEnabled() in hg/lib/quickLift.c is the one read, and
validateOneTdb in hg/lib/trackHub.c is the one place that asks it, before an
alignment track may enter a quickLift hub. That is the only door:
quickLiftUrl and quickLiftDb, the pair every lift path keys off, are written
by the quickLift hub writer and by nothing else, so with the flag off an
alignment track never gets them and the lifting, drawing and details code
behind them cannot be reached. pslTrack.c, chainTrack.c, wigMafTrack.c,
bigBedTrack.c and hgc.c are unchanged.
With the flag off hgConvert lists psl, bigPsl, chain, bigChain, maf, bigMaf
and wigMaf tracks in its "type is not supported by QuickLift" table, which is
what it did before this work. A hub built while the flag was on keeps working
after it is turned off, since its stanzas are already in the hub file in
trash, so this holds the feature back from people who have not used it rather
than switching off a session that has.
Read the hg.conf half with a literal cfgOptionBooleanDefault rather than
cartOrCfgOption so harvestHgConf.py can see it; a cart variable of the same
name still overrides it. Register the flag in hgConfCatalog.py with
role="gate" so the sunset report tracks it, and turn it on in
confs/hgwdev.hg.conf.
- lines changed 6, context: html, text, full: html, text
e9ed25747e3d8ab10963306b826f7cedc5e71897 Mon Sep 7 11:35:30 2026 -0700
Merge branch 'quickLiftAlign38249' -- alignment tracks in quickLift, refs #38249
# Conflicts:
# src/hg/lib/trackHub.c
- lines changed 6, context: html, text, full: html, text
bd4501775536afbc74d45e70960872fd6e61d09c Fri Sep 11 08:50:38 2026 -0700
hgTracks: a quickLifted container no longer hides the tracks inside it
trackDbString writes "superTrack on show" for a container it is lifting, then
dumpTdbAndChildren walks the settings hash and writes the source assembly's own
superTrack setting after it. That setting reads "on hide" whenever the user has
not opened the container, and the later line is the one the hub reader keeps.
The container came across hidden and so did every track in it, so a lift of a
track that lives in a container produced a page with nothing on it but the
chain track.
Drop the stale setting before the walk, the way walkTree already drops the copy
the children inherit.
Lifting JARVIS from hg19 to hg38 used to draw only the quickLift chain track.
It now draws all ten tracks in the Constraint scores container. A top level
bigWig, a composite of bigWigs and a bigBed all lift as they did before, and
the hub file now carries one superTrack line rather than two that disagree.
The crash this ticket was opened for is a separate thing and was already fixed
by 07ba37612b1 and 5dd36916814. Rechecked on genome-test over eight windows
down to 200 bases: no crash, and the lifted bigWig draws.
refs #37969
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/lib/trashDir.c
- lines changed 1, context: html, text, full: html, text
59e2bbf2d3a99e5f18436a756e1e5c8b64308e33 Tue Sep 8 09:21:02 2026 -0700
Copy a track collection's hub file when the program that writes it asks for a copy, instead of on every session load.
cartCopyLocalHubs ran on every session load, in every CGI and in four more places in
hgSession. For each customComposite-<db> cart variable it copied the collection's hub
file to a fresh trash name and registered the copy in hgcentral.hubStatus to get a hub
id. Nothing removed the old row. 81% of the 3.1 million rows in hubStatus on the RR are
these dead registrations, and each load also took the central_hubStatus advisory lock
and wrote to a MyISAM table, which serializes every concurrent load of a shared session
that carries a collection.
The copy itself is needed. A saved session's hub file lives under sessionDataDir and
every load of that session names it, so hgCollection must not write it in place. It was
just being made for every load, and almost no load is followed by an edit. hgCollection
is the only program that writes one of these files, so it now asks for its own copy:
main() calls cartRequestLocalHubCopy() before it opens the cart, and cartNew() makes the
copy for it.
That has to be a property of the program rather than of the request, and the copy has to
be made at cart open, above hubConnectLoadHubs. Copying the file gives the hub a new id,
the hubs are loaded during cart open, and trackList carries that id in every track name,
so a copy made any later leaves printTrackDbListToHub looking for the collection under
an id trackList does not have and the hub file comes out with a header and no tracks. A
condition that instead tries to work out whether this particular request will write
cannot be correct either: "cmd" is not in hgTracks' excludeVars, so it persists in the
cart and hgCollection dispatches on the cart rather than on the CGI variable, a settings
file can carry it, and a command-line run has no SCRIPT_NAME to test.
Three other parts. copyLocalHubs skips a hub the cart already owns, so repeated edits
reuse one file and one hub id rather than renumbering the hub on every drag; the test
requires a plain file, because saveTrackFile leaves the trash name behind as a symbolic
link to the durable copy and writing through that link would rewrite the saved session's
own file. copyLocalHubs also screens the path with isServerUserFilePath before opening
it, the way getHubName does, and selects on customComposite-<db> with the dash so that
the names it acts on are the ones fileNameCartVarPrefixes screens. saveTrackFile copies
when the source is a local hub outside trash that is not under this session's own
directory, so loading one session and saving it under a new name gives the new session
its own file instead of a reference into the first one's directory. pathIsUnderDir is no
longer static in trashDir.c.
All of it is behind the hg.conf gate collectionHubCopyOnWrite, default off, which
reproduces the old behavior exactly. Retiring the gate is not uniform: the body of
cartCopyLocalHubsOnSessionLoad is the old behavior and that function and its five
callers go away with the gate, while the other three tests lose only the gate term.
Measured on hgcentraltest with one binary. With the gate off, five loads of a session
carrying a collection add five hubStatus rows and three later edits add none; with it
on, the five loads add none and the first edit adds one. Saving a loaded session costs
two rows off and one on. The resulting hub file is byte identical either way apart from
the hub id, and the file the saved session names is unchanged by md5 and mtime,
including after an edit made straight after a save, and including when the cart names a
trash symbolic link into session storage.
refs #38273
- lines changed 22, context: html, text, full: html, text
67700f0c574a5b49df7e49eff54e16221ff7e6ea Tue Sep 8 15:38:12 2026 -0700
cart: accept a session file path spelled through a symlinked config directory
The file-name check added in 9ad04e0a0b0 compares a path against trashDir(),
sessionDataDir, sessionDataDirOld and myVariantsDataDir as plain strings. Those
directories are often reached through a symlink, and sessionData.c stores the
resolved spelling whenever the trash file it is saving is already a relative
symlink, so a saved session can hold either spelling. On the RR /userdata is a
symlink to /shared/userdata and 583 saved multi-region sessions hold the
resolved form, so hgTracks dropped multiRegionsBedUrl and said "No BED or BED
URL specified" rather than drawing the saved view.
Accept a path under a trusted directory or under whatever that directory
resolves to. Only the configured directory is resolved. Resolving the value
itself is not an option, because a trash file is deliberately a symlink into
session storage.
Use a stack buffer rather than realpath(dir, NULL): that allocates with the
system malloc, and freeMem() goes through the kent handler stack, which under
pushCarefulMemHandler() reads a block header that is not there. hgc, hgTables,
hgVai, hgLogin and hgLinkIn all install that handler before loading a cart.
refs #38303, refs #37623
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 14, context: html, text, full: html, text
71475dc7c4a4a603ed7334eb4d27fbda8abdbdec Tue Sep 8 17:54:23 2026 -0700
cart: leave the trash directory out of the symlink-resolving path check
Code review of 67700f0c574 found that resolving trashDir() does more than widen
a check. trashDir() is the relative "../trash", so its resolved spelling is a
different string, and sessionDataPathFromTrash() substitutes exactly the
relative spelling. A path accepted as trash in its resolved spelling therefore
rewrites to itself, and the callers treat the two as two files:
saveTrackFile() opens the old one for reading and truncates the new one for
writing, then unlinks the old name and points it at itself, and
sessionDataSaveTrashFile() unlinks the file and then aborts when link() fails.
Since ctfile_<db>, customComposite-<db> and hubQuickLift-<db> reach
saveTrackFile() from a cart value, a session save could have destroyed another
session's file.
Resolving a relative directory also resolves it against the working directory of
the process, so the answer moved with the caller: on the command line trashDir()
is $JKTRASH or ".", which made anything under $TMPDIR or under cgi-bin count as
trash.
Put isTrashPath() back on the plain string compare and keep the resolving
version for sessionDataDir, sessionDataDirOld and myVariantsDataDir, which are
the configured absolute directories the #38303 sessions are stored under. Only
resolve an absolute directory, so the result no longer depends on the working
directory. All 297,487 saved sessions in the September dumps of the RR, hgwbeta
and hgwdev centrals were checked: none holds a resolved trash path, so nothing
loses the widening.
refs #38303, refs #38304, refs #37623
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/lib/web.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/lib/wikiLink.c
- lines changed 10, context: html, text, full: html, text
543c9ee045ba1832faaa9b75c6dc1e369dffce5a Thu Sep 10 05:21:42 2026 -0700
Login and Sign out come back to a page that was reached by POST, refs #38192
Clicking a track name in the list below the browser image submits the
hgTracks form to hgTrackUi, so the request is a POST even though the
track name sits in the URL. The return URL builder threw the query
string away for anything that was not a GET, which left a returnto of
hgTrackUi?hgsid= alone, and hgTrackUi cannot draw a page from that
because the track name is deliberately not kept in the cart. Login and
Sign out therefore ended in an error instead of coming back.
The query string of a POST lives in the form's action URL, which is the
address the browser is showing, so returning to it is no different from
the visitor pressing reload. Only the form body is left behind, and the
cart already holds what mattered from it. hgTracks stays the exception:
its query string can hold a one-shot zoom or drag.
Also, hgTrackUi now says which parameter is missing when it is reached
without a track name, rather than failing on a bare hash lookup, and
hgCollection's own "you must be logged in" link brings the visitor back
to hgCollection instead of the sessions page.
- src/hg/liftAgp/makefile
- lines changed 2, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/makeDb/doc/contrib/bTaeGut7/bTaeGut7.txt
- lines changed 212, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/hubHeader.txt
- lines changed 7, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/makeHtml.py
- lines changed 886, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/makeHub.py
- lines changed 156, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/mkContribLinks.sh
- lines changed 67, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/21097885.html
- lines changed 11, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/26667931.html
- lines changed 12, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/29109402.html
- lines changed 11, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/31842948.html
- lines changed 13, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/31843001.html
- lines changed 13, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/35279659.html
- lines changed 10, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/38709825.html
- lines changed 12, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/39240653.html
- lines changed 11, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/41262969.html
- lines changed 11, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/42561917.html
- lines changed 11, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/refs/9862982.html
- lines changed 11, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/sourceTracks.tsv
- lines changed 24, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/bTaeGut7/splitHap.report.txt
- lines changed 54, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/doc/contrib/hprc2annot/hprc2annot.txt
- lines changed 138, context: html, text, full: html, text
1682366b1827b7559f8e1e41635acff6c5ea15e9 Wed Sep 9 06:05:05 2026 -0700
hprc2annot: move the makeDoc into its own directory and repoint the links
The makeDoc has grown a companion (an hg38 pcLAI doc is in progress), so it
moves from doc/contrib/hprc2annot.txt into doc/contrib/hprc2annot/, matching
how the scripts and trackDb copies are already laid out. The file itself gains
a section on the pcLAI scatterplot on the details page: where the reference
panel comes from, the four ancestry centroids the discretized field takes
across the release, and why the file is read through hgTrackUi rather than
fetched by the browser.
All seven track description pages linked to the old flat path and would have
404'd, so they are repointed. Six of them change only that link; pclai.html has
further edits still in progress and keeps its own copy of the change.
refs #35415
- src/hg/makeDb/doc/fishAsmHub/fish.orderList.tsv
- lines changed 3, context: html, text, full: html, text
3cb4d150856825931e385cff4d4d19173ec5a723 Wed Sep 9 16:39:37 2026 -0700
adding a couple per user request refs #29545
- src/hg/makeDb/doc/hg38/alphaGenome.txt
- lines changed 10, context: html, text, full: html, text
3dd0329ee0e804f5841a9998c4e7f024d3276921 Wed Sep 9 17:20:20 2026 -0700
Releasing the AlphaGenome Variant Impact (AVI) score track for hg38. refs #38261
Moves the bigWigs to /gbdb/hg38/_alphaGenome/, the underscore convention that keeps
non-redistributable data off the download server, matching PromoterAI and PrimateAI,
and drops the alpha release tag.
Shortens the composite longLabel to fit the 85 character limit and lowercases Score
in the longLabels. Adds a New pennantIcon and points the Deleteriousness Predictions
container pennant at the Sept. 10 news post.
Corrects the median on the description page from 1.6 to 2.9. The old figure counted
the zero-filled reference base slots, which are not variants. Removes two threshold
statements that are not in the AlphaGenome Atlas preprint or any public source. Adds
the /gbdb symlink step to the makeDoc, and fixes three container page links that were
missing target=_blank.
- src/hg/makeDb/doc/hg38/fiberSeq.txt
- lines changed 258, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- lines changed 42, context: html, text, full: html, text
01bc05ac9a282a6862111502f13601e513d5b60b Tue Sep 8 06:16:04 2026 -0700
hg38 Fiber-seq: merge the methylation composite into the compendium
The accessibility compendium and the separate Methylation composite covered the
identical 41 samples, and cartDump.c assigns priority with the data element as
the outer loop and the data type as the inner one. So one composite keeps a
sample's six subtracks contiguous in the image, where two composites drew an
accessibility block followed by a methylation block and comparing the two assays
for one sample meant reading past every other sample. Both come off the same
molecules in the same experiment, so side by side is the point. fiberSeqMeth is
gone and its three data types moved in as cpg, cpgHap and cpgDiff, renamed
because "hap" was already taken by the accessibility overlay and a data type
name cannot contain an underscore.
Subtracks now carry an explicit priority, sample outer and declared data type
inner. Without one they fell back to a label sort, so a first visit showed a
sample's data types as Peaks, CpG, Acc rather than in the order of the checkbox
row above the table.
Metadata columns renamed from camelCase to Accession, Sample_class, _Cell_type
and _Sample. toTitleStyle() in facetedComposite.js renders an underscore as a
space but does not split camelCase, so "sampleClass" appeared verbatim as a
column heading. A literal space cannot be used instead: the saved sort order is
a space-separated list of column names and the submit code drops any name
containing whitespace, which would have made sorting silently fail to persist.
Cell type is no longer faceted. A facet value is only offered when it occurs
more than once, and 12 of the 14 cell types here are a single sample, so as a
facet it drew two checkboxes and left 12 samples unreachable by any cell-type
filter. It is a searchable column now, and Sample_class is the only facet until
the lab gives us real HPRC metadata that would facet properly.
Description page intro rewritten, and it now says the assay measures the same
property as DNase-seq and ATAC-seq.
refs #36210
- lines changed 75, context: html, text, full: html, text
d07356d76401059fbe6c2d0890b1490546a62a4e Mon Sep 14 02:54:50 2026 -0700
hg38 Fiber-seq: reissued GM12878 data and a nucleosome density track, refs #36210
The lab reprocessed GM12878 (PM00001) and replaced the files in place under the
same hash directory. Ten of its twelve files changed; a size sweep over all 41
samples confirmed no other sample is affected. This fixes the two placeholder
haplotype accessibility bigWigs that covered a single base, so that overlay now
draws real data for the sample that comes up by default. Its peak calls changed
substantially as well, 429,883 source peaks before and 196,742 now, which is
noted on the description page since figures made from the first version of the
track will not reproduce for GM12878.
The downloader now fetches into <file>.part and moves it into place when
complete. It used curl -C - straight onto the final file, which is right for an
interrupted transfer and silently corrupting when the server has replaced the
file: it would have appended the tail of the new 5.2 GB hap1 file to the
512-byte stub, and the size check afterwards would have passed. It also takes
an optional list of accessions now, to refresh one sample without walking all 41.
Nucleosome density (all.nucleosome.coverage.bw) was sent separately and is not
in the lab's own hub. It is on the server for all 41 samples and is added as a
seventh data type in the compendium. Unlike every other wiggle here it is a
read depth rather than a percentage, so it cannot take fixed viewLimits: the
genome-wide mean runs from 25 to 142 across samples with sequencing depth and
single loci reach 1.7e5. It is drawn with autoScale, which the description page
explains, and reads as the complement of the accessibility signal.
- lines changed 22, context: html, text, full: html, text
75e828960283291546d2c1a27845e2cf3823adcd Mon Sep 14 05:29:02 2026 -0700
uniprot otto: the miniprot cluster job needs absolute paths
GRCz12ab failed with the parasol job crashing four times, return 1, no output.
The wrapper I wrote ran
miniprot -t 16 --gff protToGenome/GRCz12ab/.../genome.fa fasta/7955.fa > $1
and a parasol job runs with its working directory set to the batch directory, not
to the directory the pipeline runs in, so neither input existed from the job's
point of view. The BLAST batch next door gets away with relative paths because it
cds into its own workdir and its jobList is written relative to that; this batch
directory sits a level deeper and its paths were relative to the otto root.
Every path in the wrapper, the jobList command and the output check is now
absolute.
Verified on the cluster against the real 1.48 Gb zebrafish genome: successful
batch, a 195 MB GFF with 93518 mRNA records.
refs #38300
- src/hg/makeDb/doc/hg38/hprcPclai.txt
- lines changed 137, context: html, text, full: html, text
2c0adaa48b2f9a14109c3f90713405f259d920bc Wed Sep 9 06:08:46 2026 -0700
hg38: pcLAI local ancestry track for HPRC Release 2 haplotypes
Point cloud local ancestry inference (pcLAI) for HPRC Release 2, projected
onto GRCh38: a composite with one subtrack per haplotype, both haplotypes of
231 samples plus CHM13, 463 in all. 11,936,603 windows, autosomes only, no
windows dropped from the source files.
Each window carries the (PC1,PC2) coordinate pcLAI predicts for it, the
discretized ancestry centroid, and a confidence score. The details page draws
the window's position against the 1000 Genomes reference panel that defines
the space, via detailsScript/scatterPlot; metaDataUrl is what lets hgTrackUi
serve that panel file for a native (non-hub) track.
thickStart is one base before chromStart in 54,811 of the windows (0.46%),
against the format the pcLAI authors document, so bedToBigBed rejects it.
Neither thick column carries information here, so both are set to the item
bounds rather than dropping those windows.
Testing only for now, alpha, no ticket yet; a ticket may follow if this
becomes a real track. The makeDoc carries the detail in the meantime,
including what was deliberately left undone.
- src/hg/makeDb/doc/hg38/hprcRdt.txt
- lines changed 73, context: html, text, full: html, text
ef779a5a2ed508cb00f0b0139a12d696e439ee5f Fri Sep 11 06:06:55 2026 -0700
new hg38 track hprcRdt: reference-divergent transcripts from 206 HPRC Release 2 genomes
Added as a third child of the existing long_read_transcripts superTrack,
alpha only for now. Data from Max Marin (DFCI), a bigPsl of RDT cluster
representative sequences aligned to GRCh38: 180,464 alignments of 120,451
distinct sequences from 412 haplotypes. Rebuilt from the submitted file only
to add a name index, refs #33822
- src/hg/makeDb/doc/hg38/imprinting.txt
- lines changed 5, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/doc/hg38/kaplanImprint.txt
- lines changed 176, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- lines changed 6, context: html, text, full: html, text
d033cea2063e9362949baf5b4d8b837597173a0d Thu Sep 10 05:16:05 2026 -0700
Address the code review of the Sep 9 commits
Faceted composite: text that comes from a hub - a metadata column's
description, the name and title of a data type, and the values quoted back in
the "could not load the metadata" row - is put on the page as text rather than
as markup. The three places built their markup from template strings, so a
value carrying angle brackets or a quote was read as HTML: the column
description now goes through the shared htmlEncode() once where the header is
parsed, and the other two build their elements as nodes. The error row reads
better for it as well, since a value with brackets in it used to disappear from
the message that was meant to show it.
The saved UI state keys on the assembly as well as the metadata id. localStorage
is per-origin, so two assemblies whose tracks share a name were sharing one
entry, and a row order dragged on one came back on the other over a different
set of samples. hgTrackUi passes the database down for it. State saved under the
old key is dropped, which costs a facet selection or a page length.
Imprinting: the five subtrack description pages link back to the container as
hgTrackUi?db=$db&g=$parentTrack, without the hgsid. Native trackDb html is
substituted by hgTrackDb as it loads the table, where there is no cart, so
${hgsid} came out empty and the link read 'hgsid=&g=...'. Matches what the
Fiber-seq pages already do. The makeDoc note that described the old form is
updated with the reason.
UniProt otto: README.txt lists all eight things that reach runLog.txt. It had
four, and was missing LOCKED, along with PREFLIGHT-FAIL, END and INTERRUPTED.
refs #36210
refs #37599
refs #38300
- src/hg/makeDb/doc/mm10/mouseStrainsCactus.txt
- lines changed 79, context: html, text, full: html, text
efebc8a0a29aeef60bc470a40ced7a2aa6652efd Tue Sep 8 19:25:02 2026 -0700
Adding native mm10 track for the mouse strains Cactus alignment. refs #38308
New alpha-gated track mouseStrainsCactus exposing the Progressive Cactus
alignment of the 16 Mouse Genomes Project strain assemblies plus rat, which
until now was only reachable by attaching the mouseStrains assembly hub.
bigDataUrl, summary and frames point at the existing bigMaf files on
hgdownload rather than copying 8.8 GB into /gbdb, the same way the hg38
cactus241wayBM track is served.
Polish over the hub stanza: renamed from the generic "bigMaf", off by
default, speciesGroups splitting the strains into wild-derived, classical
laboratory and Rat/rn6, speciesLabels so side labels read 129S1/SvImJ
rather than 129S1_SvImJ, plus treeImage and speciesCodonDefault. The three
new sGroup_ tags are registered in tagTypes.tab.
Description page written from Lilue et al. 2018; the hub page had an empty
Description section and its Display Convention text was wigMaf boilerplate
that did not match this track. Also notes that the alignment is a poor
source for large rearrangements, since Ragout built the strain
pseudo-chromosomes against the reference and discarded most adjacencies
that disagreed with it.
Added a reciprocal relatedTracks.ra pair between this track and
mm10Strains1 ("Alternate strains"), since #38227 came in from a user who
kept landing on mm10Strains1 while looking for this alignment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/makeDb/doc/plantsAsmHub/plants.orderList.tsv
- lines changed 1, context: html, text, full: html, text
3cb4d150856825931e385cff4d4d19173ec5a723 Wed Sep 9 16:39:37 2026 -0700
adding a couple per user request refs #29545
- src/hg/makeDb/doc/vertebrateAsmHub/vertebrate.orderList.tsv
- lines changed 2, context: html, text, full: html, text
3cb4d150856825931e385cff4d4d19173ec5a723 Wed Sep 9 16:39:37 2026 -0700
adding a couple per user request refs #29545
- src/hg/makeDb/genbank/common.mk
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/makeDb/hgGoldGapGl/makefile
- lines changed 1, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/makeDb/hgTomRough/makefile
- lines changed 2, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/makeDb/scripts/fiberSeq/fiberSeqCheck.sh
- lines changed 72, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- lines changed 1, context: html, text, full: html, text
d07356d76401059fbe6c2d0890b1490546a62a4e Mon Sep 14 02:54:50 2026 -0700
hg38 Fiber-seq: reissued GM12878 data and a nucleosome density track, refs #36210
The lab reprocessed GM12878 (PM00001) and replaced the files in place under the
same hash directory. Ten of its twelve files changed; a size sweep over all 41
samples confirmed no other sample is affected. This fixes the two placeholder
haplotype accessibility bigWigs that covered a single base, so that overlay now
draws real data for the sample that comes up by default. Its peak calls changed
substantially as well, 429,883 source peaks before and 196,742 now, which is
noted on the description page since figures made from the first version of the
track will not reproduce for GM12878.
The downloader now fetches into <file>.part and moves it into place when
complete. It used curl -C - straight onto the final file, which is right for an
interrupted transfer and silently corrupting when the server has replaced the
file: it would have appended the tail of the new 5.2 GB hap1 file to the
512-byte stub, and the size check afterwards would have passed. It also takes
an optional list of accessions now, to refresh one sample without walking all 41.
Nucleosome density (all.nucleosome.coverage.bw) was sent separately and is not
in the lab's own hub. It is on the server for all 41 samples and is added as a
seventh data type in the compendium. Unlike every other wiggle here it is a
read depth rather than a percentage, so it cannot take fixed viewLimits: the
genome-wide mean runs from 25 to 142 across samples with sequencing depth and
single loci reach 1.7e5. It is drawn with autoScale, which the description page
explains, and reads as the complement of the accessibility signal.
- src/hg/makeDb/scripts/fiberSeq/fiberSeqDownload.sh
- lines changed 90, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- lines changed 30, context: html, text, full: html, text
d07356d76401059fbe6c2d0890b1490546a62a4e Mon Sep 14 02:54:50 2026 -0700
hg38 Fiber-seq: reissued GM12878 data and a nucleosome density track, refs #36210
The lab reprocessed GM12878 (PM00001) and replaced the files in place under the
same hash directory. Ten of its twelve files changed; a size sweep over all 41
samples confirmed no other sample is affected. This fixes the two placeholder
haplotype accessibility bigWigs that covered a single base, so that overlay now
draws real data for the sample that comes up by default. Its peak calls changed
substantially as well, 429,883 source peaks before and 196,742 now, which is
noted on the description page since figures made from the first version of the
track will not reproduce for GM12878.
The downloader now fetches into <file>.part and moves it into place when
complete. It used curl -C - straight onto the final file, which is right for an
interrupted transfer and silently corrupting when the server has replaced the
file: it would have appended the tail of the new 5.2 GB hap1 file to the
512-byte stub, and the size check afterwards would have passed. It also takes
an optional list of accessions now, to refresh one sample without walking all 41.
Nucleosome density (all.nucleosome.coverage.bw) was sent separately and is not
in the lab's own hub. It is on the server for all 41 samples and is added as a
seventh data type in the compendium. Unlike every other wiggle here it is a
read depth rather than a percentage, so it cannot take fixed viewLimits: the
genome-wide mean runs from 25 to 142 across samples with sequencing depth and
single loci reach 1.7e5. It is drawn with autoScale, which the description page
explains, and reads as the complement of the accessibility signal.
- src/hg/makeDb/scripts/fiberSeq/fiberSeqFixPeaks.sh
- lines changed 108, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- src/hg/makeDb/scripts/fiberSeq/fiberSeqSamples.tsv
- lines changed 42, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- src/hg/makeDb/scripts/fiberSeq/fiberSeqTrackDb.py
- lines changed 404, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- lines changed 197, context: html, text, full: html, text
01bc05ac9a282a6862111502f13601e513d5b60b Tue Sep 8 06:16:04 2026 -0700
hg38 Fiber-seq: merge the methylation composite into the compendium
The accessibility compendium and the separate Methylation composite covered the
identical 41 samples, and cartDump.c assigns priority with the data element as
the outer loop and the data type as the inner one. So one composite keeps a
sample's six subtracks contiguous in the image, where two composites drew an
accessibility block followed by a methylation block and comparing the two assays
for one sample meant reading past every other sample. Both come off the same
molecules in the same experiment, so side by side is the point. fiberSeqMeth is
gone and its three data types moved in as cpg, cpgHap and cpgDiff, renamed
because "hap" was already taken by the accessibility overlay and a data type
name cannot contain an underscore.
Subtracks now carry an explicit priority, sample outer and declared data type
inner. Without one they fell back to a label sort, so a first visit showed a
sample's data types as Peaks, CpG, Acc rather than in the order of the checkbox
row above the table.
Metadata columns renamed from camelCase to Accession, Sample_class, _Cell_type
and _Sample. toTitleStyle() in facetedComposite.js renders an underscore as a
space but does not split camelCase, so "sampleClass" appeared verbatim as a
column heading. A literal space cannot be used instead: the saved sort order is
a space-separated list of column names and the submit code drops any name
containing whitespace, which would have made sorting silently fail to persist.
Cell type is no longer faceted. A facet value is only offered when it occurs
more than once, and 12 of the 14 cell types here are a single sample, so as a
facet it drew two checkboxes and left 12 samples unreachable by any cell-type
filter. It is a searchable column now, and Sample_class is the only facet until
the lab gives us real HPRC metadata that would facet properly.
Description page intro rewritten, and it now says the assay measures the same
property as DNase-seq and ATAC-seq.
refs #36210
- lines changed 61, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- lines changed 33, context: html, text, full: html, text
d07356d76401059fbe6c2d0890b1490546a62a4e Mon Sep 14 02:54:50 2026 -0700
hg38 Fiber-seq: reissued GM12878 data and a nucleosome density track, refs #36210
The lab reprocessed GM12878 (PM00001) and replaced the files in place under the
same hash directory. Ten of its twelve files changed; a size sweep over all 41
samples confirmed no other sample is affected. This fixes the two placeholder
haplotype accessibility bigWigs that covered a single base, so that overlay now
draws real data for the sample that comes up by default. Its peak calls changed
substantially as well, 429,883 source peaks before and 196,742 now, which is
noted on the description page since figures made from the first version of the
track will not reproduce for GM12878.
The downloader now fetches into <file>.part and moves it into place when
complete. It used curl -C - straight onto the final file, which is right for an
interrupted transfer and silently corrupting when the server has replaced the
file: it would have appended the tail of the new 5.2 GB hap1 file to the
512-byte stub, and the size check afterwards would have passed. It also takes
an optional list of accessions now, to refresh one sample without walking all 41.
Nucleosome density (all.nucleosome.coverage.bw) was sent separately and is not
in the lab's own hub. It is on the server for all 41 samples and is added as a
seventh data type in the compendium. Unlike every other wiggle here it is a
read depth rather than a percentage, so it cannot take fixed viewLimits: the
genome-wide mean runs from 25 to 142 across samples with sequencing depth and
single loci reach 1.7e5. It is drawn with autoScale, which the description page
explains, and reads as the complement of the accessibility signal.
- lines changed 48, context: html, text, full: html, text
75e828960283291546d2c1a27845e2cf3823adcd Mon Sep 14 05:29:02 2026 -0700
uniprot otto: the miniprot cluster job needs absolute paths
GRCz12ab failed with the parasol job crashing four times, return 1, no output.
The wrapper I wrote ran
miniprot -t 16 --gff protToGenome/GRCz12ab/.../genome.fa fasta/7955.fa > $1
and a parasol job runs with its working directory set to the batch directory, not
to the directory the pipeline runs in, so neither input existed from the job's
point of view. The BLAST batch next door gets away with relative paths because it
cds into its own workdir and its jobList is written relative to that; this batch
directory sits a level deeper and its paths were relative to the otto root.
Every path in the wrapper, the jobList command and the output check is now
absolute.
Verified on the cluster against the real 1.48 Gb zebrafish genome: successful
batch, a 195 MB GFF with 93518 mRNA records.
refs #38300
- src/hg/makeDb/scripts/hprc2annot/hprc2annotBuildOne.sh
- lines changed 4, context: html, text, full: html, text
6b0035d19769346baffe193ef9419c269d46f8d8 Wed Sep 9 06:09:41 2026 -0700
hprc2annot: pcLAI column 10 is the ancestry centroid, not a segment coordinate
Reading the pcLAI authors' own format description
(github.com/AI-sandbox/hprc-pclai, README "Output format (BED)") while building
the same annotation as a native hg38 track showed that column 10 of the source
BED had been described wrongly here. It is not the PCA coordinate of a longer
ancestry segment the window belongs to; the authors call it the centroid, the
discretized pcLAI ancestry of the window written as the PCA centroid of its
ancestry cluster. That is why it only ever takes four values -- four clusters,
not four long shared segments. The old reading also implied a segmentation step
the method does not have: pcLAI predicts one coordinate per window, and the
blocks visible in the display are runs of windows with similar predictions.
Field renamed pcaSegment -> centroid in pclai.as with the description corrected,
and the mouseOver, the detailsScript exportFields and the description page
follow. The README also settles that windows are a fixed 1000 SNPs rather than a
fixed number of bases, and that thickStart is specified to equal chromStart, so
the occasional thickStart == chromStart-1 the converter works around is a bug in
their files rather than something we misread.
A field name and its description live inside each bigBed, so editing pclai.as
does nothing to a built collection. hprc2annotRewriteAs.sh re-emits a built
bigBed with the current .as -- no re-download, no column change, item count
checked across the round trip, and safe to re-run, unlike hprc2annotFixBed.sh.
All 460 pclai.bb were rewritten with it. Worth knowing: those files had been
built from an older pclai.as than the tree and nothing had noticed, so this is
the tool to run after any .as description edit.
genark: the "...Url" inside a detailsScript value must not be rebased the way
bigDataUrl is. hgc resolves a relative detailsScript Url against the track's own
bigDataUrl when it builds the details page, and bigDataUrl has already been
rebased, so the prefix landed twice: the pcLAI scatterplot had been asking for
contrib/hprc2annot/contrib/hprc2annot/pclaiRefPanel.json and quietly getting
nothing on every GenArk hub. In this layout the panel file is symlinked beside
the .bb, so relative-to-the-.bb is the bare file name; rebaseBeside() does that
and is idempotent, so addContrib can be re-run.
refs #35415
- src/hg/makeDb/scripts/hprc2annot/hprc2annotFixBed.sh
- lines changed 2, context: html, text, full: html, text
6b0035d19769346baffe193ef9419c269d46f8d8 Wed Sep 9 06:09:41 2026 -0700
hprc2annot: pcLAI column 10 is the ancestry centroid, not a segment coordinate
Reading the pcLAI authors' own format description
(github.com/AI-sandbox/hprc-pclai, README "Output format (BED)") while building
the same annotation as a native hg38 track showed that column 10 of the source
BED had been described wrongly here. It is not the PCA coordinate of a longer
ancestry segment the window belongs to; the authors call it the centroid, the
discretized pcLAI ancestry of the window written as the PCA centroid of its
ancestry cluster. That is why it only ever takes four values -- four clusters,
not four long shared segments. The old reading also implied a segmentation step
the method does not have: pcLAI predicts one coordinate per window, and the
blocks visible in the display are runs of windows with similar predictions.
Field renamed pcaSegment -> centroid in pclai.as with the description corrected,
and the mouseOver, the detailsScript exportFields and the description page
follow. The README also settles that windows are a fixed 1000 SNPs rather than a
fixed number of bases, and that thickStart is specified to equal chromStart, so
the occasional thickStart == chromStart-1 the converter works around is a bug in
their files rather than something we misread.
A field name and its description live inside each bigBed, so editing pclai.as
does nothing to a built collection. hprc2annotRewriteAs.sh re-emits a built
bigBed with the current .as -- no re-download, no column change, item count
checked across the round trip, and safe to re-run, unlike hprc2annotFixBed.sh.
All 460 pclai.bb were rewritten with it. Worth knowing: those files had been
built from an older pclai.as than the tree and nothing had noticed, so this is
the tool to run after any .as description edit.
genark: the "...Url" inside a detailsScript value must not be rebased the way
bigDataUrl is. hgc resolves a relative detailsScript Url against the track's own
bigDataUrl when it builds the details page, and bigDataUrl has already been
rebased, so the prefix landed twice: the pcLAI scatterplot had been asking for
contrib/hprc2annot/contrib/hprc2annot/pclaiRefPanel.json and quietly getting
nothing on every GenArk hub. In this layout the panel file is symlinked beside
the .bb, so relative-to-the-.bb is the bare file name; rebaseBeside() does that
and is idempotent, so addContrib can be re-run.
refs #35415
- src/hg/makeDb/scripts/hprc2annot/hprc2annotMakePclaiRefPanel.py
- lines changed 81, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- src/hg/makeDb/scripts/hprc2annot/hprc2annotRewriteAs.sh
- lines changed 45, context: html, text, full: html, text
6b0035d19769346baffe193ef9419c269d46f8d8 Wed Sep 9 06:09:41 2026 -0700
hprc2annot: pcLAI column 10 is the ancestry centroid, not a segment coordinate
Reading the pcLAI authors' own format description
(github.com/AI-sandbox/hprc-pclai, README "Output format (BED)") while building
the same annotation as a native hg38 track showed that column 10 of the source
BED had been described wrongly here. It is not the PCA coordinate of a longer
ancestry segment the window belongs to; the authors call it the centroid, the
discretized pcLAI ancestry of the window written as the PCA centroid of its
ancestry cluster. That is why it only ever takes four values -- four clusters,
not four long shared segments. The old reading also implied a segmentation step
the method does not have: pcLAI predicts one coordinate per window, and the
blocks visible in the display are runs of windows with similar predictions.
Field renamed pcaSegment -> centroid in pclai.as with the description corrected,
and the mouseOver, the detailsScript exportFields and the description page
follow. The README also settles that windows are a fixed 1000 SNPs rather than a
fixed number of bases, and that thickStart is specified to equal chromStart, so
the occasional thickStart == chromStart-1 the converter works around is a bug in
their files rather than something we misread.
A field name and its description live inside each bigBed, so editing pclai.as
does nothing to a built collection. hprc2annotRewriteAs.sh re-emits a built
bigBed with the current .as -- no re-download, no column change, item count
checked across the round trip, and safe to re-run, unlike hprc2annotFixBed.sh.
All 460 pclai.bb were rewritten with it. Worth knowing: those files had been
built from an older pclai.as than the tree and nothing had noticed, so this is
the tool to run after any .as description edit.
genark: the "...Url" inside a detailsScript value must not be rebased the way
bigDataUrl is. hgc resolves a relative detailsScript Url against the track's own
bigDataUrl when it builds the details page, and bigDataUrl has already been
rebased, so the prefix landed twice: the pcLAI scatterplot had been asking for
contrib/hprc2annot/contrib/hprc2annot/pclaiRefPanel.json and quietly getting
nothing on every GenArk hub. In this layout the panel file is symlinked beside
the .bb, so relative-to-the-.bb is the bare file name; rebaseBeside() does that
and is idempotent, so addContrib can be re-run.
refs #35415
- src/hg/makeDb/scripts/hprc2annot/pclai.as
- lines changed 5, context: html, text, full: html, text
6b0035d19769346baffe193ef9419c269d46f8d8 Wed Sep 9 06:09:41 2026 -0700
hprc2annot: pcLAI column 10 is the ancestry centroid, not a segment coordinate
Reading the pcLAI authors' own format description
(github.com/AI-sandbox/hprc-pclai, README "Output format (BED)") while building
the same annotation as a native hg38 track showed that column 10 of the source
BED had been described wrongly here. It is not the PCA coordinate of a longer
ancestry segment the window belongs to; the authors call it the centroid, the
discretized pcLAI ancestry of the window written as the PCA centroid of its
ancestry cluster. That is why it only ever takes four values -- four clusters,
not four long shared segments. The old reading also implied a segmentation step
the method does not have: pcLAI predicts one coordinate per window, and the
blocks visible in the display are runs of windows with similar predictions.
Field renamed pcaSegment -> centroid in pclai.as with the description corrected,
and the mouseOver, the detailsScript exportFields and the description page
follow. The README also settles that windows are a fixed 1000 SNPs rather than a
fixed number of bases, and that thickStart is specified to equal chromStart, so
the occasional thickStart == chromStart-1 the converter works around is a bug in
their files rather than something we misread.
A field name and its description live inside each bigBed, so editing pclai.as
does nothing to a built collection. hprc2annotRewriteAs.sh re-emits a built
bigBed with the current .as -- no re-download, no column change, item count
checked across the round trip, and safe to re-run, unlike hprc2annotFixBed.sh.
All 460 pclai.bb were rewritten with it. Worth knowing: those files had been
built from an older pclai.as than the tree and nothing had noticed, so this is
the tool to run after any .as description edit.
genark: the "...Url" inside a detailsScript value must not be rebased the way
bigDataUrl is. hgc resolves a relative detailsScript Url against the track's own
bigDataUrl when it builds the details page, and bigDataUrl has already been
rebased, so the prefix landed twice: the pcLAI scatterplot had been asking for
contrib/hprc2annot/contrib/hprc2annot/pclaiRefPanel.json and quietly getting
nothing on every GenArk hub. In this layout the panel file is symlinked beside
the .bb, so relative-to-the-.bb is the bare file name; rebaseBeside() does that
and is idempotent, so addContrib can be re-run.
refs #35415
- src/hg/makeDb/scripts/hprcPclai/hprcPclai.as
- lines changed 17, context: html, text, full: html, text
2c0adaa48b2f9a14109c3f90713405f259d920bc Wed Sep 9 06:08:46 2026 -0700
hg38: pcLAI local ancestry track for HPRC Release 2 haplotypes
Point cloud local ancestry inference (pcLAI) for HPRC Release 2, projected
onto GRCh38: a composite with one subtrack per haplotype, both haplotypes of
231 samples plus CHM13, 463 in all. 11,936,603 windows, autosomes only, no
windows dropped from the source files.
Each window carries the (PC1,PC2) coordinate pcLAI predicts for it, the
discretized ancestry centroid, and a confidence score. The details page draws
the window's position against the 1000 Genomes reference panel that defines
the space, via detailsScript/scatterPlot; metaDataUrl is what lets hgTrackUi
serve that panel file for a native (non-hub) track.
thickStart is one base before chromStart in 54,811 of the windows (0.46%),
against the format the pcLAI authors document, so bedToBigBed rejects it.
Neither thick column carries information here, so both are set to the item
bounds rather than dropping those windows.
Testing only for now, alpha, no ticket yet; a ticket may follow if this
becomes a real track. The makeDoc carries the detail in the meantime,
including what was deliberately left undone.
- src/hg/makeDb/scripts/hprcPclai/hprcPclaiDefaultOn.txt
- lines changed 11, context: html, text, full: html, text
2c0adaa48b2f9a14109c3f90713405f259d920bc Wed Sep 9 06:08:46 2026 -0700
hg38: pcLAI local ancestry track for HPRC Release 2 haplotypes
Point cloud local ancestry inference (pcLAI) for HPRC Release 2, projected
onto GRCh38: a composite with one subtrack per haplotype, both haplotypes of
231 samples plus CHM13, 463 in all. 11,936,603 windows, autosomes only, no
windows dropped from the source files.
Each window carries the (PC1,PC2) coordinate pcLAI predicts for it, the
discretized ancestry centroid, and a confidence score. The details page draws
the window's position against the 1000 Genomes reference panel that defines
the space, via detailsScript/scatterPlot; metaDataUrl is what lets hgTrackUi
serve that panel file for a native (non-hub) track.
thickStart is one base before chromStart in 54,811 of the windows (0.46%),
against the format the pcLAI authors document, so bedToBigBed rejects it.
Neither thick column carries information here, so both are set to the item
bounds rather than dropping those windows.
Testing only for now, alpha, no ticket yet; a ticket may follow if this
becomes a real track. The makeDoc carries the detail in the meantime,
including what was deliberately left undone.
- src/hg/makeDb/scripts/hprcPclai/hprcPclaiDownload.sh
- lines changed 28, context: html, text, full: html, text
2c0adaa48b2f9a14109c3f90713405f259d920bc Wed Sep 9 06:08:46 2026 -0700
hg38: pcLAI local ancestry track for HPRC Release 2 haplotypes
Point cloud local ancestry inference (pcLAI) for HPRC Release 2, projected
onto GRCh38: a composite with one subtrack per haplotype, both haplotypes of
231 samples plus CHM13, 463 in all. 11,936,603 windows, autosomes only, no
windows dropped from the source files.
Each window carries the (PC1,PC2) coordinate pcLAI predicts for it, the
discretized ancestry centroid, and a confidence score. The details page draws
the window's position against the 1000 Genomes reference panel that defines
the space, via detailsScript/scatterPlot; metaDataUrl is what lets hgTrackUi
serve that panel file for a native (non-hub) track.
thickStart is one base before chromStart in 54,811 of the windows (0.46%),
against the format the pcLAI authors document, so bedToBigBed rejects it.
Neither thick column carries information here, so both are set to the item
bounds rather than dropping those windows.
Testing only for now, alpha, no ticket yet; a ticket may follow if this
becomes a real track. The makeDoc carries the detail in the meantime,
including what was deliberately left undone.
- src/hg/makeDb/scripts/hprcPclai/hprcPclaiMakeBb.sh
- lines changed 46, context: html, text, full: html, text
2c0adaa48b2f9a14109c3f90713405f259d920bc Wed Sep 9 06:08:46 2026 -0700
hg38: pcLAI local ancestry track for HPRC Release 2 haplotypes
Point cloud local ancestry inference (pcLAI) for HPRC Release 2, projected
onto GRCh38: a composite with one subtrack per haplotype, both haplotypes of
231 samples plus CHM13, 463 in all. 11,936,603 windows, autosomes only, no
windows dropped from the source files.
Each window carries the (PC1,PC2) coordinate pcLAI predicts for it, the
discretized ancestry centroid, and a confidence score. The details page draws
the window's position against the 1000 Genomes reference panel that defines
the space, via detailsScript/scatterPlot; metaDataUrl is what lets hgTrackUi
serve that panel file for a native (non-hub) track.
thickStart is one base before chromStart in 54,811 of the windows (0.46%),
against the format the pcLAI authors document, so bedToBigBed rejects it.
Neither thick column carries information here, so both are set to the item
bounds rather than dropping those windows.
Testing only for now, alpha, no ticket yet; a ticket may follow if this
becomes a real track. The makeDoc carries the detail in the meantime,
including what was deliberately left undone.
- src/hg/makeDb/scripts/hprcPclai/hprcPclaiMakeTrackDb.py
- lines changed 130, context: html, text, full: html, text
2c0adaa48b2f9a14109c3f90713405f259d920bc Wed Sep 9 06:08:46 2026 -0700
hg38: pcLAI local ancestry track for HPRC Release 2 haplotypes
Point cloud local ancestry inference (pcLAI) for HPRC Release 2, projected
onto GRCh38: a composite with one subtrack per haplotype, both haplotypes of
231 samples plus CHM13, 463 in all. 11,936,603 windows, autosomes only, no
windows dropped from the source files.
Each window carries the (PC1,PC2) coordinate pcLAI predicts for it, the
discretized ancestry centroid, and a confidence score. The details page draws
the window's position against the 1000 Genomes reference panel that defines
the space, via detailsScript/scatterPlot; metaDataUrl is what lets hgTrackUi
serve that panel file for a native (non-hub) track.
thickStart is one base before chromStart in 54,811 of the windows (0.46%),
against the format the pcLAI authors document, so bedToBigBed rejects it.
Neither thick column carries information here, so both are set to the item
bounds rather than dropping those windows.
Testing only for now, alpha, no ticket yet; a ticket may follow if this
becomes a real track. The makeDoc carries the detail in the meantime,
including what was deliberately left undone.
- src/hg/makeDb/scripts/hprcRdt/hprcRdtBuild.sh
- lines changed 53, context: html, text, full: html, text
ef779a5a2ed508cb00f0b0139a12d696e439ee5f Fri Sep 11 06:06:55 2026 -0700
new hg38 track hprcRdt: reference-divergent transcripts from 206 HPRC Release 2 genomes
Added as a third child of the existing long_read_transcripts superTrack,
alpha only for now. Data from Max Marin (DFCI), a bigPsl of RDT cluster
representative sequences aligned to GRCh38: 180,464 alignments of 120,451
distinct sequences from 412 haplotypes. Rebuilt from the submitted file only
to add a name index, refs #33822
- src/hg/makeDb/scripts/imprinting/kaplanAsm.as
- lines changed 20, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/scripts/imprinting/kaplanBimodal.as
- lines changed 17, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/scripts/imprinting/kaplanIcr.as
- lines changed 18, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/scripts/imprinting/kaplanIcrAddOrig.py
- lines changed 40, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/scripts/imprinting/kaplanImprintLift.sh
- lines changed 56, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/scripts/imprinting/kaplanImprintToBed.py
- lines changed 357, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/scripts/imprinting/kaplanImprintXlsxToTsv.py
- lines changed 54, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/scripts/imprinting/kaplanLiftNote.py
- lines changed 41, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/scripts/imprinting/kaplanParentalAsm.as
- lines changed 29, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/scripts/imprinting/omimImprint.as
- lines changed 1, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/scripts/imprinting/omimImprintToBed.py
- lines changed 4, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/trackDb/README
- lines changed 11, context: html, text, full: html, text
c0e8fa6df3a0bd406c4188d49ee00f20aef203e5 Mon Sep 7 12:07:18 2026 -0700
Substitute trackDb variables in hub track description pages
A hub's description page comes straight off the hub's web server and has
never been through variable substitution, so a $db or $parentTrack in it
reached the reader as literal text. Native trackDb pages are fine, since
hgTrackDb substitutes them when it loads trackDb, but there was no
equivalent step for a hub.
hgc's getTrackHtml and hgTrackUi's trackUi both call hVarSubstTrackDbHtml
on a hub track's html. Only a short list of variables is recognized there and nothing is an
error, because a hub page written before this existed can easily contain
a dollar sign inside a shell example, and silently rewriting that would
be worse than not substituting at all.
Adds $parentTrack, the name of the container a track sits in, which is
what a subtrack description page needs to link back to its superTrack or
composite. Views are skipped, since a view has no page of its own, and
the hub_<id>_ prefix is kept so the name works as hgTrackUi's g=
parameter. Documents $track, $parentTrack and $hgsid in trackDb/README.
refs #37599
- lines changed 38, context: html, text, full: html, text
9c620cab64ca86a0044de2c4a7c610b13332558e Mon Sep 7 19:43:50 2026 -0700
trackDb/README: document the braced ${name} form only
The bare $name form still works and old pages use it, but braces are the form
worth writing. Inside braces any character is allowed up to the closing brace,
so ${name} can carry a structured name like ${hgTrackUi/caddSuper} if we ever
want one, while $name stops at the first character outside [0-9A-Za-Z_] and
cannot express it. Braces also settle the adjacency case: ${db}Something is
unambiguous, $dbSomething reads as one long name.
Every variable in the list is now written braced, and the closing paragraph says
not to write or document the bare form in anything new.
Also spells out why ${hgsid} is a hub page variable in practice. Native trackDb
html is substituted once by hgTrackDb when it loads trackDb, and that happens
without a cart, so there is no session id to put there. Only hgc and hgTrackUi
substitute with a cart, at render time.
One non-variable change: 'an $otherDb field' becomes 'an otherDb field'. That is
the name of a .ra setting rather than a substitution, and leaving a dollar on it
next to a braces-only rule would only confuse.
refs #38283
- src/hg/makeDb/trackDb/contrib/RCPediaVGP_v1/RCPediaVGP_v1.html
- lines changed 67, context: html, text, full: html, text
3cdeeced0d0a25cf6de114092f6ad62a35d63a97 Fri Sep 11 16:09:35 2026 -0700
meat puppet here, claude didn't like the name mismatch, oh well, whatever, it isn't important what the name is here refs #38224
- src/hg/makeDb/trackDb/contrib/RCPediaVGP_v1/README.txt
- lines changed 61, context: html, text, full: html, text
ca3e7270f30b554962e47a8ee1cec08a00de5862 Fri Sep 11 16:24:26 2026 -0700
silence claude noise refs #38290
- src/hg/makeDb/trackDb/contrib/RCPediaVGP_v1/mkSymLinks.sh
- lines changed 47, context: html, text, full: html, text
ca3e7270f30b554962e47a8ee1cec08a00de5862 Fri Sep 11 16:24:26 2026 -0700
silence claude noise refs #38290
- src/hg/makeDb/trackDb/contrib/RCPediaVGP_v1/retrocopies.html
- lines changed 67, context: html, text, full: html, text
0b88e0fa63875f67aaf77a4fefc4038ee6caee58 Fri Sep 11 16:09:16 2026 -0700
meat puppet here, claude didn't like the name mismatch, oh well, whatever, it isn't important what the name is here refs #38224
- src/hg/makeDb/trackDb/contrib/bTaeGut7/bTaeGut7.trackDb.txt
- lines changed 292, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/centroCores.html
- lines changed 70, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/centroMarkers.html
- lines changed 81, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/centroSat.html
- lines changed 79, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/centroTelo.html
- lines changed 39, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/chromatin.html
- lines changed 38, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/compartAB.html
- lines changed 82, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/compartE1.html
- lines changed 76, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/covClr.html
- lines changed 63, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/covHifi.html
- lines changed 68, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/covOnt.html
- lines changed 67, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/egapx.html
- lines changed 76, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/gcPercent.html
- lines changed 59, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/genes.html
- lines changed 38, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/hubDescription.html
- lines changed 54, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/itsRepeats.html
- lines changed 68, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/largeSv.html
- lines changed 77, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/methyl5mC.html
- lines changed 65, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/newRegions.html
- lines changed 79, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/nonBdna.html
- lines changed 107, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/readCoverage.html
- lines changed 38, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/repeats.html
- lines changed 38, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/retrocopies.html
- lines changed 85, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/satellome.html
- lines changed 70, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/seqEntropy.html
- lines changed 60, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/structVar.html
- lines changed 38, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/tandemRepeats.html
- lines changed 79, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/telomeres.html
- lines changed 69, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/bTaeGut7/transposons.html
- lines changed 89, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/hg/makeDb/trackDb/contrib/hprc2annot/catGenes.html
- lines changed 1, context: html, text, full: html, text
1682366b1827b7559f8e1e41635acff6c5ea15e9 Wed Sep 9 06:05:05 2026 -0700
hprc2annot: move the makeDoc into its own directory and repoint the links
The makeDoc has grown a companion (an hg38 pcLAI doc is in progress), so it
moves from doc/contrib/hprc2annot.txt into doc/contrib/hprc2annot/, matching
how the scripts and trackDb copies are already laid out. The file itself gains
a section on the pcLAI scatterplot on the details page: where the reference
panel comes from, the four ancestry centroids the discretized field takes
across the release, and why the file is read through hgTrackUi rather than
fetched by the browser.
All seven track description pages linked to the old flat path and would have
404'd, so they are repointed. Six of them change only that link; pclai.html has
further edits still in progress and keeps its own copy of the change.
refs #35415
- src/hg/makeDb/trackDb/contrib/hprc2annot/censat.html
- lines changed 1, context: html, text, full: html, text
1682366b1827b7559f8e1e41635acff6c5ea15e9 Wed Sep 9 06:05:05 2026 -0700
hprc2annot: move the makeDoc into its own directory and repoint the links
The makeDoc has grown a companion (an hg38 pcLAI doc is in progress), so it
moves from doc/contrib/hprc2annot.txt into doc/contrib/hprc2annot/, matching
how the scripts and trackDb copies are already laid out. The file itself gains
a section on the pcLAI scatterplot on the details page: where the reference
panel comes from, the four ancestry centroids the discretized field takes
across the release, and why the file is read through hgTrackUi rather than
fetched by the browser.
All seven track description pages linked to the old flat path and would have
404'd, so they are repointed. Six of them change only that link; pclai.html has
further edits still in progress and keeps its own copy of the change.
refs #35415
- src/hg/makeDb/trackDb/contrib/hprc2annot/censatCentromeres.html
- lines changed 1, context: html, text, full: html, text
1682366b1827b7559f8e1e41635acff6c5ea15e9 Wed Sep 9 06:05:05 2026 -0700
hprc2annot: move the makeDoc into its own directory and repoint the links
The makeDoc has grown a companion (an hg38 pcLAI doc is in progress), so it
moves from doc/contrib/hprc2annot.txt into doc/contrib/hprc2annot/, matching
how the scripts and trackDb copies are already laid out. The file itself gains
a section on the pcLAI scatterplot on the details page: where the reference
panel comes from, the four ancestry centroids the discretized field takes
across the release, and why the file is read through hgTrackUi rather than
fetched by the browser.
All seven track description pages linked to the old flat path and would have
404'd, so they are repointed. Six of them change only that link; pclai.html has
further edits still in progress and keeps its own copy of the change.
refs #35415
- src/hg/makeDb/trackDb/contrib/hprc2annot/hprc2annot.trackDb.txt
- lines changed 1, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- lines changed 3, context: html, text, full: html, text
6b0035d19769346baffe193ef9419c269d46f8d8 Wed Sep 9 06:09:41 2026 -0700
hprc2annot: pcLAI column 10 is the ancestry centroid, not a segment coordinate
Reading the pcLAI authors' own format description
(github.com/AI-sandbox/hprc-pclai, README "Output format (BED)") while building
the same annotation as a native hg38 track showed that column 10 of the source
BED had been described wrongly here. It is not the PCA coordinate of a longer
ancestry segment the window belongs to; the authors call it the centroid, the
discretized pcLAI ancestry of the window written as the PCA centroid of its
ancestry cluster. That is why it only ever takes four values -- four clusters,
not four long shared segments. The old reading also implied a segmentation step
the method does not have: pcLAI predicts one coordinate per window, and the
blocks visible in the display are runs of windows with similar predictions.
Field renamed pcaSegment -> centroid in pclai.as with the description corrected,
and the mouseOver, the detailsScript exportFields and the description page
follow. The README also settles that windows are a fixed 1000 SNPs rather than a
fixed number of bases, and that thickStart is specified to equal chromStart, so
the occasional thickStart == chromStart-1 the converter works around is a bug in
their files rather than something we misread.
A field name and its description live inside each bigBed, so editing pclai.as
does nothing to a built collection. hprc2annotRewriteAs.sh re-emits a built
bigBed with the current .as -- no re-download, no column change, item count
checked across the round trip, and safe to re-run, unlike hprc2annotFixBed.sh.
All 460 pclai.bb were rewritten with it. Worth knowing: those files had been
built from an older pclai.as than the tree and nothing had noticed, so this is
the tool to run after any .as description edit.
genark: the "...Url" inside a detailsScript value must not be rebased the way
bigDataUrl is. hgc resolves a relative detailsScript Url against the track's own
bigDataUrl when it builds the details page, and bigDataUrl has already been
rebased, so the prefix landed twice: the pcLAI scatterplot had been asking for
contrib/hprc2annot/contrib/hprc2annot/pclaiRefPanel.json and quietly getting
nothing on every GenArk hub. In this layout the panel file is symlinked beside
the .bb, so relative-to-the-.bb is the bare file name; rebaseBeside() does that
and is idempotent, so addContrib can be re-run.
refs #35415
- src/hg/makeDb/trackDb/contrib/hprc2annot/liftoffGenes.html
- lines changed 1, context: html, text, full: html, text
1682366b1827b7559f8e1e41635acff6c5ea15e9 Wed Sep 9 06:05:05 2026 -0700
hprc2annot: move the makeDoc into its own directory and repoint the links
The makeDoc has grown a companion (an hg38 pcLAI doc is in progress), so it
moves from doc/contrib/hprc2annot.txt into doc/contrib/hprc2annot/, matching
how the scripts and trackDb copies are already laid out. The file itself gains
a section on the pcLAI scatterplot on the details page: where the reference
panel comes from, the four ancestry centroids the discretized field takes
across the release, and why the file is read through hgTrackUi rather than
fetched by the browser.
All seven track description pages linked to the old flat path and would have
404'd, so they are repointed. Six of them change only that link; pclai.html has
further edits still in progress and keeps its own copy of the change.
refs #35415
- src/hg/makeDb/trackDb/contrib/hprc2annot/methylation.html
- lines changed 1, context: html, text, full: html, text
1682366b1827b7559f8e1e41635acff6c5ea15e9 Wed Sep 9 06:05:05 2026 -0700
hprc2annot: move the makeDoc into its own directory and repoint the links
The makeDoc has grown a companion (an hg38 pcLAI doc is in progress), so it
moves from doc/contrib/hprc2annot.txt into doc/contrib/hprc2annot/, matching
how the scripts and trackDb copies are already laid out. The file itself gains
a section on the pcLAI scatterplot on the details page: where the reference
panel comes from, the four ancestry centroids the discretized field takes
across the release, and why the file is read through hgTrackUi rather than
fetched by the browser.
All seven track description pages linked to the old flat path and would have
404'd, so they are repointed. Six of them change only that link; pclai.html has
further edits still in progress and keeps its own copy of the change.
refs #35415
- src/hg/makeDb/trackDb/contrib/hprc2annot/pclai.html
- lines changed 21, context: html, text, full: html, text
fa8084416354441c8c817069b7f92344f833d8dc Tue Sep 8 01:13:43 2026 -0700
hprc2annot: say in the pcLAI docs that uniform color is the expected case
Two of us in a row zoomed a pcLAI track to a few megabases, saw a single flat
color, and concluded the itemRgb was broken. It is not: pclai.bb for
GCA_046629565.1 holds 258 distinct RGB values, but 25,416 of its 25,438 windows
sit in one tight cluster in PCA space (PC1 ~0.40-0.44) and so map to one tight
cluster in color space, R 203-255 G 149-164 B 255, which is a couple of
perceptual steps wide. The color column is a byte-for-byte pass-through of the
HPRC source BED, verified against the S3 file.
Two things the page did not say. A haplotype with one ancestry throughout is
uniform across every chromosome and that is the correct render. And where a
haplotype does carry several ancestries, the blocks are tens of megabases long,
so any view narrower than a chromosome tends to land inside one block and look
uniform too.
Add an example so this is checkable rather than asserted: GCA_018466835.2
(HG02257) has four segment-level ancestry calls, and its chr17 crosses four
blocks with transitions near 10.0, 50.7 and 73.3 Mb. Name the three
chromosomes in that same assembly, chr2 chr13 chr18, that are single-ancestry
end to end, since chr2 is the one that started this.
refs #35415
- lines changed 29, context: html, text, full: html, text
6b0035d19769346baffe193ef9419c269d46f8d8 Wed Sep 9 06:09:41 2026 -0700
hprc2annot: pcLAI column 10 is the ancestry centroid, not a segment coordinate
Reading the pcLAI authors' own format description
(github.com/AI-sandbox/hprc-pclai, README "Output format (BED)") while building
the same annotation as a native hg38 track showed that column 10 of the source
BED had been described wrongly here. It is not the PCA coordinate of a longer
ancestry segment the window belongs to; the authors call it the centroid, the
discretized pcLAI ancestry of the window written as the PCA centroid of its
ancestry cluster. That is why it only ever takes four values -- four clusters,
not four long shared segments. The old reading also implied a segmentation step
the method does not have: pcLAI predicts one coordinate per window, and the
blocks visible in the display are runs of windows with similar predictions.
Field renamed pcaSegment -> centroid in pclai.as with the description corrected,
and the mouseOver, the detailsScript exportFields and the description page
follow. The README also settles that windows are a fixed 1000 SNPs rather than a
fixed number of bases, and that thickStart is specified to equal chromStart, so
the occasional thickStart == chromStart-1 the converter works around is a bug in
their files rather than something we misread.
A field name and its description live inside each bigBed, so editing pclai.as
does nothing to a built collection. hprc2annotRewriteAs.sh re-emits a built
bigBed with the current .as -- no re-download, no column change, item count
checked across the round trip, and safe to re-run, unlike hprc2annotFixBed.sh.
All 460 pclai.bb were rewritten with it. Worth knowing: those files had been
built from an older pclai.as than the tree and nothing had noticed, so this is
the tool to run after any .as description edit.
genark: the "...Url" inside a detailsScript value must not be rebased the way
bigDataUrl is. hgc resolves a relative detailsScript Url against the track's own
bigDataUrl when it builds the details page, and bigDataUrl has already been
rebased, so the prefix landed twice: the pcLAI scatterplot had been asking for
contrib/hprc2annot/contrib/hprc2annot/pclaiRefPanel.json and quietly getting
nothing on every GenArk hub. In this layout the panel file is symlinked beside
the .bb, so relative-to-the-.bb is the bare file name; rebaseBeside() does that
and is idempotent, so addContrib can be re-run.
refs #35415
- src/hg/makeDb/trackDb/contrib/hprc2annot/segdups.html
- lines changed 1, context: html, text, full: html, text
1682366b1827b7559f8e1e41635acff6c5ea15e9 Wed Sep 9 06:05:05 2026 -0700
hprc2annot: move the makeDoc into its own directory and repoint the links
The makeDoc has grown a companion (an hg38 pcLAI doc is in progress), so it
moves from doc/contrib/hprc2annot.txt into doc/contrib/hprc2annot/, matching
how the scripts and trackDb copies are already laid out. The file itself gains
a section on the pcLAI scatterplot on the details page: where the reference
panel comes from, the four ancestry centroids the discretized field takes
across the release, and why the file is read through hgTrackUi rather than
fetched by the browser.
All seven track description pages linked to the old flat path and would have
404'd, so they are repointed. Six of them change only that link; pclai.html has
further edits still in progress and keeps its own copy of the change.
refs #35415
- src/hg/makeDb/trackDb/human/alphaGenome.html
- lines changed 28, context: html, text, full: html, text
7ee42571461ed5a384f3d1b628253a8773850cb2 Wed Sep 9 08:54:16 2026 -0700
AlphaGenome track doc: preprint reference, threshold guidance from the authors
The AlphaGenome authors reviewed the description page and asked for the
peer-review status to be updated, for their threshold recommendation to
be stated, and for links to the Atlas preprint and the educational
guides.
- lines changed 4, context: html, text, full: html, text
3dd0329ee0e804f5841a9998c4e7f024d3276921 Wed Sep 9 17:20:20 2026 -0700
Releasing the AlphaGenome Variant Impact (AVI) score track for hg38. refs #38261
Moves the bigWigs to /gbdb/hg38/_alphaGenome/, the underscore convention that keeps
non-redistributable data off the download server, matching PromoterAI and PrimateAI,
and drops the alpha release tag.
Shortens the composite longLabel to fit the 85 character limit and lowercases Score
in the longLabels. Adds a New pennantIcon and points the Deleteriousness Predictions
container pennant at the Sept. 10 news post.
Corrects the median on the description page from 1.6 to 2.9. The old figure counted
the zero-filled reference base slots, which are not variants. Removes two threshold
statements that are not in the AlphaGenome Atlas preprint or any public source. Adds
the /gbdb symlink step to the makeDoc, and fixes three container page links that were
missing target=_blank.
- lines changed 1, context: html, text, full: html, text
7faad3c1104fce7a96b3aff918dcc5546035824c Fri Sep 11 05:23:29 2026 -0700
small docs update after google request by email
- src/hg/makeDb/trackDb/human/alphaGenome.ra
- lines changed 10, context: html, text, full: html, text
3dd0329ee0e804f5841a9998c4e7f024d3276921 Wed Sep 9 17:20:20 2026 -0700
Releasing the AlphaGenome Variant Impact (AVI) score track for hg38. refs #38261
Moves the bigWigs to /gbdb/hg38/_alphaGenome/, the underscore convention that keeps
non-redistributable data off the download server, matching PromoterAI and PrimateAI,
and drops the alpha release tag.
Shortens the composite longLabel to fit the 85 character limit and lowercases Score
in the longLabels. Adds a New pennantIcon and points the Deleteriousness Predictions
container pennant at the Sept. 10 news post.
Corrects the median on the description page from 1.6 to 2.9. The old figure counted
the zero-filled reference base slots, which are not variants. Removes two threshold
statements that are not in the AlphaGenome Atlas preprint or any public source. Adds
the /gbdb symlink step to the makeDoc, and fixes three container page links that were
missing target=_blank.
- src/hg/makeDb/trackDb/human/hg19/refSeqComposite.html
- lines changed 6, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- src/hg/makeDb/trackDb/human/hg38/akbariIdmr.html
- lines changed 8, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- lines changed 1, context: html, text, full: html, text
d033cea2063e9362949baf5b4d8b837597173a0d Thu Sep 10 05:16:05 2026 -0700
Address the code review of the Sep 9 commits
Faceted composite: text that comes from a hub - a metadata column's
description, the name and title of a data type, and the values quoted back in
the "could not load the metadata" row - is put on the page as text rather than
as markup. The three places built their markup from template strings, so a
value carrying angle brackets or a quote was read as HTML: the column
description now goes through the shared htmlEncode() once where the header is
parsed, and the other two build their elements as nodes. The error row reads
better for it as well, since a value with brackets in it used to disappear from
the message that was meant to show it.
The saved UI state keys on the assembly as well as the metadata id. localStorage
is per-origin, so two assemblies whose tracks share a name were sharing one
entry, and a row order dragged on one came back on the other over a different
set of samples. hgTrackUi passes the database down for it. State saved under the
old key is dropped, which costs a facet selection or a page length.
Imprinting: the five subtrack description pages link back to the container as
hgTrackUi?db=$db&g=$parentTrack, without the hgsid. Native trackDb html is
substituted by hgTrackDb as it loads the table, where there is no cart, so
${hgsid} came out empty and the link read 'hgsid=&g=...'. Matches what the
Fiber-seq pages already do. The makeDoc note that described the old form is
updated with the reason.
UniProt otto: README.txt lists all eight things that reach runLog.txt. It had
four, and was missing LOCKED, along with PREFLIGHT-FAIL, END and INTERRUPTED.
refs #36210
refs #37599
refs #38300
- src/hg/makeDb/trackDb/human/hg38/fiberSeq.html
- lines changed 89, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- lines changed 25, context: html, text, full: html, text
01bc05ac9a282a6862111502f13601e513d5b60b Tue Sep 8 06:16:04 2026 -0700
hg38 Fiber-seq: merge the methylation composite into the compendium
The accessibility compendium and the separate Methylation composite covered the
identical 41 samples, and cartDump.c assigns priority with the data element as
the outer loop and the data type as the inner one. So one composite keeps a
sample's six subtracks contiguous in the image, where two composites drew an
accessibility block followed by a methylation block and comparing the two assays
for one sample meant reading past every other sample. Both come off the same
molecules in the same experiment, so side by side is the point. fiberSeqMeth is
gone and its three data types moved in as cpg, cpgHap and cpgDiff, renamed
because "hap" was already taken by the accessibility overlay and a data type
name cannot contain an underscore.
Subtracks now carry an explicit priority, sample outer and declared data type
inner. Without one they fell back to a label sort, so a first visit showed a
sample's data types as Peaks, CpG, Acc rather than in the order of the checkbox
row above the table.
Metadata columns renamed from camelCase to Accession, Sample_class, _Cell_type
and _Sample. toTitleStyle() in facetedComposite.js renders an underscore as a
space but does not split camelCase, so "sampleClass" appeared verbatim as a
column heading. A literal space cannot be used instead: the saved sort order is
a space-separated list of column names and the submit code drops any name
containing whitespace, which would have made sorting silently fail to persist.
Cell type is no longer faceted. A facet value is only offered when it occurs
more than once, and 12 of the 14 cell types here are a single sample, so as a
facet it drew two checkboxes and left 12 samples unreachable by any cell-type
filter. It is a searchable column now, and Sample_class is the only facet until
the lab gives us real HPRC metadata that would facet properly.
Description page intro rewritten, and it now says the assay measures the same
property as DNase-seq and ATAC-seq.
refs #36210
- src/hg/makeDb/trackDb/human/hg38/fiberSeq.ra
- lines changed 6267, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- lines changed 7205, context: html, text, full: html, text
01bc05ac9a282a6862111502f13601e513d5b60b Tue Sep 8 06:16:04 2026 -0700
hg38 Fiber-seq: merge the methylation composite into the compendium
The accessibility compendium and the separate Methylation composite covered the
identical 41 samples, and cartDump.c assigns priority with the data element as
the outer loop and the data type as the inner one. So one composite keeps a
sample's six subtracks contiguous in the image, where two composites drew an
accessibility block followed by a methylation block and comparing the two assays
for one sample meant reading past every other sample. Both come off the same
molecules in the same experiment, so side by side is the point. fiberSeqMeth is
gone and its three data types moved in as cpg, cpgHap and cpgDiff, renamed
because "hap" was already taken by the accessibility overlay and a data type
name cannot contain an underscore.
Subtracks now carry an explicit priority, sample outer and declared data type
inner. Without one they fell back to a label sort, so a first visit showed a
sample's data types as Peaks, CpG, Acc rather than in the order of the checkbox
row above the table.
Metadata columns renamed from camelCase to Accession, Sample_class, _Cell_type
and _Sample. toTitleStyle() in facetedComposite.js renders an underscore as a
space but does not split camelCase, so "sampleClass" appeared verbatim as a
column heading. A literal space cannot be used instead: the saved sort order is
a space-separated list of column names and the submit code drops any name
containing whitespace, which would have made sorting silently fail to persist.
Cell type is no longer faceted. A facet value is only offered when it occurs
more than once, and 12 of the 14 cell types here are a single sample, so as a
facet it drew two checkboxes and left 12 samples unreachable by any cell-type
filter. It is a searchable column now, and Sample_class is the only facet until
the lab gives us real HPRC metadata that would facet properly.
Description page intro rewritten, and it now says the assay measures the same
property as DNase-seq and ATAC-seq.
refs #36210
- lines changed 1, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- lines changed 699, context: html, text, full: html, text
d07356d76401059fbe6c2d0890b1490546a62a4e Mon Sep 14 02:54:50 2026 -0700
hg38 Fiber-seq: reissued GM12878 data and a nucleosome density track, refs #36210
The lab reprocessed GM12878 (PM00001) and replaced the files in place under the
same hash directory. Ten of its twelve files changed; a size sweep over all 41
samples confirmed no other sample is affected. This fixes the two placeholder
haplotype accessibility bigWigs that covered a single base, so that overlay now
draws real data for the sample that comes up by default. Its peak calls changed
substantially as well, 429,883 source peaks before and 196,742 now, which is
noted on the description page since figures made from the first version of the
track will not reproduce for GM12878.
The downloader now fetches into <file>.part and moves it into place when
complete. It used curl -C - straight onto the final file, which is right for an
interrupted transfer and silently corrupting when the server has replaced the
file: it would have appended the tail of the new 5.2 GB hap1 file to the
512-byte stub, and the size check afterwards would have passed. It also takes
an optional list of accessions now, to refresh one sample without walking all 41.
Nucleosome density (all.nucleosome.coverage.bw) was sent separately and is not
in the lab's own hub. It is on the server for all 41 samples and is added as a
seventh data type in the compendium. Unlike every other wiggle here it is a
read depth rather than a percentage, so it cannot take fixed viewLimits: the
genome-wide mean runs from 25 to 142 across samples with sequencing depth and
single loci reach 1.7e5. It is drawn with autoScale, which the description page
explains, and reads as the complement of the accessibility signal.
- lines changed 699, context: html, text, full: html, text
75e828960283291546d2c1a27845e2cf3823adcd Mon Sep 14 05:29:02 2026 -0700
uniprot otto: the miniprot cluster job needs absolute paths
GRCz12ab failed with the parasol job crashing four times, return 1, no output.
The wrapper I wrote ran
miniprot -t 16 --gff protToGenome/GRCz12ab/.../genome.fa fasta/7955.fa > $1
and a parasol job runs with its working directory set to the batch directory, not
to the directory the pipeline runs in, so neither input existed from the job's
point of view. The BLAST batch next door gets away with relative paths because it
cds into its own workdir and its jobList is written relative to that; this batch
directory sits a level deeper and its paths were relative to the otto root.
Every path in the wrapper, the jobList command and the output check is now
absolute.
Verified on the cluster against the real 1.48 Gb zebrafish genome: successful
batch, a 195 MB GFF with 93518 mRNA records.
refs #38300
- src/hg/makeDb/trackDb/human/hg38/fiberSeqAcc.html
- lines changed 122, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- lines changed 2, context: html, text, full: html, text
b947161926eed5f7a41eb598465d1d5cb8e88bc1 Wed Sep 9 06:33:41 2026 -0700
hg38 Fiber-seq: open the accessibility description by naming the collection too
Matches the compendium page. This one already linked to its sibling track
further down, it just did not say what container it sits in.
refs #36210
- src/hg/makeDb/trackDb/human/hg38/fiberSeqCompendium.html
- lines changed 185, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- lines changed 106, context: html, text, full: html, text
01bc05ac9a282a6862111502f13601e513d5b60b Tue Sep 8 06:16:04 2026 -0700
hg38 Fiber-seq: merge the methylation composite into the compendium
The accessibility compendium and the separate Methylation composite covered the
identical 41 samples, and cartDump.c assigns priority with the data element as
the outer loop and the data type as the inner one. So one composite keeps a
sample's six subtracks contiguous in the image, where two composites drew an
accessibility block followed by a methylation block and comparing the two assays
for one sample meant reading past every other sample. Both come off the same
molecules in the same experiment, so side by side is the point. fiberSeqMeth is
gone and its three data types moved in as cpg, cpgHap and cpgDiff, renamed
because "hap" was already taken by the accessibility overlay and a data type
name cannot contain an underscore.
Subtracks now carry an explicit priority, sample outer and declared data type
inner. Without one they fell back to a label sort, so a first visit showed a
sample's data types as Peaks, CpG, Acc rather than in the order of the checkbox
row above the table.
Metadata columns renamed from camelCase to Accession, Sample_class, _Cell_type
and _Sample. toTitleStyle() in facetedComposite.js renders an underscore as a
space but does not split camelCase, so "sampleClass" appeared verbatim as a
column heading. A literal space cannot be used instead: the saved sort order is
a space-separated list of column names and the submit code drops any name
containing whitespace, which would have made sorting silently fail to persist.
Cell type is no longer faceted. A facet value is only offered when it occurs
more than once, and 12 of the 14 cell types here are a single sample, so as a
facet it drew two checkboxes and left 12 samples unreachable by any cell-type
filter. It is a searchable column now, and Sample_class is the only facet until
the lab gives us real HPRC metadata that would facet properly.
Description page intro rewritten, and it now says the assay measures the same
property as DNase-seq and ATAC-seq.
refs #36210
- lines changed 12, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- lines changed 2, context: html, text, full: html, text
4a39786c0e5473f3e987017bcb616b63423a89e2 Wed Sep 9 06:16:03 2026 -0700
hg38 Fiber-seq: open the compendium description by naming the collection it belongs to
A subtrack description page should say which collection it is part of and
link back to that page, so a reader who lands on it from a search result
can get to the container. A bare hgTrackUi link rather than one carrying
${hgsid}: native trackDb html is substituted by hgTrackDb when it loads
the table, where there is no cart, so $hgsid resolves to the empty string
and the link would come out as 'hgsid=&g=fiberSeq'. The ${hgsid} form
works on hub pages, which are substituted at render time instead.
refs #36210
- lines changed 30, context: html, text, full: html, text
d07356d76401059fbe6c2d0890b1490546a62a4e Mon Sep 14 02:54:50 2026 -0700
hg38 Fiber-seq: reissued GM12878 data and a nucleosome density track, refs #36210
The lab reprocessed GM12878 (PM00001) and replaced the files in place under the
same hash directory. Ten of its twelve files changed; a size sweep over all 41
samples confirmed no other sample is affected. This fixes the two placeholder
haplotype accessibility bigWigs that covered a single base, so that overlay now
draws real data for the sample that comes up by default. Its peak calls changed
substantially as well, 429,883 source peaks before and 196,742 now, which is
noted on the description page since figures made from the first version of the
track will not reproduce for GM12878.
The downloader now fetches into <file>.part and moves it into place when
complete. It used curl -C - straight onto the final file, which is right for an
interrupted transfer and silently corrupting when the server has replaced the
file: it would have appended the tail of the new 5.2 GB hap1 file to the
512-byte stub, and the size check afterwards would have passed. It also takes
an optional list of accessions now, to refresh one sample without walking all 41.
Nucleosome density (all.nucleosome.coverage.bw) was sent separately and is not
in the lab's own hub. It is on the server for all 41 samples and is added as a
seventh data type in the compendium. Unlike every other wiggle here it is a
read depth rather than a percentage, so it cannot take fixed viewLimits: the
genome-wide mean runs from 25 to 142 across samples with sequencing depth and
single loci reach 1.7e5. It is drawn with autoScale, which the description page
explains, and reads as the complement of the accessibility signal.
- lines changed 23, context: html, text, full: html, text
75e828960283291546d2c1a27845e2cf3823adcd Mon Sep 14 05:29:02 2026 -0700
uniprot otto: the miniprot cluster job needs absolute paths
GRCz12ab failed with the parasol job crashing four times, return 1, no output.
The wrapper I wrote ran
miniprot -t 16 --gff protToGenome/GRCz12ab/.../genome.fa fasta/7955.fa > $1
and a parasol job runs with its working directory set to the batch directory, not
to the directory the pipeline runs in, so neither input existed from the job's
point of view. The BLAST batch next door gets away with relative paths because it
cds into its own workdir and its jobList is written relative to that; this batch
directory sits a level deeper and its paths were relative to the otto root.
Every path in the wrapper, the jobList command and the output check is now
absolute.
Verified on the cluster against the real 1.48 Gb zebrafish genome: successful
batch, a 195 MB GFF with 93518 mRNA records.
refs #38300
- src/hg/makeDb/trackDb/human/hg38/fiberSeqMeth.html
- lines changed 154, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- lines changed 154, context: html, text, full: html, text
01bc05ac9a282a6862111502f13601e513d5b60b Tue Sep 8 06:16:04 2026 -0700
hg38 Fiber-seq: merge the methylation composite into the compendium
The accessibility compendium and the separate Methylation composite covered the
identical 41 samples, and cartDump.c assigns priority with the data element as
the outer loop and the data type as the inner one. So one composite keeps a
sample's six subtracks contiguous in the image, where two composites drew an
accessibility block followed by a methylation block and comparing the two assays
for one sample meant reading past every other sample. Both come off the same
molecules in the same experiment, so side by side is the point. fiberSeqMeth is
gone and its three data types moved in as cpg, cpgHap and cpgDiff, renamed
because "hap" was already taken by the accessibility overlay and a data type
name cannot contain an underscore.
Subtracks now carry an explicit priority, sample outer and declared data type
inner. Without one they fell back to a label sort, so a first visit showed a
sample's data types as Peaks, CpG, Acc rather than in the order of the checkbox
row above the table.
Metadata columns renamed from camelCase to Accession, Sample_class, _Cell_type
and _Sample. toTitleStyle() in facetedComposite.js renders an underscore as a
space but does not split camelCase, so "sampleClass" appeared verbatim as a
column heading. A literal space cannot be used instead: the saved sort order is
a space-separated list of column names and the submit code drops any name
containing whitespace, which would have made sorting silently fail to persist.
Cell type is no longer faceted. A facet value is only offered when it occurs
more than once, and 12 of the 14 cell types here are a single sample, so as a
facet it drew two checkboxes and left 12 samples unreachable by any cell-type
filter. It is a searchable column now, and Sample_class is the only facet until
the lab gives us real HPRC metadata that would facet properly.
Description page intro rewritten, and it now says the assay measures the same
property as DNase-seq and ATAC-seq.
refs #36210
- src/hg/makeDb/trackDb/human/hg38/geneimprint.html
- lines changed 6, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- lines changed 1, context: html, text, full: html, text
d033cea2063e9362949baf5b4d8b837597173a0d Thu Sep 10 05:16:05 2026 -0700
Address the code review of the Sep 9 commits
Faceted composite: text that comes from a hub - a metadata column's
description, the name and title of a data type, and the values quoted back in
the "could not load the metadata" row - is put on the page as text rather than
as markup. The three places built their markup from template strings, so a
value carrying angle brackets or a quote was read as HTML: the column
description now goes through the shared htmlEncode() once where the header is
parsed, and the other two build their elements as nodes. The error row reads
better for it as well, since a value with brackets in it used to disappear from
the message that was meant to show it.
The saved UI state keys on the assembly as well as the metadata id. localStorage
is per-origin, so two assemblies whose tracks share a name were sharing one
entry, and a row order dragged on one came back on the other over a different
set of samples. hgTrackUi passes the database down for it. State saved under the
old key is dropped, which costs a facet selection or a page length.
Imprinting: the five subtrack description pages link back to the container as
hgTrackUi?db=$db&g=$parentTrack, without the hgsid. Native trackDb html is
substituted by hgTrackDb as it loads the table, where there is no cart, so
${hgsid} came out empty and the link read 'hgsid=&g=...'. Matches what the
Fiber-seq pages already do. The makeDoc note that described the old form is
updated with the reason.
UniProt otto: README.txt lists all eight things that reach runLog.txt. It had
four, and was missing LOCKED, along with PREFLIGHT-FAIL, END and INTERRUPTED.
refs #36210
refs #37599
refs #38300
- src/hg/makeDb/trackDb/human/hg38/hprcPclai.html
- lines changed 146, context: html, text, full: html, text
2c0adaa48b2f9a14109c3f90713405f259d920bc Wed Sep 9 06:08:46 2026 -0700
hg38: pcLAI local ancestry track for HPRC Release 2 haplotypes
Point cloud local ancestry inference (pcLAI) for HPRC Release 2, projected
onto GRCh38: a composite with one subtrack per haplotype, both haplotypes of
231 samples plus CHM13, 463 in all. 11,936,603 windows, autosomes only, no
windows dropped from the source files.
Each window carries the (PC1,PC2) coordinate pcLAI predicts for it, the
discretized ancestry centroid, and a confidence score. The details page draws
the window's position against the 1000 Genomes reference panel that defines
the space, via detailsScript/scatterPlot; metaDataUrl is what lets hgTrackUi
serve that panel file for a native (non-hub) track.
thickStart is one base before chromStart in 54,811 of the windows (0.46%),
against the format the pcLAI authors document, so bedToBigBed rejects it.
Neither thick column carries information here, so both are set to the item
bounds rather than dropping those windows.
Testing only for now, alpha, no ticket yet; a ticket may follow if this
becomes a real track. The makeDoc carries the detail in the meantime,
including what was deliberately left undone.
- src/hg/makeDb/trackDb/human/hg38/hprcPclai.ra
- lines changed 6040, context: html, text, full: html, text
2c0adaa48b2f9a14109c3f90713405f259d920bc Wed Sep 9 06:08:46 2026 -0700
hg38: pcLAI local ancestry track for HPRC Release 2 haplotypes
Point cloud local ancestry inference (pcLAI) for HPRC Release 2, projected
onto GRCh38: a composite with one subtrack per haplotype, both haplotypes of
231 samples plus CHM13, 463 in all. 11,936,603 windows, autosomes only, no
windows dropped from the source files.
Each window carries the (PC1,PC2) coordinate pcLAI predicts for it, the
discretized ancestry centroid, and a confidence score. The details page draws
the window's position against the 1000 Genomes reference panel that defines
the space, via detailsScript/scatterPlot; metaDataUrl is what lets hgTrackUi
serve that panel file for a native (non-hub) track.
thickStart is one base before chromStart in 54,811 of the windows (0.46%),
against the format the pcLAI authors document, so bedToBigBed rejects it.
Neither thick column carries information here, so both are set to the item
bounds rather than dropping those windows.
Testing only for now, alpha, no ticket yet; a ticket may follow if this
becomes a real track. The makeDoc carries the detail in the meantime,
including what was deliberately left undone.
- src/hg/makeDb/trackDb/human/hg38/hprcRdt.html
- lines changed 134, context: html, text, full: html, text
ef779a5a2ed508cb00f0b0139a12d696e439ee5f Fri Sep 11 06:06:55 2026 -0700
new hg38 track hprcRdt: reference-divergent transcripts from 206 HPRC Release 2 genomes
Added as a third child of the existing long_read_transcripts superTrack,
alpha only for now. Data from Max Marin (DFCI), a bigPsl of RDT cluster
representative sequences aligned to GRCh38: 180,464 alignments of 120,451
distinct sequences from 412 haplotypes. Rebuilt from the submitted file only
to add a name index, refs #33822
- src/hg/makeDb/trackDb/human/hg38/hprcRdt.ra
- lines changed 25, context: html, text, full: html, text
ef779a5a2ed508cb00f0b0139a12d696e439ee5f Fri Sep 11 06:06:55 2026 -0700
new hg38 track hprcRdt: reference-divergent transcripts from 206 HPRC Release 2 genomes
Added as a third child of the existing long_read_transcripts superTrack,
alpha only for now. Data from Max Marin (DFCI), a bigPsl of RDT cluster
representative sequences aligned to GRCh38: 180,464 alignments of 120,451
distinct sequences from 412 haplotypes. Rebuilt from the submitted file only
to add a name index, refs #33822
- src/hg/makeDb/trackDb/human/hg38/imprinting.html
- lines changed 27, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/trackDb/human/hg38/imprinting.ra
- lines changed 108, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- src/hg/makeDb/trackDb/human/hg38/kaplanImprint.html
- lines changed 180, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- lines changed 1, context: html, text, full: html, text
d033cea2063e9362949baf5b4d8b837597173a0d Thu Sep 10 05:16:05 2026 -0700
Address the code review of the Sep 9 commits
Faceted composite: text that comes from a hub - a metadata column's
description, the name and title of a data type, and the values quoted back in
the "could not load the metadata" row - is put on the page as text rather than
as markup. The three places built their markup from template strings, so a
value carrying angle brackets or a quote was read as HTML: the column
description now goes through the shared htmlEncode() once where the header is
parsed, and the other two build their elements as nodes. The error row reads
better for it as well, since a value with brackets in it used to disappear from
the message that was meant to show it.
The saved UI state keys on the assembly as well as the metadata id. localStorage
is per-origin, so two assemblies whose tracks share a name were sharing one
entry, and a row order dragged on one came back on the other over a different
set of samples. hgTrackUi passes the database down for it. State saved under the
old key is dropped, which costs a facet selection or a page length.
Imprinting: the five subtrack description pages link back to the container as
hgTrackUi?db=$db&g=$parentTrack, without the hgsid. Native trackDb html is
substituted by hgTrackDb as it loads the table, where there is no cart, so
${hgsid} came out empty and the link read 'hgsid=&g=...'. Matches what the
Fiber-seq pages already do. The makeDoc note that described the old form is
updated with the reason.
UniProt otto: README.txt lists all eight things that reach runLog.txt. It had
four, and was missing LOCKED, along with PREFLIGHT-FAIL, END and INTERRUPTED.
refs #36210
refs #37599
refs #38300
- src/hg/makeDb/trackDb/human/hg38/long_read_transcripts.ra
- lines changed 1, context: html, text, full: html, text
ef779a5a2ed508cb00f0b0139a12d696e439ee5f Fri Sep 11 06:06:55 2026 -0700
new hg38 track hprcRdt: reference-divergent transcripts from 206 HPRC Release 2 genomes
Added as a third child of the existing long_read_transcripts superTrack,
alpha only for now. Data from Max Marin (DFCI), a bigPsl of RDT cluster
representative sequences aligned to GRCh38: 180,464 alignments of 120,451
distinct sequences from 412 haplotypes. Rebuilt from the submitted file only
to add a name index, refs #33822
- src/hg/makeDb/trackDb/human/hg38/methBaseAsm.html
- lines changed 7, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- lines changed 1, context: html, text, full: html, text
d033cea2063e9362949baf5b4d8b837597173a0d Thu Sep 10 05:16:05 2026 -0700
Address the code review of the Sep 9 commits
Faceted composite: text that comes from a hub - a metadata column's
description, the name and title of a data type, and the values quoted back in
the "could not load the metadata" row - is put on the page as text rather than
as markup. The three places built their markup from template strings, so a
value carrying angle brackets or a quote was read as HTML: the column
description now goes through the shared htmlEncode() once where the header is
parsed, and the other two build their elements as nodes. The error row reads
better for it as well, since a value with brackets in it used to disappear from
the message that was meant to show it.
The saved UI state keys on the assembly as well as the metadata id. localStorage
is per-origin, so two assemblies whose tracks share a name were sharing one
entry, and a row order dragged on one came back on the other over a different
set of samples. hgTrackUi passes the database down for it. State saved under the
old key is dropped, which costs a facet selection or a page length.
Imprinting: the five subtrack description pages link back to the container as
hgTrackUi?db=$db&g=$parentTrack, without the hgsid. Native trackDb html is
substituted by hgTrackDb as it loads the table, where there is no cart, so
${hgsid} came out empty and the link read 'hgsid=&g=...'. Matches what the
Fiber-seq pages already do. The makeDoc note that described the old form is
updated with the reason.
UniProt otto: README.txt lists all eight things that reach runLog.txt. It had
four, and was missing LOCKED, along with PREFLIGHT-FAIL, END and INTERRUPTED.
refs #36210
refs #37599
refs #38300
- src/hg/makeDb/trackDb/human/hg38/methbase2.ra
- lines changed 1, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- src/hg/makeDb/trackDb/human/hg38/omimImprint.html
- lines changed 21, context: html, text, full: html, text
691a2b8981d6db69e8707ea44041c4661cdac97e Wed Sep 9 06:38:29 2026 -0700
Imprinting: add the ASM Atlas tracks, and tidy the collection's labels
Adds a composite built from Rosenski et al. 2025, "Atlas of imprinted and
allele-specific DNA methylation in the human body". Three subtracks: the
458 regions whose methylation follows the parent of origin, the 72 known
control regions with the boundaries the paper redrew, and the pool of
385,235 regions carrying two methylation states that those came out of.
A fourth set, the regions whose methylation follows a nearby SNP, is
built by the scripts but its stanza is commented out, since sequence
driven methylation is not imprinting.
The authors released hg19 only, so all three are lifted. Their published
files are close to bare BED, so the SNPs, cell types, p-values, gene
links and gamete methylation on the details pages are read out of the
paper's supplementary tables and joined on by position. Regions that
lift but change length by more than 10%, because hg38 added sequence
inside them, are kept with a note rather than dropped: one of them is
TCEB3C, the only control region on chr18.
Also across the collection:
- long labels name their source right after "Imprinting", so that a
label read on its own says where the data came from
- the two gene catalogs are worded alike, and ordered OMIM, Geneimprint,
MethBase2, Akbari, ASM Atlas
- the OMIM curators confirmed that their (I) marker covers established
and candidate imprinted genes alike, with nothing in the export to
tell them apart. Labels, description page and makeDoc now say so, and
the claim that the set is "more conservative" than the computational
tracks is gone. The bigBed was rebuilt for the autoSql line, same 459
features.
- every subtrack page opens by naming the collection, linked back to
its hgTrackUi page, and no longer repeats the collection page's
introduction to imprinting
refs #37599
- lines changed 1, context: html, text, full: html, text
d033cea2063e9362949baf5b4d8b837597173a0d Thu Sep 10 05:16:05 2026 -0700
Address the code review of the Sep 9 commits
Faceted composite: text that comes from a hub - a metadata column's
description, the name and title of a data type, and the values quoted back in
the "could not load the metadata" row - is put on the page as text rather than
as markup. The three places built their markup from template strings, so a
value carrying angle brackets or a quote was read as HTML: the column
description now goes through the shared htmlEncode() once where the header is
parsed, and the other two build their elements as nodes. The error row reads
better for it as well, since a value with brackets in it used to disappear from
the message that was meant to show it.
The saved UI state keys on the assembly as well as the metadata id. localStorage
is per-origin, so two assemblies whose tracks share a name were sharing one
entry, and a row order dragged on one came back on the other over a different
set of samples. hgTrackUi passes the database down for it. State saved under the
old key is dropped, which costs a facet selection or a page length.
Imprinting: the five subtrack description pages link back to the container as
hgTrackUi?db=$db&g=$parentTrack, without the hgsid. Native trackDb html is
substituted by hgTrackDb as it loads the table, where there is no cart, so
${hgsid} came out empty and the link read 'hgsid=&g=...'. Matches what the
Fiber-seq pages already do. The makeDoc note that described the old form is
updated with the reason.
UniProt otto: README.txt lists all eight things that reach runLog.txt. It had
four, and was missing LOCKED, along with PREFLIGHT-FAIL, END and INTERRUPTED.
refs #36210
refs #37599
refs #38300
- src/hg/makeDb/trackDb/human/hg38/trackDb.ra
- lines changed 2, context: html, text, full: html, text
ce780dd2f1216ce728ab6bb69ac19a39ddc694fd Tue Sep 8 00:26:39 2026 -0700
hg38: Fiber-seq container with accessibility, FIRE peaks and CpG methylation, 41 samples
Native version of the Stergachis/Vollger lab hub at
https://fiberseq.github.io/UCSC-Fiber-seq-hub/hub.txt, plus the per-sample CpG
methylation Shane Neph asked to have alongside it. Both cover the same 41
samples: 14 cell lines and 27 lymphoblastoid lines from HPRC and GIAB
individuals.
fiberSeq container, group regulation
fiberSeqAcc multiWig overlay of 7 common cell lines, on by default
fiberSeqCompendium faceted composite, dataTypes acc/peaks/hap
fiberSeqMeth faceted composite, dataTypes comb/hap/diffs, "Methylation"
Both composites use the Methbase faceted-composite machinery. Subtracks are
named <composite>_<accession>_<dataType> with the accession as the only middle
component, because facetedCompositeUi() cuts the data element at the first
underscore and cartDump.c reassembles the name from the pieces; the hub's
<composite>_<sample>_<accession>_<type> names would have resolved to tracks that
do not exist. Sample name and cell type live in the metadata TSV instead. Using
dataTypes also brings onlyVisibility, which is what lets the peaks default to
dense while the signal tracks default to full, the mixed-visibility default
Andrew Stergachis asked for.
397 GB mirrored from the UW Kopah S3 server rather than pointed at over the
network, since a native track should not depend on it.
The FIRE peak bigBeds had to be rebuilt: they carry full narrowPeak data but
their header records a field count of 3, which hides signalValue and qValue
from the browser and would have made hgTracks errAbort in
bigNarrowPeakLoadItems(). The rebuild fixes the header and rounds the two float
columns to 3 decimals, 467 MB to 313 MB. It drops 421 of 9,487,043 peaks called
on chrEBV, the EBV decoy of the GRCh38 analysis set, which hg38 does not have;
9,486,622 remain and every sample reconciles exactly. Reported upstream, along
with GM12878's two haplotype accessibility bigWigs, which are one-base
placeholders at the source.
refs #36210
- lines changed 2, context: html, text, full: html, text
2c0adaa48b2f9a14109c3f90713405f259d920bc Wed Sep 9 06:08:46 2026 -0700
hg38: pcLAI local ancestry track for HPRC Release 2 haplotypes
Point cloud local ancestry inference (pcLAI) for HPRC Release 2, projected
onto GRCh38: a composite with one subtrack per haplotype, both haplotypes of
231 samples plus CHM13, 463 in all. 11,936,603 windows, autosomes only, no
windows dropped from the source files.
Each window carries the (PC1,PC2) coordinate pcLAI predicts for it, the
discretized ancestry centroid, and a confidence score. The details page draws
the window's position against the 1000 Genomes reference panel that defines
the space, via detailsScript/scatterPlot; metaDataUrl is what lets hgTrackUi
serve that panel file for a native (non-hub) track.
thickStart is one base before chromStart in 54,811 of the windows (0.46%),
against the format the pcLAI authors document, so bedToBigBed rejects it.
Neither thick column carries information here, so both are set to the item
bounds rather than dropping those windows.
Testing only for now, alpha, no ticket yet; a ticket may follow if this
becomes a real track. The makeDoc carries the detail in the meantime,
including what was deliberately left undone.
- src/hg/makeDb/trackDb/human/predictionScoresSuper.html
- lines changed 2, context: html, text, full: html, text
81bd65b4c221b9288848d61f75b3c69197e9d7fd Mon Sep 7 11:28:21 2026 -0700
removing alpha genome text temporarily as per google group refs #38261
- lines changed 5, context: html, text, full: html, text
3dd0329ee0e804f5841a9998c4e7f024d3276921 Wed Sep 9 17:20:20 2026 -0700
Releasing the AlphaGenome Variant Impact (AVI) score track for hg38. refs #38261
Moves the bigWigs to /gbdb/hg38/_alphaGenome/, the underscore convention that keeps
non-redistributable data off the download server, matching PromoterAI and PrimateAI,
and drops the alpha release tag.
Shortens the composite longLabel to fit the 85 character limit and lowercases Score
in the longLabels. Adds a New pennantIcon and points the Deleteriousness Predictions
container pennant at the Sept. 10 news post.
Corrects the median on the description page from 1.6 to 2.9. The old figure counted
the zero-filled reference base slots, which are not variants. Removes two threshold
statements that are not in the AlphaGenome Atlas preprint or any public source. Adds
the /gbdb symlink step to the makeDoc, and fixes three container page links that were
missing target=_blank.
- src/hg/makeDb/trackDb/human/predictionScoresSuper.ra
- lines changed 2, context: html, text, full: html, text
3dd0329ee0e804f5841a9998c4e7f024d3276921 Wed Sep 9 17:20:20 2026 -0700
Releasing the AlphaGenome Variant Impact (AVI) score track for hg38. refs #38261
Moves the bigWigs to /gbdb/hg38/_alphaGenome/, the underscore convention that keeps
non-redistributable data off the download server, matching PromoterAI and PrimateAI,
and drops the alpha release tag.
Shortens the composite longLabel to fit the 85 character limit and lowercases Score
in the longLabels. Adds a New pennantIcon and points the Deleteriousness Predictions
container pennant at the Sept. 10 news post.
Corrects the median on the description page from 1.6 to 2.9. The old figure counted
the zero-filled reference base slots, which are not variants. Removes two threshold
statements that are not in the AlphaGenome Atlas preprint or any public source. Adds
the /gbdb symlink step to the makeDoc, and fixes three container page links that were
missing target=_blank.
- src/hg/makeDb/trackDb/human/refSeqComposite.html
- lines changed 6, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- src/hg/makeDb/trackDb/human/trackDb.ra
- lines changed 1, context: html, text, full: html, text
74bea0582059450e508b0f69a6b70c072ac1d3a2 Wed Sep 9 16:29:28 2026 -0700
List the newest deprecated transcript first for a deprecated NP_ search, per CR feedback.
When a superseded protein accession maps to several deprecated transcript
versions, the search returns all of them, but they were listed in table order so
the oldest came first. Order the xrefQuery newest-first, matching the choice the
HGVS protein path already makes. Sort by length before value so .10 beats .9
rather than losing a string comparison. refs #38292
- src/hg/makeDb/trackDb/mouse/mm10/mouseStrainsCactus.html
- lines changed 236, context: html, text, full: html, text
efebc8a0a29aeef60bc470a40ced7a2aa6652efd Tue Sep 8 19:25:02 2026 -0700
Adding native mm10 track for the mouse strains Cactus alignment. refs #38308
New alpha-gated track mouseStrainsCactus exposing the Progressive Cactus
alignment of the 16 Mouse Genomes Project strain assemblies plus rat, which
until now was only reachable by attaching the mouseStrains assembly hub.
bigDataUrl, summary and frames point at the existing bigMaf files on
hgdownload rather than copying 8.8 GB into /gbdb, the same way the hg38
cactus241wayBM track is served.
Polish over the hub stanza: renamed from the generic "bigMaf", off by
default, speciesGroups splitting the strains into wild-derived, classical
laboratory and Rat/rn6, speciesLabels so side labels read 129S1/SvImJ
rather than 129S1_SvImJ, plus treeImage and speciesCodonDefault. The three
new sGroup_ tags are registered in tagTypes.tab.
Description page written from Lilue et al. 2018; the hub page had an empty
Description section and its Display Convention text was wigMaf boilerplate
that did not match this track. Also notes that the alignment is a poor
source for large rearrangements, since Ragout built the strain
pseudo-chromosomes against the reference and discarded most adjacencies
that disagreed with it.
Added a reciprocal relatedTracks.ra pair between this track and
mm10Strains1 ("Alternate strains"), since #38227 came in from a user who
kept landing on mm10Strains1 while looking for this alignment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/makeDb/trackDb/mouse/mm10/mouseStrainsCactus.ra
- lines changed 20, context: html, text, full: html, text
efebc8a0a29aeef60bc470a40ced7a2aa6652efd Tue Sep 8 19:25:02 2026 -0700
Adding native mm10 track for the mouse strains Cactus alignment. refs #38308
New alpha-gated track mouseStrainsCactus exposing the Progressive Cactus
alignment of the 16 Mouse Genomes Project strain assemblies plus rat, which
until now was only reachable by attaching the mouseStrains assembly hub.
bigDataUrl, summary and frames point at the existing bigMaf files on
hgdownload rather than copying 8.8 GB into /gbdb, the same way the hg38
cactus241wayBM track is served.
Polish over the hub stanza: renamed from the generic "bigMaf", off by
default, speciesGroups splitting the strains into wild-derived, classical
laboratory and Rat/rn6, speciesLabels so side labels read 129S1/SvImJ
rather than 129S1_SvImJ, plus treeImage and speciesCodonDefault. The three
new sGroup_ tags are registered in tagTypes.tab.
Description page written from Lilue et al. 2018; the hub page had an empty
Description section and its Display Convention text was wigMaf boilerplate
that did not match this track. Also notes that the alignment is a poor
source for large rearrangements, since Ragout built the strain
pseudo-chromosomes against the reference and discarded most adjacencies
that disagreed with it.
Added a reciprocal relatedTracks.ra pair between this track and
mm10Strains1 ("Alternate strains"), since #38227 came in from a user who
kept landing on mm10Strains1 while looking for this alignment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/makeDb/trackDb/mouse/mm10/refSeqComposite.html
- lines changed 6, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- src/hg/makeDb/trackDb/mouse/mm10/trackDb.ra
- lines changed 1, context: html, text, full: html, text
efebc8a0a29aeef60bc470a40ced7a2aa6652efd Tue Sep 8 19:25:02 2026 -0700
Adding native mm10 track for the mouse strains Cactus alignment. refs #38308
New alpha-gated track mouseStrainsCactus exposing the Progressive Cactus
alignment of the 16 Mouse Genomes Project strain assemblies plus rat, which
until now was only reachable by attaching the mouseStrains assembly hub.
bigDataUrl, summary and frames point at the existing bigMaf files on
hgdownload rather than copying 8.8 GB into /gbdb, the same way the hg38
cactus241wayBM track is served.
Polish over the hub stanza: renamed from the generic "bigMaf", off by
default, speciesGroups splitting the strains into wild-derived, classical
laboratory and Rat/rn6, speciesLabels so side labels read 129S1/SvImJ
rather than 129S1_SvImJ, plus treeImage and speciesCodonDefault. The three
new sGroup_ tags are registered in tagTypes.tab.
Description page written from Lilue et al. 2018; the hub page had an empty
Description section and its Display Convention text was wigMaf boilerplate
that did not match this track. Also notes that the alignment is a poor
source for large rearrangements, since Ragout built the strain
pseudo-chromosomes against the reference and discarded most adjacencies
that disagreed with it.
Added a reciprocal relatedTracks.ra pair between this track and
mm10Strains1 ("Alternate strains"), since #38227 came in from a user who
kept landing on mm10Strains1 while looking for this alignment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/makeDb/trackDb/mouse/mm39/refSeqComposite.html
- lines changed 6, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- src/hg/makeDb/trackDb/refSeqComposite.html
- lines changed 6, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- src/hg/makeDb/trackDb/relatedTracks.ra
- lines changed 20, context: html, text, full: html, text
6a29bc1077e1bd2926fa1508c6fdde383397f4a4 Tue Sep 8 16:14:56 2026 -0700
relatedTracks.ra: drop 20 entries naming tracks that are not on the RR
An audit of the table against the public trackDb found 77 relationship rows
pointing at a track the RR does not have. Most are fine and stay: the track is
either gated behind an alpha include and still in development (LCRs, gerp,
imprinting, mei, srSv, t2tChain, tads, TOGAv2, clinvarMapped, colorsDbLegacy,
hg38Patch11, singleCellSignalsPeaks, cancerMutations), or the family is already
public on other assemblies and only this version or assembly has yet to be
pushed (transMapV6, crispr10K, tanDups, ukbDepletion). Those entries start
working on their own when the track ships.
The 20 lines removed here name tracks with no public counterpart under any name
on any assembly, so they could never render for a reader on the RR:
ucscRetroAli8, superseded by the V9 track that is public
wgRnaOld and cancerMutations, both release alpha
transMapV4, absent from the RR entirely, so the source page does not exist
ensGene on hg38 and mm10, where Ensembl Genes was retired
sibAltEvents, public on no assembly
chainNetHs1, which is assembly-hub machinery rather than a native hg38 track
cloneEndUcsc, a mapping that only ever existed on hgwdev
The three crispr lines came out for a different reason. On hgwdev crispr is a
superTrack over crisprRanges and crisprTargets while the RR carries only the
standalone crisprAllTargets, so the entries looked like they wanted repointing.
The file already relates crisprAllTargets and crispr10K reciprocally further
down, on all three assemblies, which made them redundant instead.
Every name still in the file resolves in the hgwdev trackDb, and no duplicate
assembly and track pair remains. Rebuilt with make update on hg38, hg19 and
mm10: 24 rows dropped, none added, nothing else changed.
The original audit checked names against alpha, which is why these got in.
refs #38016
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 4, context: html, text, full: html, text
efebc8a0a29aeef60bc470a40ced7a2aa6652efd Tue Sep 8 19:25:02 2026 -0700
Adding native mm10 track for the mouse strains Cactus alignment. refs #38308
New alpha-gated track mouseStrainsCactus exposing the Progressive Cactus
alignment of the 16 Mouse Genomes Project strain assemblies plus rat, which
until now was only reachable by attaching the mouseStrains assembly hub.
bigDataUrl, summary and frames point at the existing bigMaf files on
hgdownload rather than copying 8.8 GB into /gbdb, the same way the hg38
cactus241wayBM track is served.
Polish over the hub stanza: renamed from the generic "bigMaf", off by
default, speciesGroups splitting the strains into wild-derived, classical
laboratory and Rat/rn6, speciesLabels so side labels read 129S1/SvImJ
rather than 129S1_SvImJ, plus treeImage and speciesCodonDefault. The three
new sGroup_ tags are registered in tagTypes.tab.
Description page written from Lilue et al. 2018; the hub page had an empty
Description section and its Display Convention text was wigMaf boilerplate
that did not match this track. Also notes that the alignment is a poor
source for large rearrangements, since Ragout built the strain
pseudo-chromosomes against the reference and discarded most adjacencies
that disagreed with it.
Added a reciprocal relatedTracks.ra pair between this track and
mm10Strains1 ("Alternate strains"), since #38227 came in from a user who
kept landing on mm10Strains1 while looking for this alignment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 18, context: html, text, full: html, text
4624f2c72c429fd9b32532bb4d9b62f030b50182 Wed Sep 9 06:00:44 2026 -0700
trackDb: cross-link the Deleteriousness Predictions container with the CADD 1.6, CADD 1.7, REVEL and AlphaMissense tracks on hg38 and hg19, refs #38261
- src/hg/makeDb/trackDb/tagTypes.tab
- lines changed 6, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 11, context: html, text, full: html, text
96a903979dbe723e4f20a700dfc12881063b2c46 Tue Sep 8 00:26:18 2026 -0700
hgTracks: support mouseOver on bigNarrowPeak tracks, and register the peak filter tags
bigNarrowPeakLoadItems() had its own load loop and never looked at the
mouseOver setting, so a bigNarrowPeak track silently ignored it. It now uses
the mouseOverSetupForBbi() / mouseOverGetBbiText() helpers in mouseOver.c, which
also gets mouseOverField support for free. As with every other bigBed-like
track, the text only shows in pack or full, since dense makes no per-item map
boxes.
tagTypes.tab did not list bigNarrowPeak for mouseOver, scoreFilter,
scoreFilterLimits, scoreMin, scoreMax, signalFilter or signalFilterLimits, so
tdbQuery -strict rejected all of them even though the code reads them.
pValueFilter and qValueFilter, with their Limits, were not registered for any
type at all, although encodePeakCfgUi() in hui.c has always drawn them and
bigNarrowPeakLoadItems() has always applied them. Added.
refs #36210
- lines changed 3, context: html, text, full: html, text
efebc8a0a29aeef60bc470a40ced7a2aa6652efd Tue Sep 8 19:25:02 2026 -0700
Adding native mm10 track for the mouse strains Cactus alignment. refs #38308
New alpha-gated track mouseStrainsCactus exposing the Progressive Cactus
alignment of the 16 Mouse Genomes Project strain assemblies plus rat, which
until now was only reachable by attaching the mouseStrains assembly hub.
bigDataUrl, summary and frames point at the existing bigMaf files on
hgdownload rather than copying 8.8 GB into /gbdb, the same way the hg38
cactus241wayBM track is served.
Polish over the hub stanza: renamed from the generic "bigMaf", off by
default, speciesGroups splitting the strains into wild-derived, classical
laboratory and Rat/rn6, speciesLabels so side labels read 129S1/SvImJ
rather than 129S1_SvImJ, plus treeImage and speciesCodonDefault. The three
new sGroup_ tags are registered in tagTypes.tab.
Description page written from Lilue et al. 2018; the hub page had an empty
Description section and its Display Convention text was wigMaf boilerplate
that did not match this track. Also notes that the alignment is a poor
source for large rearrangements, since Ragout built the strain
pseudo-chromosomes against the reference and discarded most adjacencies
that disagreed with it.
Added a reciprocal relatedTracks.ra pair between this track and
mm10Strains1 ("Alternate strains"), since #38227 came in from a user who
kept landing on mm10Strains1 while looking for this alignment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 1, context: html, text, full: html, text
444b1eb7e7ec2938a4a9a6d3ed214179073ab6f7 Wed Sep 9 05:41:23 2026 -0700
Faceted composite: manual row reordering, group-by, saved UI state, and per-facet "only" links
The Fiber-seq compendium put 41 samples times six data types into one
faceted composite, which pushed on the parts of the page that were built
for a flat list of tracks. Changes here, all in the shared faceted
composite code rather than anything Fiber-seq specific:
Row order. Track order in the image follows the table, so the table now
lets you set that order by hand. Vendored DataTables RowReorder 1.5.1
adds a drag handle as the first column after the checkbox, enabled on
the "shown in the browser" tab where reordering means something. The
dragged order is remembered by sample name rather than by row number, so
it survives a metadata file whose contents have changed.
Group by. A container of six data types can be read two ways, so the
page offers both: group the image by sample, keeping a sample's six
tracks together, or by data type, putting all the accessibility tracks
next to each other. cartDump assigns the priorities and just swaps the
nesting of its two loops. trackDb sets the starting choice with
defaultGroupBy.
Saved state. Facets, per-column searches, sort column, page length,
which tab was open and the hand-dragged order go to localStorage keyed
by metadata id, so coming back to the page does not mean setting it all
up again.
Facet "only" links. A small "only" appears on hover behind each facet
value and narrows to just that one, instead of unticking the others by
hand.
Column descriptions. A metadata column heading can now carry a longer
explanation after a "|", shown behind an info icon on both the column
header and the facet heading.
Also: parseDataTypes() was returning its list reversed, since slPairAdd
prepends and nothing put it back, so the data type checkboxes and the
resulting subtrack order were backwards; the composite lifts itself out
of hide when the user touches anything on the page, which is what they
meant by touching it; the facet sidebar collapses when a table has no
facetable columns; and the label wording throughout says "samples" and
"in the browser" rather than "tracks" and "active".
The Methbase hg38 track gets labels for its three data types, which were
showing as the bare pipeline names hmr, levels and reads.
refs #36210
- lines changed 1, context: html, text, full: html, text
9d9210bb7b34131505ab50b5e62bb88680dc6129 Wed Sep 9 15:14:09 2026 -0700
trackDb docs: add vcfPhasedColorBy, refs #38010
vcfPhasedColorBy has been read by vcfUi.c since the trio display went in, but it
was documented nowhere: not on the VCF help pages, not in trackDbLibrary, and
tdbQuery -check would have rejected it because tagTypes.tab did not list it
either. So a hub author had no way to find the setting and no way to use it
without tripping the checker.
Add the library blurb, the rows in trackDbDoc.html and trackDbHub.v3.html, a
changes.html entry, and the tagTypes.tab registration. The blurb spells out that
mendelDiff needs vcfParentSamples and that function is only offered when
geneTrack is set, since both conditions are enforced in vcfUi.c and neither is
obvious from the value name.
Companion to the two commits documenting the same settings on vcf.html and
hgVcfTrackHelp.html.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 2, context: html, text, full: html, text
a8377d52dc28dbd8a424bf1536009d72359c8cb1 Wed Sep 9 15:25:08 2026 -0700
trackDb: register minAc and vcfDoMinAc in tagTypes.tab, refs #38010
Both are read from trackDb but neither was listed, so tdbQuery -check would
reject a stanza that used them.
minAc is not just a UI default: vcfTrack.c:98 reads it, minAcFail() at :190
drops records whose INFO AC is below it, and filterRecords applies that on all
three code paths, vcfPhasedLoadItems included. vcfDoMinAc gates the matching
control in vcfCfgUi the same way its three siblings do.
Registered for vcf, vcfTabix and vcfPhasedTrio, since filterRecords runs for the
trio type too (vcfTrack.c:2344). Note minFreq and minQual are still listed as
vcf and vcfTabix only, although filterRecords treats them the same way; left
alone rather than widened here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/mouseStuff/whyConserved/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/nci60/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/oneShot/alphaGenomeToWig/alphaGenomeToWig.c
- lines changed 19, context: html, text, full: html, text
5a249cd50f592a3b7598110eec792cfc942fab3f Wed Sep 9 08:52:06 2026 -0700
Address the v504 code review
hgSession: an anonymous share name that arrives with the request is saved only
when it is not already in the table. Every anonymous link sits under the one
reserved user "l", so a name already there stays as it is and the caller is told
so. The top-right Share dialog is unaffected, since it passes a name it has
just reserved and such a name does not exist yet. Snapshot names are now left
out of both My Sessions listings, which is what their "__" prefix has claimed
all along.
Share dialog: "Create link & copy" reports the copy instead of promising it.
copyToClipboard says whether the text reached the clipboard, the dialog passes
that on when a browser refuses, and it tries the asynchronous clipboard API
before giving up. The preview is built with the same encoding the server uses,
so a name holding a hyphen or a slash previews as the link that really gets
made. Cancelling out of the name editor no longer copies a second time, and a
reply with no link in it says so rather than showing "undefined".
hgBlat: a second click on the share button while the first request is still out
no longer mints a second snapshot session, and a box dismissed during the wait
stays closed.
Also: a snapshot moves a cart value into durable storage only when it is a
trash path, the way sessionData's own callers check; sqlAddressMatch keeps to
its own documented precondition when handed an empty address; alphaGenomeToWig
compares its output with its input rather than with itself, rejects a position
that is not all digits and skips an empty score; and hgc's default iframe width
reaches the browser as one percent sign.
refs #38294
- src/hg/oneShot/testCart/cart.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/orthoMap/makefile
- lines changed 2, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/phyloPng/phyloPng.c
- lines changed 15, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/pslCDnaFilter/makefile
- lines changed 3, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/pslDiff/makefile
- lines changed 2, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/sageVisCGI/sageVisCGI.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/trfBig/trfBig.c
- lines changed 6, context: html, text, full: html, text
73625c7a1d0620a7e6e9de61a2b4fd2fd1fa7685 Wed Sep 9 16:08:15 2026 -0700
adjust trfBig to use 500 kb windows for calculation and corresponding script from claude to reassembly broken annotations at this smaller window size refs #38321
- src/hg/useCount/useCount.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/utils/automation/asmHubGc5Percent.pl
- lines changed 78, context: html, text, full: html, text
8f030ca5d3d1ff1983acb8bcb0ea1de7fe56e3a5 Fri Sep 11 16:06:50 2026 -0700
correctly lead users to download or generate GC data from bigWigs or 2bits refs #38290
- lines changed 1, context: html, text, full: html, text
cbf5cf161c1be2f6bfdcee0a29f334564ac9435e Fri Sep 11 16:11:38 2026 -0700
silence claude noise refs #38290
- src/hg/utils/automation/asmHubTrackDb.sh
- lines changed 2, context: html, text, full: html, text
646f392ab8ff4d3ca9740c199ecb59317eccae78 Fri Sep 11 16:13:11 2026 -0700
silence claude noise refs #38290
- src/hg/utils/automation/doAssemblyHub.pl
- lines changed 5, context: html, text, full: html, text
d40445e1154b5384876ca6b69bd069518762ea6d Thu Sep 10 21:52:57 2026 -0700
working with new simpleRepeat output refs #38321
- src/hg/utils/automation/doRepeatMasker.pl
- lines changed 6, context: html, text, full: html, text
af19b68a0dc2b8e4057fe762f1805b060e38ed77 Thu Sep 10 21:51:33 2026 -0700
updating RepeatMasker and RepeatModeler to next version refs #38337
- src/hg/utils/automation/doRepeatModeler.pl
- lines changed 1, context: html, text, full: html, text
af19b68a0dc2b8e4057fe762f1805b060e38ed77 Thu Sep 10 21:51:33 2026 -0700
updating RepeatMasker and RepeatModeler to next version refs #38337
- src/hg/utils/automation/doSimpleRepeat.pl
- lines changed 3, context: html, text, full: html, text
3de452af33eb095927dc7305001f950b81f05693 Wed Sep 9 16:13:30 2026 -0700
adjust trfBig to use 500 kb windows for calculation and corresponding script from claude to reassembly broken annotations at this smaller window size refs #38321
- lines changed 17, context: html, text, full: html, text
f271ed112e3a9facaebc08ec6d05e463790988e0 Wed Sep 9 16:34:19 2026 -0700
now using the merged trf output from the 500 kb operation refs #38321
- lines changed 3, context: html, text, full: html, text
6359c2dd09c8520e2a03b14d916f1f23deac63ed Fri Sep 11 09:55:28 2026 -0700
correct reference to the mergeTrf script refs #38321
- src/hg/utils/automation/mergeTrf.py
- lines changed 270, context: html, text, full: html, text
73625c7a1d0620a7e6e9de61a2b4fd2fd1fa7685 Wed Sep 9 16:08:15 2026 -0700
adjust trfBig to use 500 kb windows for calculation and corresponding script from claude to reassembly broken annotations at this smaller window size refs #38321
- src/hg/utils/cartTrackVarCatalog/cartTrackVarCatalog.py
- lines changed 14, context: html, text, full: html, text
91259a696e598fb4b6807cda9b40eb72971e9209 Sat Sep 12 06:59:18 2026 -0700
cartTrackVarCatalog: catalog the faceted composite's groupBy, refs #37838
<track>.groupBy says which of a faceted composite's two dimensions is kept
together in the image: "sample" puts one sample's data types side by side,
"dataType" puts the same data type for every sample side by side. Written by
cartDump from facetedComposite.js and read back in hgTrackUi, both of which
store it only if it is one of those two words, so nothing unvalidated reaches
the <script> block. trackDb supplies the starting choice with defaultGroupBy
and the cart value wins over it. Added under #36210.
- lines changed 117, context: html, text, full: html, text
a3df9f62a5995302b5a07ce5e3dd0eadda3a7768 Sat Sep 12 09:59:01 2026 -0700
cartTrackVarCatalog: describe hgTables' own cart variables, refs #37979
With peel() fixed, 2,331 hgta_ names in the saved sessions were left honestly
uncatalogued rather than absorbed by a catch-all. They fall into four groups,
each read at its call site before a row was written for it.
The two linked-table checkboxes, hgta_fs.linked.<db>.<table> and
hgta_fil.linked.<db>.<table>, which offer a joinable table's fields on the
Select Fields and filter pages. extraTableList finds the checked tables by
scanning the cart for the prefix, so the set is whatever the cart holds.
The filter ops. The catalog covered .pat alone; hgTables.h defines six, and
all six are in live sessions. pat, dd and cmp belong to one field, while
rawLogic, rawQuery and maxOutput apply to the whole table and still carry a
field slot in the name, filled with an empty string or a bare _.
Fifty-one session-scoped variables: intersection, correlation, subtrack merge,
identifiers, user regions, output naming, MAF output, and which table the
Select Fields, filter and histogram pages are about. The header itself
documents the convention that shapes half of them - the pages with a Cancel
button hold their state twice, hgta_<var> in force and hgta_next<Var> proposed,
copied one way on open and the other on Submit - so that is in the group's
description rather than in every note.
Renaming the .pat row to .<op> also removed an accidental cover: a row
registers its trailing component, and that "pat" had been standing in for
gvfTrack.c's %s_pat, which is an item label rather than a cart variable. Its
seven siblings were already in the baseline, so pat joins them there.
hgta_ names matched only by a catch-all go from 4,299 to 3. The three left are
hgta_identifierFile and hgta_userRegionsFile, described in the #37623
file-variable registry that this audit does not read, and hgta_userRegionsTable,
which nothing in the tree reads at all.
- src/hg/utils/cartTrackVarCatalog/cartVarsNotCataloged.txt
- lines changed 1, context: html, text, full: html, text
a3df9f62a5995302b5a07ce5e3dd0eadda3a7768 Sat Sep 12 09:59:01 2026 -0700
cartTrackVarCatalog: describe hgTables' own cart variables, refs #37979
With peel() fixed, 2,331 hgta_ names in the saved sessions were left honestly
uncatalogued rather than absorbed by a catch-all. They fall into four groups,
each read at its call site before a row was written for it.
The two linked-table checkboxes, hgta_fs.linked.<db>.<table> and
hgta_fil.linked.<db>.<table>, which offer a joinable table's fields on the
Select Fields and filter pages. extraTableList finds the checked tables by
scanning the cart for the prefix, so the set is whatever the cart holds.
The filter ops. The catalog covered .pat alone; hgTables.h defines six, and
all six are in live sessions. pat, dd and cmp belong to one field, while
rawLogic, rawQuery and maxOutput apply to the whole table and still carry a
field slot in the name, filled with an empty string or a bare _.
Fifty-one session-scoped variables: intersection, correlation, subtrack merge,
identifiers, user regions, output naming, MAF output, and which table the
Select Fields, filter and histogram pages are about. The header itself
documents the convention that shapes half of them - the pages with a Cancel
button hold their state twice, hgta_<var> in force and hgta_next<Var> proposed,
copied one way on open and the other on Submit - so that is in the group's
description rather than in every note.
Renaming the .pat row to .<op> also removed an accidental cover: a row
registers its trailing component, and that "pat" had been standing in for
gvfTrack.c's %s_pat, which is an item label rather than a cart variable. Its
seven siblings were already in the baseline, so pat joins them there.
hgta_ names matched only by a catch-all go from 4,299 to 3. The three left are
hgta_identifierFile and hgta_userRegionsFile, described in the #37623
file-variable registry that this audit does not read, and hgta_userRegionsTable,
which nothing in the tree reads at all.
- src/hg/utils/docent/README.md
- lines changed 27, context: html, text, full: html, text
c4bcca06ef1a06c434c9136a79459f1512cd0606 Tue Sep 8 07:41:02 2026 -0700
docent: expect: can assert the color a track's row was drawn in, and a test for #36212
A bug about color leaves the page identical -- same rows, same height, same item
names, same tooltips -- so every check expect: had was blind to it. `color:` reads
the pixels instead: it names the color a row is mostly drawn in (`is:`), or one it
must not be (`not:`), with `part: label` for the center label rather than the
items, `at:`/`frac:`/`x:` for one item rather than the whole row, and a list form
so one step can state a whole color matrix and a failure name every row that came
out wrong.
hgTracks draws the whole view into one png and shows each row as a CSS-offset
slice of it, so a row's pixels are that slice drawn into a canvas at its offset.
The clipping box is the img's own div.sliceDiv, not the table cell: the center
label and the data are two slices inside one td_data_<key>, and measuring the cell
runs the canvas past the end of this row and into the next track's, which reads
that track's color as part of this one. The side labels are a separate png and are
never included, since "what color is this row" must not be answered by the label
text. White is background; everything else counts, black included, because a track
with no color of its own draws black items. is:/not: take r,g,b or #rrggbb and no
CSS color names, because trackDb's `color 0,255,0` is not CSS green.
tests/colorchecks covers all of it and tests/colorchecks.xfail aims all six forms
wrong in one step, so the failure has to name all six.
tests/regress/rm36212.xfail is the bug it was written for: a stanza that sets both
`itemRgb on` and `color` draws its items in the color setting instead of in the
file's own RGB column, because bedItemRgb() (hg/cgilib/bedCart.c) tests for the
presence of `color` before it tests for an explicit `itemRgb on`, so the explicit
setting is never reached. It is an xfail because the bug is live on the RR, on beta
and on genome-test.
It is also the first script in that directory that has been watched both to fail on
a build with the bug and to pass on a build with the fix -- the three-line reorder
built into parked #36212 and the same script pointed at that port. The fixture is
~/public_html/docentFixtures/itemRgbHub/, four tracks over four copies of one bed9
file whose items all carry a pure blue itemRgb column.
refs #36212, refs #37892
- src/hg/utils/docent/docent.js
- lines changed 139, context: html, text, full: html, text
c4bcca06ef1a06c434c9136a79459f1512cd0606 Tue Sep 8 07:41:02 2026 -0700
docent: expect: can assert the color a track's row was drawn in, and a test for #36212
A bug about color leaves the page identical -- same rows, same height, same item
names, same tooltips -- so every check expect: had was blind to it. `color:` reads
the pixels instead: it names the color a row is mostly drawn in (`is:`), or one it
must not be (`not:`), with `part: label` for the center label rather than the
items, `at:`/`frac:`/`x:` for one item rather than the whole row, and a list form
so one step can state a whole color matrix and a failure name every row that came
out wrong.
hgTracks draws the whole view into one png and shows each row as a CSS-offset
slice of it, so a row's pixels are that slice drawn into a canvas at its offset.
The clipping box is the img's own div.sliceDiv, not the table cell: the center
label and the data are two slices inside one td_data_<key>, and measuring the cell
runs the canvas past the end of this row and into the next track's, which reads
that track's color as part of this one. The side labels are a separate png and are
never included, since "what color is this row" must not be answered by the label
text. White is background; everything else counts, black included, because a track
with no color of its own draws black items. is:/not: take r,g,b or #rrggbb and no
CSS color names, because trackDb's `color 0,255,0` is not CSS green.
tests/colorchecks covers all of it and tests/colorchecks.xfail aims all six forms
wrong in one step, so the failure has to name all six.
tests/regress/rm36212.xfail is the bug it was written for: a stanza that sets both
`itemRgb on` and `color` draws its items in the color setting instead of in the
file's own RGB column, because bedItemRgb() (hg/cgilib/bedCart.c) tests for the
presence of `color` before it tests for an explicit `itemRgb on`, so the explicit
setting is never reached. It is an xfail because the bug is live on the RR, on beta
and on genome-test.
It is also the first script in that directory that has been watched both to fail on
a build with the bug and to pass on a build with the fix -- the three-line reorder
built into parked #36212 and the same script pointed at that port. The fixture is
~/public_html/docentFixtures/itemRgbHub/, four tracks over four copies of one bed9
file whose items all carry a pure blue itemRgb column.
refs #36212, refs #37892
- lines changed 62, context: html, text, full: html, text
6b2ef5505a6121a2343661bea48ec63d4bdb7477 Wed Sep 9 09:24:32 2026 -0700
docent: itemXY picked another track's item when this track had none
`mouseover: {track: X, item: "n"}` and `click: {track: X, item: "n"}` both resolve
the name through itemXY, which gathered every map box on the PAGE carrying that
name and then did:
const inBand = cands.filter(h => h.inBand);
const pick = (inBand[0] || cands[0]) || null;
so when track X had no box of its own, `cands[0]` handed back a box belonging to
some other track. Nothing warned. The step passed, the tooltip that came up was
real, and it was the wrong row.
This is not hypothetical. A hub track that declares more bigBed fields than its
file has draws no items at all (#38310). A probe hub with eight tracks over one
file, four of them drawing nothing, reported all eight as drawing uc.1, because
hg38 and the three tracks that do work use that name too. rm35920 had been
reading hg38's native `ultras` rather than its own fixture hub for as long as it
existed, and looked green the whole time. An answer that is wrong but reads as a
pass is worse than a failure.
Candidates are now scoped by MAP NAME, which is how areaXY and itemXY's own error
message already picked a row: hgTracks names each map after the track it draws
(map_data_<key>, map_center_<key>), so the test is exact. The y-band survives only
as the tie-break between several boxes OF THIS TRACK that share a name -- geometry
was never a safe primary test, since a packed row stacks items above and below its
middle and a quickLift target puts them outside the band altogether.
The one case that still gets to decide by geometry is a page where NO map can be
attributed to this key at all, i.e. hgTracks named it something we do not
recognise. `anyMine` keeps the old behavior there rather than turning an
unrecognised name into a hard failure.
When the lookup now fails, the error says where the name actually was, which is
the sentence that would have saved the most time here:
item "uc.1" not found in track "pT5" ... 1 map box(es) in that row, addressable
as: title: "Click to alter the display density of pT5". That name IS on this
page, in hub_195310_pT1, hub_195310_pT2, hub_195310_pT8 -- another track's box
is never used for this one
Checked against that probe hub: the three tracks that really draw uc.1 still pass,
the ones that draw nothing now fail with the message above. Both suites are green,
tests/ (15) and tests/regress/.
refs #38252, refs #38310
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 11, context: html, text, full: html, text
a0adfc55d38c2fbdf34f3f2667052dbdb64db576 Thu Sep 10 14:39:03 2026 -0700
docent: montage label gutter now fits the widest label
`montage:` reserved a fixed gutter of 1.5 em for the panel label. That fits the
one-letter auto labels (A, B, C) and nothing else. A word label -- "virtChrom",
"quickLift" -- overflowed the gutter and painted over the left edge of its own
panel. The panels were all placed at the same x, but the figure read as if they
were misaligned, because the text ran into the first column of pixels of one
panel and not the other.
The gutter is now measured. Each label is laid out at width 0, its scrollWidth
is read, and the gutter is the widest of those plus a 0.4 em pad, with the old
1.5 em kept as the floor so short labels still line up the way they did.
white-space:nowrap stops a two-word label from wrapping into a taller row and
shifting its panel down.
refs #37892
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/README.txt
- lines changed 18, context: html, text, full: html, text
c4bcca06ef1a06c434c9136a79459f1512cd0606 Tue Sep 8 07:41:02 2026 -0700
docent: expect: can assert the color a track's row was drawn in, and a test for #36212
A bug about color leaves the page identical -- same rows, same height, same item
names, same tooltips -- so every check expect: had was blind to it. `color:` reads
the pixels instead: it names the color a row is mostly drawn in (`is:`), or one it
must not be (`not:`), with `part: label` for the center label rather than the
items, `at:`/`frac:`/`x:` for one item rather than the whole row, and a list form
so one step can state a whole color matrix and a failure name every row that came
out wrong.
hgTracks draws the whole view into one png and shows each row as a CSS-offset
slice of it, so a row's pixels are that slice drawn into a canvas at its offset.
The clipping box is the img's own div.sliceDiv, not the table cell: the center
label and the data are two slices inside one td_data_<key>, and measuring the cell
runs the canvas past the end of this row and into the next track's, which reads
that track's color as part of this one. The side labels are a separate png and are
never included, since "what color is this row" must not be answered by the label
text. White is background; everything else counts, black included, because a track
with no color of its own draws black items. is:/not: take r,g,b or #rrggbb and no
CSS color names, because trackDb's `color 0,255,0` is not CSS green.
tests/colorchecks covers all of it and tests/colorchecks.xfail aims all six forms
wrong in one step, so the failure has to name all six.
tests/regress/rm36212.xfail is the bug it was written for: a stanza that sets both
`itemRgb on` and `color` draws its items in the color setting instead of in the
file's own RGB column, because bedItemRgb() (hg/cgilib/bedCart.c) tests for the
presence of `color` before it tests for an explicit `itemRgb on`, so the explicit
setting is never reached. It is an xfail because the bug is live on the RR, on beta
and on genome-test.
It is also the first script in that directory that has been watched both to fail on
a build with the bug and to pass on a build with the fix -- the three-line reorder
built into parked #36212 and the same script pointed at that port. The fixture is
~/public_html/docentFixtures/itemRgbHub/, four tracks over four copies of one bed9
file whose items all carry a pure blue itemRgb column.
refs #36212, refs #37892
- src/hg/utils/docent/tests/colorchecks.docent.yaml
- lines changed 56, context: html, text, full: html, text
c4bcca06ef1a06c434c9136a79459f1512cd0606 Tue Sep 8 07:41:02 2026 -0700
docent: expect: can assert the color a track's row was drawn in, and a test for #36212
A bug about color leaves the page identical -- same rows, same height, same item
names, same tooltips -- so every check expect: had was blind to it. `color:` reads
the pixels instead: it names the color a row is mostly drawn in (`is:`), or one it
must not be (`not:`), with `part: label` for the center label rather than the
items, `at:`/`frac:`/`x:` for one item rather than the whole row, and a list form
so one step can state a whole color matrix and a failure name every row that came
out wrong.
hgTracks draws the whole view into one png and shows each row as a CSS-offset
slice of it, so a row's pixels are that slice drawn into a canvas at its offset.
The clipping box is the img's own div.sliceDiv, not the table cell: the center
label and the data are two slices inside one td_data_<key>, and measuring the cell
runs the canvas past the end of this row and into the next track's, which reads
that track's color as part of this one. The side labels are a separate png and are
never included, since "what color is this row" must not be answered by the label
text. White is background; everything else counts, black included, because a track
with no color of its own draws black items. is:/not: take r,g,b or #rrggbb and no
CSS color names, because trackDb's `color 0,255,0` is not CSS green.
tests/colorchecks covers all of it and tests/colorchecks.xfail aims all six forms
wrong in one step, so the failure has to name all six.
tests/regress/rm36212.xfail is the bug it was written for: a stanza that sets both
`itemRgb on` and `color` draws its items in the color setting instead of in the
file's own RGB column, because bedItemRgb() (hg/cgilib/bedCart.c) tests for the
presence of `color` before it tests for an explicit `itemRgb on`, so the explicit
setting is never reached. It is an xfail because the bug is live on the RR, on beta
and on genome-test.
It is also the first script in that directory that has been watched both to fail on
a build with the bug and to pass on a build with the fix -- the three-line reorder
built into parked #36212 and the same script pointed at that port. The fixture is
~/public_html/docentFixtures/itemRgbHub/, four tracks over four copies of one bed9
file whose items all carry a pure blue itemRgb column.
refs #36212, refs #37892
- src/hg/utils/docent/tests/colorchecks.xfail.docent.yaml
- lines changed 30, context: html, text, full: html, text
c4bcca06ef1a06c434c9136a79459f1512cd0606 Tue Sep 8 07:41:02 2026 -0700
docent: expect: can assert the color a track's row was drawn in, and a test for #36212
A bug about color leaves the page identical -- same rows, same height, same item
names, same tooltips -- so every check expect: had was blind to it. `color:` reads
the pixels instead: it names the color a row is mostly drawn in (`is:`), or one it
must not be (`not:`), with `part: label` for the center label rather than the
items, `at:`/`frac:`/`x:` for one item rather than the whole row, and a list form
so one step can state a whole color matrix and a failure name every row that came
out wrong.
hgTracks draws the whole view into one png and shows each row as a CSS-offset
slice of it, so a row's pixels are that slice drawn into a canvas at its offset.
The clipping box is the img's own div.sliceDiv, not the table cell: the center
label and the data are two slices inside one td_data_<key>, and measuring the cell
runs the canvas past the end of this row and into the next track's, which reads
that track's color as part of this one. The side labels are a separate png and are
never included, since "what color is this row" must not be answered by the label
text. White is background; everything else counts, black included, because a track
with no color of its own draws black items. is:/not: take r,g,b or #rrggbb and no
CSS color names, because trackDb's `color 0,255,0` is not CSS green.
tests/colorchecks covers all of it and tests/colorchecks.xfail aims all six forms
wrong in one step, so the failure has to name all six.
tests/regress/rm36212.xfail is the bug it was written for: a stanza that sets both
`itemRgb on` and `color` draws its items in the color setting instead of in the
file's own RGB column, because bedItemRgb() (hg/cgilib/bedCart.c) tests for the
presence of `color` before it tests for an explicit `itemRgb on`, so the explicit
setting is never reached. It is an xfail because the bug is live on the RR, on beta
and on genome-test.
It is also the first script in that directory that has been watched both to fail on
a build with the bug and to pass on a build with the fix -- the three-line reorder
built into parked #36212 and the same script pointed at that port. The fixture is
~/public_html/docentFixtures/itemRgbHub/, four tracks over four copies of one bed9
file whose items all carry a pure blue itemRgb column.
refs #36212, refs #37892
- src/hg/utils/docent/tests/docentTest.mk
- lines changed 14, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 32, context: html, text, full: html, text
4a5041811d79f95f11000fbc3fc310f9a4ae417a Sun Sep 13 15:36:26 2026 -0700
docent: a test run clears up after itself, refs #38252
A run left a log per script and a stills/ directory per script, and nothing
ever removed them, so `git status` in the test directories reported 89 files
that were not work. The obvious answer is a .gitignore, and it is the wrong
one: the files stay on disk, and git is taught to look away from the directory
new tests are written in.
`make test` now removes a passing script's log, stills and sessions. Nothing
reads any of it once the run is over -- nightly.sh reads this target's OUTPUT,
and the failure branch prints a failing log into that output while the file is
still there. A failing script keeps its log, and so does an xfail that passed,
which is the flip `make proof` is about and the one morning someone will want to
read the whole run. `parity` does the same with its three logs and the mp4 its
slow run records.
WARNING lines are echoed before the log goes. make test sends each script's
output to its log and prints only "ok", so a docent warning on a PASSING script
reached a file nobody opens -- and "tooltip never showed its own text" means the
step measured nothing. None of the 67 scripts warns today; the point is that
one that starts to will say so.
Five orphan logs are removed by hand in passing: rm36212.xfail.log,
rm36540.log, rm37388ui.xfail.log, rm37389.xfail.log and rm38272.xfail.log, left
behind when those scripts were renamed. make test deliberately does not sweep
logs it did not just write, since a rename in progress should not lose its
evidence; make clean is still there for a full wipe.
Measured: 67 of 67 green in 10m39s and the directory came back with nothing
untracked in it, the fifteen language tests next door pass and clear their
sessions/ too, and a script made to fail on purpose kept its log and its stills.
- src/hg/utils/docent/tests/proof.js
- lines changed 171, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/README.txt
- lines changed 19, context: html, text, full: html, text
c4bcca06ef1a06c434c9136a79459f1512cd0606 Tue Sep 8 07:41:02 2026 -0700
docent: expect: can assert the color a track's row was drawn in, and a test for #36212
A bug about color leaves the page identical -- same rows, same height, same item
names, same tooltips -- so every check expect: had was blind to it. `color:` reads
the pixels instead: it names the color a row is mostly drawn in (`is:`), or one it
must not be (`not:`), with `part: label` for the center label rather than the
items, `at:`/`frac:`/`x:` for one item rather than the whole row, and a list form
so one step can state a whole color matrix and a failure name every row that came
out wrong.
hgTracks draws the whole view into one png and shows each row as a CSS-offset
slice of it, so a row's pixels are that slice drawn into a canvas at its offset.
The clipping box is the img's own div.sliceDiv, not the table cell: the center
label and the data are two slices inside one td_data_<key>, and measuring the cell
runs the canvas past the end of this row and into the next track's, which reads
that track's color as part of this one. The side labels are a separate png and are
never included, since "what color is this row" must not be answered by the label
text. White is background; everything else counts, black included, because a track
with no color of its own draws black items. is:/not: take r,g,b or #rrggbb and no
CSS color names, because trackDb's `color 0,255,0` is not CSS green.
tests/colorchecks covers all of it and tests/colorchecks.xfail aims all six forms
wrong in one step, so the failure has to name all six.
tests/regress/rm36212.xfail is the bug it was written for: a stanza that sets both
`itemRgb on` and `color` draws its items in the color setting instead of in the
file's own RGB column, because bedItemRgb() (hg/cgilib/bedCart.c) tests for the
presence of `color` before it tests for an explicit `itemRgb on`, so the explicit
setting is never reached. It is an xfail because the bug is live on the RR, on beta
and on genome-test.
It is also the first script in that directory that has been watched both to fail on
a build with the bug and to pass on a build with the fix -- the three-line reorder
built into parked #36212 and the same script pointed at that port. The fixture is
~/public_html/docentFixtures/itemRgbHub/, four tracks over four copies of one bed9
file whose items all carry a pure blue itemRgb column.
refs #36212, refs #37892
- lines changed 10, context: html, text, full: html, text
236b28263edfc3b0cb9da780d65d2b647a0d98a6 Wed Sep 9 08:47:28 2026 -0700
docent: rm35920 was reading hg38's own ultras track, not the fixture hub's
The nightly went red on 2026-09-08 at rm35920's tooltip check. The cause was not
the bug the script is about. hg38 has a native `ultras` track under the
unusualcons superTrack, our copy of the reporter's hub called its track `ultras`
too, and a name resolves to `img_data_<name>` before it resolves to a hub row's
`hub_<n>_<name>`. The exact native id won every time: `track: {ultras: pack}`
turned on the native track and its superTrack, and `mouseover: {track: ultras}`
read the native row. The native items are named uc.N as well, so
`expect: {tip: "uc.1"}` passed on native data and nothing warned. The script
looked green for as long as it existed and tested nothing.
The fixture at ~/public_html/docentFixtures/Auto-generated_hub/ now calls its
tracks rm35920Ultras and rm35920UltraZoos, and declares `visibility pack` itself,
so the `track:` step is gone -- a hub track's cart name carries the per-run
hub_<n>_ prefix, which `track:` cannot write, so that step only ever moved native
tracks. README.txt gains the rule: a fixture hub must never name a track anything
a native assembly might also call it.
With the collision gone, what the two malformed tracks do on genome-test, hgwbeta
and the RR alike, measured 2026-09-09:
rm35920Ultras bigBed 12 + over a 4-field file -- row drawn, EMPTY. No crash,
no dialog, no garbage. #19984's auto-detection does not apply
when the declared type carries a number, so hgTracks trusts
the 12 and drops every row.
rm35920UltraZoos bigBed 4 + over a 3-field file -- items drawn with an EMPTY
name, byte for byte what a correctly declared `type bigBed 3`
over the same file produces, measured against a probe hub.
That is the garbage string gone, and it is #31771's fix.
So the tooltip assertion cannot be restored: the hub track has no named item to
hover, on any server. The script now asserts the crash half (both rows drawn, no
jsEmbedded) and clicks an ultraZoos item by position -- `at:` rather than `item:`,
because the fixed behavior leaves those boxes nameless -- and asserts the click
reaches a real hgc detail page. Checked against a deliberately wrong expectation
so it is not passing vacuously.
refs #35920, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 24, context: html, text, full: html, text
bb27fdc2f612380ae05efcd1959b4b26297738e5 Wed Sep 9 13:16:38 2026 -0700
docent: an xfail for #38310, and the second script that needs pixels
A hub track whose `type bigBed N` declares more fields than the file holds drew
its row with no items in it. The fix is written and verified but is not on
master, so no server passes this today. That is the only reason it is an .xfail:
`make test` fails if an xfail passes, so the day the fix reaches genome-test the
suite says there is a test waiting.
The ticket says the row came up empty "with no warning". The row is empty, but
there is a message. hgTracks catches the abort into networkErrMsg and swaps in
bigDrawWarning, which paints the row as a pale yellow bar, 240,240,180
(undefinedYellowColor, hg/hgTracks/simpleTracks.c), with the text inside it. The
text read `invalid signed integer: ""`, the fourth field of a four-field row
array that was never filled in. So the message was there, said nothing useful,
and is drawn INSIDE the png, which is why every text check in this suite is
blind to it and why the original measurement, counting item map boxes in the
HTML, reported silence.
That is why the assertion is a color check, the second one here after rm36212.
Each row is asked for the color it is drawn in: fixed gives a black item box and
a black pack label, dominant 0,0,0; broken gives a full-width bigWarn bar,
dominant 240,240,180. `is: "0,0,0"` states the item is there and
`not: "240,240,180"` states the warning bar is not. `rows:` cannot express this,
because the broken build draws all eight rows.
The second half of the same bug is on the details page: hgc took the declared
count too and aborted, so an item that did draw could not be clicked through.
That is live on the RR today for hg38 setDups and two hg19 exomeProbesets
subtracks, all `bigBed 4` over a three-field file. The step clicks an item and
asserts the item's POSITION, because the aborted page carries the track's
longLabel twice in its own header and a text: check on that alone passes on it.
It clicks bb9 rather than bb12, since "type bigBed 12" is a prefix of
"type bigBed 12 +".
Do not add a mouseover: step here. Before docent's e2b5b26b925, itemXY handed a
track with no box of its own a neighbour's box, so a tooltip check reported uc.1
for rows that drew nothing.
Measured both ways on 2026-09-09. Against genome-test it fails at the color step
naming all five over-declared rows, each 240,240,180 at 95% of the row; against
the #38310 sandbox all eight are 0,0,0 at 100% and the run exits 0. Then the
whole directory was run against that sandbox twice, once with the patched
hgTracks and hgc and once with unpatched controls built from the same tree:
thirty-seven scripts, identical verdicts, except this one. Notes in
/hive/groups/browser/redmineNotes/38310/claude/.
The fixture is ours, at ~/public_html/docentFixtures/bigBedFieldCount/. One
bigBed with four fields, eight tracks over it, one declared type each, so the
only thing that differs between the rows is the number on the type line.
hubCheck rejects five of the eight, correctly; it is the empty row that is the
bug, not the hub.
README.txt gains the section, and with it two rules that apply to any script
here: `rows:` cannot express "this track drew its items", and a drawn item that
cannot be clicked through is half a bug.
refs #38310, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 44, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 32, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- lines changed 34, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- lines changed 33, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/makefile
- lines changed 9, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 5, context: html, text, full: html, text
4a5041811d79f95f11000fbc3fc310f9a4ae417a Sun Sep 13 15:36:26 2026 -0700
docent: a test run clears up after itself, refs #38252
A run left a log per script and a stills/ directory per script, and nothing
ever removed them, so `git status` in the test directories reported 89 files
that were not work. The obvious answer is a .gitignore, and it is the wrong
one: the files stay on disk, and git is taught to look away from the directory
new tests are written in.
`make test` now removes a passing script's log, stills and sessions. Nothing
reads any of it once the run is over -- nightly.sh reads this target's OUTPUT,
and the failure branch prints a failing log into that output while the file is
still there. A failing script keeps its log, and so does an xfail that passed,
which is the flip `make proof` is about and the one morning someone will want to
read the whole run. `parity` does the same with its three logs and the mp4 its
slow run records.
WARNING lines are echoed before the log goes. make test sends each script's
output to its log and prints only "ok", so a docent warning on a PASSING script
reached a file nobody opens -- and "tooltip never showed its own text" means the
step measured nothing. None of the 67 scripts warns today; the point is that
one that starts to will say so.
Five orphan logs are removed by hand in passing: rm36212.xfail.log,
rm36540.log, rm37388ui.xfail.log, rm37389.xfail.log and rm38272.xfail.log, left
behind when those scripts were renamed. make test deliberately does not sweep
logs it did not just write, since a rename in progress should not lose its
evidence; make clean is still there for a full wipe.
Measured: 67 of 67 green in 10m39s and the directory came back with nothing
untracked in it, the fifteen language tests next door pass and clear their
sessions/ too, and a script made to fail on purpose kept its log and its stills.
- src/hg/utils/docent/tests/regress/nightly.sh
- lines changed 49, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm20460.docent.yaml
- lines changed 43, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/rm22144.docent.yaml
- lines changed 56, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/rm23922.docent.yaml
- lines changed 62, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/rm26772.docent.yaml
- lines changed 48, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/rm27113.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm27855.docent.yaml
- lines changed 47, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/rm29452.docent.yaml
- lines changed 83, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/rm29787.docent.yaml
- lines changed 46, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/rm30833.docent.yaml
- lines changed 52, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/rm32263.docent.yaml
- lines changed 36, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/rm34250.docent.yaml
- lines changed 60, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/rm34651.docent.yaml
- lines changed 39, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/rm35333.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm35472.docent.yaml
- lines changed 57, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/rm35580.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm35865.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm35906.docent.yaml
- lines changed 44, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/rm35920.docent.yaml
- lines changed 58, context: html, text, full: html, text
236b28263edfc3b0cb9da780d65d2b647a0d98a6 Wed Sep 9 08:47:28 2026 -0700
docent: rm35920 was reading hg38's own ultras track, not the fixture hub's
The nightly went red on 2026-09-08 at rm35920's tooltip check. The cause was not
the bug the script is about. hg38 has a native `ultras` track under the
unusualcons superTrack, our copy of the reporter's hub called its track `ultras`
too, and a name resolves to `img_data_<name>` before it resolves to a hub row's
`hub_<n>_<name>`. The exact native id won every time: `track: {ultras: pack}`
turned on the native track and its superTrack, and `mouseover: {track: ultras}`
read the native row. The native items are named uc.N as well, so
`expect: {tip: "uc.1"}` passed on native data and nothing warned. The script
looked green for as long as it existed and tested nothing.
The fixture at ~/public_html/docentFixtures/Auto-generated_hub/ now calls its
tracks rm35920Ultras and rm35920UltraZoos, and declares `visibility pack` itself,
so the `track:` step is gone -- a hub track's cart name carries the per-run
hub_<n>_ prefix, which `track:` cannot write, so that step only ever moved native
tracks. README.txt gains the rule: a fixture hub must never name a track anything
a native assembly might also call it.
With the collision gone, what the two malformed tracks do on genome-test, hgwbeta
and the RR alike, measured 2026-09-09:
rm35920Ultras bigBed 12 + over a 4-field file -- row drawn, EMPTY. No crash,
no dialog, no garbage. #19984's auto-detection does not apply
when the declared type carries a number, so hgTracks trusts
the 12 and drops every row.
rm35920UltraZoos bigBed 4 + over a 3-field file -- items drawn with an EMPTY
name, byte for byte what a correctly declared `type bigBed 3`
over the same file produces, measured against a probe hub.
That is the garbage string gone, and it is #31771's fix.
So the tooltip assertion cannot be restored: the hub track has no named item to
hover, on any server. The script now asserts the crash half (both rows drawn, no
jsEmbedded) and clicks an ultraZoos item by position -- `at:` rather than `item:`,
because the fixed behavior leaves those boxes nameless -- and asserts the click
reaches a real hgc detail page. Checked against a deliberately wrong expectation
so it is not passing vacuously.
refs #35920, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36029.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36048.docent.yaml
- lines changed 42, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/rm36059.docent.yaml
- lines changed 53, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/rm36061.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36125.docent.yaml
- lines changed 72, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/rm36212.docent.yaml
- lines changed 11, context: html, text, full: html, text
fe722461f833a6ccf990003f22be9fe9e8962ebf Wed Sep 9 08:47:44 2026 -0700
docent: promote rm36212 out of .xfail, the fix is on master
The nightly reported "this was supposed to fail, and it passed" on 2026-09-09.
cbb406cd96e (an explicit `itemRgb on` beats the presence of a `color` setting)
reached origin/master and so genome-test, and the script's color assertions now
hold there: itemRgbAndColor draws its items 0,0,255 from the file's own RGB
column while its center label stays 0,255,0.
It ships in v504, so the script still fails on the RR and on hgwbeta until that
release goes out. The header records that, and the three-server measurement is
kept with the dates rather than deleted, since it is the before half of the only
before-and-after this directory has.
refs #36212, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 4, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36212.xfail.docent.yaml
- lines changed 81, context: html, text, full: html, text
c4bcca06ef1a06c434c9136a79459f1512cd0606 Tue Sep 8 07:41:02 2026 -0700
docent: expect: can assert the color a track's row was drawn in, and a test for #36212
A bug about color leaves the page identical -- same rows, same height, same item
names, same tooltips -- so every check expect: had was blind to it. `color:` reads
the pixels instead: it names the color a row is mostly drawn in (`is:`), or one it
must not be (`not:`), with `part: label` for the center label rather than the
items, `at:`/`frac:`/`x:` for one item rather than the whole row, and a list form
so one step can state a whole color matrix and a failure name every row that came
out wrong.
hgTracks draws the whole view into one png and shows each row as a CSS-offset
slice of it, so a row's pixels are that slice drawn into a canvas at its offset.
The clipping box is the img's own div.sliceDiv, not the table cell: the center
label and the data are two slices inside one td_data_<key>, and measuring the cell
runs the canvas past the end of this row and into the next track's, which reads
that track's color as part of this one. The side labels are a separate png and are
never included, since "what color is this row" must not be answered by the label
text. White is background; everything else counts, black included, because a track
with no color of its own draws black items. is:/not: take r,g,b or #rrggbb and no
CSS color names, because trackDb's `color 0,255,0` is not CSS green.
tests/colorchecks covers all of it and tests/colorchecks.xfail aims all six forms
wrong in one step, so the failure has to name all six.
tests/regress/rm36212.xfail is the bug it was written for: a stanza that sets both
`itemRgb on` and `color` draws its items in the color setting instead of in the
file's own RGB column, because bedItemRgb() (hg/cgilib/bedCart.c) tests for the
presence of `color` before it tests for an explicit `itemRgb on`, so the explicit
setting is never reached. It is an xfail because the bug is live on the RR, on beta
and on genome-test.
It is also the first script in that directory that has been watched both to fail on
a build with the bug and to pass on a build with the fix -- the three-line reorder
built into parked #36212 and the same script pointed at that port. The fixture is
~/public_html/docentFixtures/itemRgbHub/, four tracks over four copies of one bed9
file whose items all carry a pure blue itemRgb column.
refs #36212, refs #37892
- src/hg/utils/docent/tests/regress/rm36331.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36335.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36340.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36370.docent.yaml
- lines changed 41, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/rm36387.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36484.docent.yaml
- lines changed 32, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/rm36514.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36540.xfail.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36668.docent.yaml
- lines changed 58, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/rm36702.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36798.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36805.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36810.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36836.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36888.xfail.docent.yaml
- lines changed 80, context: html, text, full: html, text
8cb4edc2f77e57fb03e7a043a287c40d74b6814e Wed Sep 9 13:16:19 2026 -0700
docent: an xfail for #36888, a bigMethyl track that vanishes when zoomed out
Mark's report is that a bigMethyl track draws its data at base level and has
nothing between its labels two 100x zoom-outs later. No "zoom in" message, no
empty row, no error.
One missing method causes it. bigBedSelectRangeExt (hg/hgTracks/bigBedTrack.c)
asks for bigBedMaxItems()+1 intervals and throws the whole list away when the
count is over the limit, recording it in the trackDb setting bigBedItemsCount.
Every other bigBed type recovers, because commonBigBedMethods installs
track->loadSummary = loadBigBedSummary, which reads that setting and falls back
to the coverage graph. bigMethylMethods (hg/hgTracks/simpleTracks.c) sets
isBigBed and its own loadItems but never loadSummary, so an overflowed window
gets an empty item list and nothing else.
Watched both ways on 2026-09-08, which most of this suite cannot claim:
genome-test (v503, unfixed) draws no img_data_ row and no note; ts park 48090,
v503 plus the one-line loadSummary, draws the row and the note. Both checks
flipped together. The fix is not on master, so this is an .xfail: the day it
reaches genome-test the script passes, `make test` fails because an xfail
passed, and the fix is to drop the .xfail from the name.
The fixture is ours, at ~/public_html/docentFixtures/bigMethyl36888/, a
single-file hub over a 6 MB synthetic bigBed. The ticket's own hub is a 24 GB
file in another user's public_html, and #37490 already lost a fixture that way.
The two windows straddle the 100,000-item limit by a factor of six, so raising
bigBedMaxItems a little cannot quietly defeat the test, and the control step
plus noHas: mean a fixture that stopped resolving fails loudly instead of
satisfying the xfail.
The note is checked with has: on the track's control link, not with text:,
because labelTrackAsDensityTooManyItems appends it to longLabel and hgTracks
draws longLabel into the center-label image. The selector matches both title=
and mouseovertext=, since addMouseover() in hg/js/utils.js moves the text
between them under the showMouseovers hg.conf setting.
refs #36888, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36917.docent.yaml
- lines changed 48, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/rm36942.docent.yaml
- lines changed 46, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/rm37130.docent.yaml
- lines changed 45, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/rm37175.docent.yaml
- lines changed 55, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/rm37282.docent.yaml
- lines changed 48, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/rm37326.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37388.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37389.docent.yaml
- lines changed 44, context: html, text, full: html, text
2bd879990be58c3a533783a25f74b3b8e937dda4 Sun Sep 6 15:46:38 2026 -0700
docent: make rm37389 assert what 12d4ad442f7 said it already asserted
12d4ad442f7 renamed rm37389 out of .xfail and its message said "the script now
also asks for a phrase out of the hub's own description file". It did not:
.../regress/{rm37389.xfail.docent.yaml => rm37389.docent.yaml} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
The rename was the whole commit. So the script has been asserting `text:
"Description"` and nothing else, while its own comment described a second check
that was not in the file. This adds it.
"Description" was not doing nothing -- measured on the crash1 hgc page, it occurs
exactly once in the visible text and it is the heading printTrackHtml writes above
the block, which goes away with the block. So this is a tightening rather than a
rescue. What the phrase adds is that it can only be on the page if the
description file was really fetched from the GenArk hub, which is the thing
#38275 broke.
The comment is rewritten to match, and is shorter. The bisect scaffolding is
gone -- the ruled-out list was worth writing while the cause was unknown and is
noise now that it is known. Both causes are kept, since a reader who sees only
one of them cannot understand why the script is worded as it is.
First commit on this branch since it was merged. The branch was 99 commits
behind and fully contained in master, so it was fast-forwarded to 79aa96ab96e
first. Note that the nightly runs from a clone of origin/master, so this does
not run nightly until the branch is merged.
refs #38252
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37489.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37491.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37520.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37553.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37562.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37615.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37646.docent.yaml
- lines changed 49, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/rm37743.docent.yaml
- lines changed 36, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/rm37785.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37805.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm37815.docent.yaml
- lines changed 43, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/rm37906.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm38032.docent.yaml
- lines changed 53, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/rm38042.docent.yaml
- lines changed 53, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/rm38108.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm38126.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm38146.docent.yaml
- lines changed 50, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/rm38185.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm38272.docent.yaml
- lines changed 3, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm38310.docent.yaml
- lines changed 7, context: html, text, full: html, text
7eb4d115091bd389d5296d9198dbf0f327b35bdd Thu Sep 10 07:10:50 2026 -0700
docent: promote rm38310 out of .xfail, the #38310 fix reached genome-test
68f831e1209 landed on master 2026-09-09 and genome-test built it overnight, so
the 2026-09-10 nightly went red the way an xfail is supposed to: the script that
was expected to fail passed. Drop the .xfail from the name and rewrite the header
paragraph that said the fix was not on master yet.
This script has now been watched failing on genome-test and passing on the next
build of that same server, which is stronger evidence than the sandbox A/B it
already carried. refs #38310, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 4, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm38310.xfail.docent.yaml
- lines changed 103, context: html, text, full: html, text
bb27fdc2f612380ae05efcd1959b4b26297738e5 Wed Sep 9 13:16:38 2026 -0700
docent: an xfail for #38310, and the second script that needs pixels
A hub track whose `type bigBed N` declares more fields than the file holds drew
its row with no items in it. The fix is written and verified but is not on
master, so no server passes this today. That is the only reason it is an .xfail:
`make test` fails if an xfail passes, so the day the fix reaches genome-test the
suite says there is a test waiting.
The ticket says the row came up empty "with no warning". The row is empty, but
there is a message. hgTracks catches the abort into networkErrMsg and swaps in
bigDrawWarning, which paints the row as a pale yellow bar, 240,240,180
(undefinedYellowColor, hg/hgTracks/simpleTracks.c), with the text inside it. The
text read `invalid signed integer: ""`, the fourth field of a four-field row
array that was never filled in. So the message was there, said nothing useful,
and is drawn INSIDE the png, which is why every text check in this suite is
blind to it and why the original measurement, counting item map boxes in the
HTML, reported silence.
That is why the assertion is a color check, the second one here after rm36212.
Each row is asked for the color it is drawn in: fixed gives a black item box and
a black pack label, dominant 0,0,0; broken gives a full-width bigWarn bar,
dominant 240,240,180. `is: "0,0,0"` states the item is there and
`not: "240,240,180"` states the warning bar is not. `rows:` cannot express this,
because the broken build draws all eight rows.
The second half of the same bug is on the details page: hgc took the declared
count too and aborted, so an item that did draw could not be clicked through.
That is live on the RR today for hg38 setDups and two hg19 exomeProbesets
subtracks, all `bigBed 4` over a three-field file. The step clicks an item and
asserts the item's POSITION, because the aborted page carries the track's
longLabel twice in its own header and a text: check on that alone passes on it.
It clicks bb9 rather than bb12, since "type bigBed 12" is a prefix of
"type bigBed 12 +".
Do not add a mouseover: step here. Before docent's e2b5b26b925, itemXY handed a
track with no box of its own a neighbour's box, so a tooltip check reported uc.1
for rows that drew nothing.
Measured both ways on 2026-09-09. Against genome-test it fails at the color step
naming all five over-declared rows, each 240,240,180 at 95% of the row; against
the #38310 sandbox all eight are 0,0,0 at 100% and the run exits 0. Then the
whole directory was run against that sandbox twice, once with the patched
hgTracks and hgc and once with unpatched controls built from the same tree:
thirty-seven scripts, identical verdicts, except this one. Notes in
/hive/groups/browser/redmineNotes/38310/claude/.
The fixture is ours, at ~/public_html/docentFixtures/bigBedFieldCount/. One
bigBed with four fields, eight tracks over it, one declared type each, so the
only thing that differs between the rows is the number on the type line.
hubCheck rejects five of the eight, correctly; it is the empty row that is the
bug, not the hub.
README.txt gains the section, and with it two rules that apply to any script
here: `rows:` cannot express "this track drew its items", and a drawn item that
cannot be clicked through is half a bug.
refs #38310, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/hgConfCatalog/hgConfCatalog.py
- lines changed 8, context: html, text, full: html, text
d11a21c48cdc2cebe0a97779b510df8317b980cd Mon Sep 7 06:29:43 2026 -0700
bigNet: gate the track type behind an hg.conf flag, refs #20824
Add the boolean hg.conf setting bigNet, default FALSE, so the type ships
dark and a machine turns it on with bigNet=on.
trackHubBigNetEnabled() in hg/lib/trackHub.c is the one read; the four
places that accept or advertise the type ask it. validateOneTrack drops
bigNet from the hub track type allowlist, validateOneTdb drops it from the
types quickLift will lift, hubCheck drops it from VALID_TRACK_TYPES and
from the message listing the valid types, and hubApi does not add it to
supportedTypes. netToBigNet, netTrack.c, chainNetDbLoad.c and the hgc
details code are unchanged, since they cannot be reached once a bigNet
track will not load.
With the flag off hgTracks does not abort the hub. It draws the track row
as a bigWarn bar reading "Unsupported type 'bigNet ...'" and the rest of
the hub loads normally.
Register the flag in hgConfCatalog.py with role="gate" so the sunset
report tracks it.
- lines changed 19, context: html, text, full: html, text
992aeef92fea7be25a2acd916578898649212a32 Mon Sep 7 11:31:07 2026 -0700
quickLift: gate the alignment lift behind an hg.conf flag, refs #38249
Add browser.quickLiftAlignments, default FALSE, so the alignment lift ships
dark and a machine turns it on with browser.quickLiftAlignments=on. It sits
beside browser.quickLift, the gate on the rest of the feature.
quickLiftAlignmentsEnabled() in hg/lib/quickLift.c is the one read, and
validateOneTdb in hg/lib/trackHub.c is the one place that asks it, before an
alignment track may enter a quickLift hub. That is the only door:
quickLiftUrl and quickLiftDb, the pair every lift path keys off, are written
by the quickLift hub writer and by nothing else, so with the flag off an
alignment track never gets them and the lifting, drawing and details code
behind them cannot be reached. pslTrack.c, chainTrack.c, wigMafTrack.c,
bigBedTrack.c and hgc.c are unchanged.
With the flag off hgConvert lists psl, bigPsl, chain, bigChain, maf, bigMaf
and wigMaf tracks in its "type is not supported by QuickLift" table, which is
what it did before this work. A hub built while the flag was on keeps working
after it is turned off, since its stanzas are already in the hub file in
trash, so this holds the feature back from people who have not used it rather
than switching off a session that has.
Read the hg.conf half with a literal cfgOptionBooleanDefault rather than
cartOrCfgOption so harvestHgConf.py can see it; a cart variable of the same
name still overrides it. Register the flag in hgConfCatalog.py with
role="gate" so the sunset report tracks it, and turn it on in
confs/hgwdev.hg.conf.
- lines changed 14, context: html, text, full: html, text
59e2bbf2d3a99e5f18436a756e1e5c8b64308e33 Tue Sep 8 09:21:02 2026 -0700
Copy a track collection's hub file when the program that writes it asks for a copy, instead of on every session load.
cartCopyLocalHubs ran on every session load, in every CGI and in four more places in
hgSession. For each customComposite-<db> cart variable it copied the collection's hub
file to a fresh trash name and registered the copy in hgcentral.hubStatus to get a hub
id. Nothing removed the old row. 81% of the 3.1 million rows in hubStatus on the RR are
these dead registrations, and each load also took the central_hubStatus advisory lock
and wrote to a MyISAM table, which serializes every concurrent load of a shared session
that carries a collection.
The copy itself is needed. A saved session's hub file lives under sessionDataDir and
every load of that session names it, so hgCollection must not write it in place. It was
just being made for every load, and almost no load is followed by an edit. hgCollection
is the only program that writes one of these files, so it now asks for its own copy:
main() calls cartRequestLocalHubCopy() before it opens the cart, and cartNew() makes the
copy for it.
That has to be a property of the program rather than of the request, and the copy has to
be made at cart open, above hubConnectLoadHubs. Copying the file gives the hub a new id,
the hubs are loaded during cart open, and trackList carries that id in every track name,
so a copy made any later leaves printTrackDbListToHub looking for the collection under
an id trackList does not have and the hub file comes out with a header and no tracks. A
condition that instead tries to work out whether this particular request will write
cannot be correct either: "cmd" is not in hgTracks' excludeVars, so it persists in the
cart and hgCollection dispatches on the cart rather than on the CGI variable, a settings
file can carry it, and a command-line run has no SCRIPT_NAME to test.
Three other parts. copyLocalHubs skips a hub the cart already owns, so repeated edits
reuse one file and one hub id rather than renumbering the hub on every drag; the test
requires a plain file, because saveTrackFile leaves the trash name behind as a symbolic
link to the durable copy and writing through that link would rewrite the saved session's
own file. copyLocalHubs also screens the path with isServerUserFilePath before opening
it, the way getHubName does, and selects on customComposite-<db> with the dash so that
the names it acts on are the ones fileNameCartVarPrefixes screens. saveTrackFile copies
when the source is a local hub outside trash that is not under this session's own
directory, so loading one session and saving it under a new name gives the new session
its own file instead of a reference into the first one's directory. pathIsUnderDir is no
longer static in trashDir.c.
All of it is behind the hg.conf gate collectionHubCopyOnWrite, default off, which
reproduces the old behavior exactly. Retiring the gate is not uniform: the body of
cartCopyLocalHubsOnSessionLoad is the old behavior and that function and its five
callers go away with the gate, while the other three tests lose only the gate term.
Measured on hgcentraltest with one binary. With the gate off, five loads of a session
carrying a collection add five hubStatus rows and three later edits add none; with it
on, the five loads add none and the first edit adds one. Saving a loaded session costs
two rows off and one on. The resulting hub file is byte identical either way apart from
the hub id, and the file the saved session names is unchanged by md5 and mtime,
including after an edit made straight after a save, and including when the cart names a
trash symbolic link into session storage.
refs #38273
- lines changed 8, context: html, text, full: html, text
fc8de100a3437b9fc33bdeb0f459ca2a93e2f318 Wed Sep 9 08:14:32 2026 -0700
hgSession: address the code review of the new Sessions page
Rename and unshare now keep the public listing's thumbnail with the session it
belongs to. The picture's file name is built from the encoded session name, so
renaming a listed session left the listing pointing at nothing and the old file
behind, and dropping a session from the listing to a plain shared link kept the
picture. The classic page had the same problem in a subtler form: it removed the
thumbnail after the row had already been renamed, so the old file survived.
Saving under a name that is already in use asks before it replaces that session,
using the failIfExists reply that the top-right Share a link menu already relies
on. The description and "only I can load it" steps that follow a save now report
a failure instead of reloading in silence, and what thumbnailAdd has to say when
it cannot build a picture reaches the user instead of being freed unread.
A session description no longer travels through a title attribute. The tooltip
machinery in utils.js inserts its text with innerHTML and an attribute is decoded
on the way, so a description containing angle brackets was interpreted as markup
rather than shown as typed. It is attached, escaped, after each table draw, which
also gives the rows DataTables renders later the same styled mouseovers as the
rest of the page.
Also: the AJAX endpoints say so when there is no session by that name, instead of
reporting a no-op as a success; the new page always offers its way back to the
classic page, since the cart variable that got the user there sticks; and four
unused CSS rules, a dead element lookup and a dead local are gone. hgConfCatalog
cited the wrong ticket for the two sessionNewPage flags.
refs #38180, refs #38157
- lines changed 14, context: html, text, full: html, text
68b9911e4c156cd1346fde5c957434b1a8780d1c Wed Sep 9 08:49:07 2026 -0700
Show the transcript's own codon number where it differs from the genomic one
The gene tracks count codons along the genome. A RefSeq transcript is a
sequence in its own right, so where it has an insertion or a deletion relative
to the assembly, every codon 3' of that point gets a different number here than
the sequence provider gives it, one codon per three bases. DNM1 on canFam3 is
the reported case: the transcript carries 21 bases canFam3 does not, so our
p.249 is NCBI's p.256. Neither number is wrong, but HGVS c./p. is defined on
the transcript, so the number people quote is the one we were not showing.
The genomic number and amino acid are unchanged. Codons whose two numbers
disagree now draw in the existing CDS_QUERY_INSERTION orange with a "!" after
the codon number, and their mouseover adds the transcript number plus a link to
a new FAQ entry. Both directions of indel are covered, and so is the case
where the alignment does not reach the start of the CDS (801 transcripts on
hg38), which needs the transcript's own CDS annotation as the anchor rather
than the alignment.
Numbers come from the transcript alignment, ncbiRefSeqPsl or refSeqAli, which
is the same source hgvsMapToGenome already uses, so the browser now agrees with
its own position search. Two queries per table per window, on the bin index,
and only at zoomedToCdsColorLevel, where the mouseover carrying the numbers is
drawn: on and off are within noise at every zoom.
Gated by showTxCodonNumbers in hg.conf, default off, catalogued as a release
gate. With it off nothing is looked up and the rendering and mouseover are
byte-identical to before.
refs #38298
- lines changed 3, context: html, text, full: html, text
ba5aa085f499f17ea3b88621e40a34c9c1ccb3d1 Thu Sep 10 11:03:15 2026 -0700
hui.c: turn the track color picker on by default, refs #20460
The showColorPicker gate has been in the tree since v496 and defaulted to
FALSE, so the color picker was only visible on hgwdev and genome-test. The
default is now TRUE. The flag stays in place, so a mirror or hgwbeta can
still set showColorPicker=off without a code change.
Also record the new default in the hg.conf catalog.
- lines changed 5, context: html, text, full: html, text
b809513a3acdf9d24d2f3b70cec639b7a33acb20 Thu Sep 10 11:14:30 2026 -0700
hgConfCatalog: drop the gcOnTheFlyCoExist row, its code is gone, refs #38208
Hiram removed the C code that read gcOnTheFlyCoExist, but the catalog row
survived it and still cited hg/hgTracks/hgTracks.c. Since gate_lifecycle()
walks the catalog rather than the tree, the flag aged past the QA grace
window and the v503 sunset report flagged it as "stalled in QA -- either
turn these on or delete the feature", for a feature that had already been
deleted. --reconcile did see the row had no reader left, but files that
under question 1, which prints only with --verbose.
Removing the row drops it from the sunset report and from the reconcile
drift list, so it never reaches hgConfGateBacklog.txt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 220, context: html, text, full: html, text
e357453b3740779250e8cf80c2f5104eb77c824b Thu Sep 10 11:23:20 2026 -0700
hgConfCatalog: report the hg.conf lines a flipped default made pointless, refs #37925
Flipping a gate's default to TRUE does not turn a feature on anywhere: it was
already on wherever somebody had appended the flag by hand while the feature
sat in QA. What the flip does is make those lines pointless, and while one is
there the flag reads on locally whatever the tree says, which is how a wrong
default survives on the machine most likely to catch it.
--redundant reads the hg.conf files readable from hgwdev, joins them against
the catalog and names the lines to delete. It keeps three populations apart:
a line that turns a shipped gate on is litter, a line that turns one off is a
live decision and has to be left alone, and a value that is not one of the six
words hgConfig.c accepts aborts every CGI that reads it. Silent and exit 0
when there is nothing to delete, the same shape as --reconcile, so
nightlyRegister.sh runs it as a third pass and the reminder rides the mail that
already goes out.
Only gates are considered. A knob is a switch a machine is entitled to set
forever, so reporting one would be the crying wolf the gate/knob split exists
to prevent.
- lines changed 3, context: html, text, full: html, text
45ebd62062de3b3a34cf306c89701089664bc36b Fri Sep 11 08:53:21 2026 -0700
hgConfCatalog: catalog showAliases as defaulting on, refs #37925
c69d3e1d9d flipped the showAliases code default to TRUE in hgTracks.c but
left the catalog entry at default="FALSE". Two reports read that field and
both were wrong because of it.
--redundant skips any gate whose catalog default is not "TRUE", so it walked
past the showAliases=on line at /usr/local/apache/cgi-bin/hg.conf:541 -- the
exact kind of line it was added to find. With this it reports 3 redundant
lines instead of 2.
--sunset is worse than quiet. gate_lifecycle() sets shipped from the same
field, and the firstTrue age comes from git history, so it sees the flip. A
flip date next to a FALSE default is read there as a flip that was reverted,
which is the opposite of what happened.
Same pairing ba5aa085f49 did for showColorPicker ten minutes after the flip.
Found in the 2026-09-11 daily code review.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 9, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 1, context: html, text, full: html, text
97557b2dab79bd19b0345f1ea49a233c191300ae Sat Sep 12 06:59:07 2026 -0700
hgConfCatalog: cite skipMalformedCgiPairs where the tree actually reads it, refs #37925
The row cited lib/cheapcgi.c, which is where the flag takes effect but not
where it is read: the kent libraries cannot read hg.conf at all. The read is
hg/lib/hgConfig.c:227, which hands the setting to cheapcgi through
cgiSkipMalformedPairs. The nightly reconcile had been reporting the row as
citing a file the read has left.
- src/hg/utils/hgConfCatalog/nightlyRegister.sh
- lines changed 34, context: html, text, full: html, text
e357453b3740779250e8cf80c2f5104eb77c824b Thu Sep 10 11:23:20 2026 -0700
hgConfCatalog: report the hg.conf lines a flipped default made pointless, refs #37925
Flipping a gate's default to TRUE does not turn a feature on anywhere: it was
already on wherever somebody had appended the flag by hand while the feature
sat in QA. What the flip does is make those lines pointless, and while one is
there the flag reads on locally whatever the tree says, which is how a wrong
default survives on the machine most likely to catch it.
--redundant reads the hg.conf files readable from hgwdev, joins them against
the catalog and names the lines to delete. It keeps three populations apart:
a line that turns a shipped gate on is litter, a line that turns one off is a
live decision and has to be left alone, and a value that is not one of the six
words hgConfig.c accepts aborts every CGI that reads it. Silent and exit 0
when there is nothing to delete, the same shape as --reconcile, so
nightlyRegister.sh runs it as a third pass and the reminder rides the mail that
already goes out.
Only gates are considered. A knob is a switch a machine is entitled to set
forever, so reporting one would be the crying wolf the gate/knob split exists
to prevent.
- src/hg/utils/hubCheck/hubCheck.c
- lines changed 1, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- lines changed 13, context: html, text, full: html, text
d11a21c48cdc2cebe0a97779b510df8317b980cd Mon Sep 7 06:29:43 2026 -0700
bigNet: gate the track type behind an hg.conf flag, refs #20824
Add the boolean hg.conf setting bigNet, default FALSE, so the type ships
dark and a machine turns it on with bigNet=on.
trackHubBigNetEnabled() in hg/lib/trackHub.c is the one read; the four
places that accept or advertise the type ask it. validateOneTrack drops
bigNet from the hub track type allowlist, validateOneTdb drops it from the
types quickLift will lift, hubCheck drops it from VALID_TRACK_TYPES and
from the message listing the valid types, and hubApi does not add it to
supportedTypes. netToBigNet, netTrack.c, chainNetDbLoad.c and the hgc
details code are unchanged, since they cannot be reached once a bigNet
track will not load.
With the flag off hgTracks does not abort the hub. It draws the track row
as a bigWarn bar reading "Unsupported type 'bigNet ...'" and the rest of
the hub loads normally.
Register the flag in hgConfCatalog.py with role="gate" so the sunset
report tracks it.
- src/hg/utils/makeTrackIndex/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/utils/makefile
- lines changed 2, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/utils/netToBigNet/makefile
- lines changed 3, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/utils/netToBigNet/netToBigNet.c
- lines changed 151, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/utils/netToBigNet/tests/expected/simpleTest.bigNet
- lines changed 14, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/utils/netToBigNet/tests/input/hg38.chrom.sizes
- lines changed 1, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/utils/netToBigNet/tests/input/hg38.mm39.test.net
- lines changed 15, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/utils/netToBigNet/tests/makefile
- lines changed 22, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/utils/otto/genArk/README
- lines changed 18, context: html, text, full: html, text
20da4c889251225224be1661d4f3f7f9d4d0d8fa Thu Sep 10 11:24:58 2026 -0700
eliminate obsolete information
- src/hg/utils/otto/genArk/asmAlias/runUpdate.sh
- lines changed 9, context: html, text, full: html, text
1fc16d4ddd373fa0c4f3ae555feac1c053b5d012 Wed Sep 9 16:55:28 2026 -0700
re-enable otto creation of new asmAlias table refs #38082
- src/hg/utils/otto/ottoMonitor/README
- lines changed 86, context: html, text, full: html, text
36668c67d3b1b7ef9da683a2473ab1dea6be6709 Mon Sep 7 11:52:39 2026 -0700
otto: a daily check that every otto job is still running, refs #38101
Otto jobs are silent when the source has published nothing, which is the
design and also why a job that stops running is invisible. #38280 is the
case that prompted this: a daily job whose source URL had disappeared ran
about 1,100 times over three years without a word.
The monitor asks one question per job, did it run when it was supposed to,
and answers it from a run stamp, meaning something the job leaves behind
whether or not the data changed. Where a job leaves nothing it is reported
as blind rather than as passing, so the gap stays visible.
A late job is not automatically a bug, so a late job with a source URL gets
that URL fetched. A dead source has to fail twice in a row before it becomes
a ticket. A live source means the failure was something else, and that files
the same day.
Filing is off unless --file is given, and the script is silent when every
job is on time.
ottoMonitorStamps.tsv carries the per-job run stamp. It comes from the
survey in /hive/groups/browser/redmineNotes/38101/claude/.
- lines changed 13, context: html, text, full: html, text
d1444e2ca8e21460228433810b1a6703c6520db1 Mon Sep 7 12:02:02 2026 -0700
otto: keep the two surveys beside the monitor that reads them, refs #38101
The monitor's stamp table is the machine-readable half of a survey that
lived only in the redmineNotes directory, which is not a repository. The
reasoning behind each of the forty globs, and the measured curl shape
behind each source URL, were therefore one copy on /hive.
Both surveys now sit beside the script. Read ottoFailureSignatures.tsv
before changing a stamp glob: it says what each job writes and when, which
is the difference between a stamp that tracks every run and one that only
moves when the data changes.
- lines changed 10, context: html, text, full: html, text
383ff66de9a1a37635a1d079cccdeb9842056524 Tue Sep 8 09:22:26 2026 -0700
ottoMonitor: correct the job counts in the README and the survey header, refs #38101
The README's "WHAT IT ASKS" section said fifteen jobs write a log or a named
file, nine leave only a directory mtime, and eight leave nothing. Those add to
32, but the monitor watches 40. The real split is twenty-four, eight and
eight, which is what ottoMonitor.py -v reports. The old "fifteen" counted only
the external jobs with a positive stamp and dropped the nine internal ones that
also write a per-run file. The old "nine" came from counting the "dir mtime"
notes in ottoMonitorStamps.tsv, which catches ottoGitVsHive, but that job is
blind, not directory-mtime. The same off-by-one is fixed in the traps section.
The two surveys cover all 47 jobs in otto.crontab while the monitor watches 40,
so the README now says where the other seven went: they are the Cell Browser
jobs, marked monitor=no in ottoOwners.tsv.
ottoFailureSignatures.tsv opened by saying the 18 internal jobs were not
surveyed yet, three sections above a fully surveyed section for exactly those
18 jobs. The sentence was left over from the draft before that section was
added.
Also drop a dead clause in sourceIsUp(). "code == 226 or code == 0 and False"
reduces to "code == 226", because and binds tighter than or, so it read as if a
curl that could not connect were handled specially when it was not. Behavior
is unchanged: curl() returns 0 when curl itself failed, and 0 means the source
did not answer.
These are the two items the 2026-09-08 code review asked for, both in files
that tell a future editor to read them before touching a stamp glob.
- lines changed 17, context: html, text, full: html, text
52357be3947a4643b735276094b8d01da8e3f8f0 Tue Sep 8 10:30:02 2026 -0700
ottoMonitor: check the last closed grace window, and let a job be owned by whoever is on duty
Three changes, all from comments on the ticket.
Lou: civic has no individual owner, so it belongs to the otto person. The owner
column of ottoOwners.tsv now accepts ottoOnDuty, which the monitor resolves from
the ottoOnDuty header at run time, so the rotation stays a one-line edit. The
ticket body says the job has no individual owner, and the same person is not
added as a watcher twice when the owner is also the person on duty.
The grace window is now measured back from the last scheduled time whose window
has already closed, instead of forward from the latest scheduled time. Written
the other way, a daily job scheduled fewer than graceHours before the monitor's
own 12:15 run could never be reported late, because every check landed inside a
fresh window. Six of the forty jobs were in that hole: clinGen, genArkPushRR,
grcIncidentDb, liftRequest, omim and pubtatorDbSnp.
Max: a uniprot run can take days, and how long depends on the size of the
release. Its stamp is created by a > redirect when the run starts, so the grace
does not have to cover the run length, and a fresh stamp does not mean the run
worked. That limit is now written down in the stamps table and the README, with
the live case: the uniprot run dies after about 37 minutes on a missing lxml and
has produced no output since January 2025, while the monitor reads it as on
time.
refs #38101
- src/hg/utils/otto/ottoMonitor/ottoFailureSignatures.tsv
- lines changed 78, context: html, text, full: html, text
d1444e2ca8e21460228433810b1a6703c6520db1 Mon Sep 7 12:02:02 2026 -0700
otto: keep the two surveys beside the monitor that reads them, refs #38101
The monitor's stamp table is the machine-readable half of a survey that
lived only in the redmineNotes directory, which is not a repository. The
reasoning behind each of the forty globs, and the measured curl shape
behind each source URL, were therefore one copy on /hive.
Both surveys now sit beside the script. Read ottoFailureSignatures.tsv
before changing a stamp glob: it says what each job writes and when, which
is the difference between a stamp that tracks every run and one that only
moves when the data changes.
- lines changed 2, context: html, text, full: html, text
383ff66de9a1a37635a1d079cccdeb9842056524 Tue Sep 8 09:22:26 2026 -0700
ottoMonitor: correct the job counts in the README and the survey header, refs #38101
The README's "WHAT IT ASKS" section said fifteen jobs write a log or a named
file, nine leave only a directory mtime, and eight leave nothing. Those add to
32, but the monitor watches 40. The real split is twenty-four, eight and
eight, which is what ottoMonitor.py -v reports. The old "fifteen" counted only
the external jobs with a positive stamp and dropped the nine internal ones that
also write a per-run file. The old "nine" came from counting the "dir mtime"
notes in ottoMonitorStamps.tsv, which catches ottoGitVsHive, but that job is
blind, not directory-mtime. The same off-by-one is fixed in the traps section.
The two surveys cover all 47 jobs in otto.crontab while the monitor watches 40,
so the README now says where the other seven went: they are the Cell Browser
jobs, marked monitor=no in ottoOwners.tsv.
ottoFailureSignatures.tsv opened by saying the 18 internal jobs were not
surveyed yet, three sections above a fully surveyed section for exactly those
18 jobs. The sentence was left over from the draft before that section was
added.
Also drop a dead clause in sourceIsUp(). "code == 226 or code == 0 and False"
reduces to "code == 226", because and binds tighter than or, so it read as if a
curl that could not connect were handled specially when it was not. Behavior
is unchanged: curl() returns 0 when curl itself failed, and 0 means the source
did not answer.
These are the two items the 2026-09-08 code review asked for, both in files
that tell a future editor to read them before touching a stamp glob.
- src/hg/utils/otto/ottoMonitor/ottoMonitor.py
- lines changed 431, context: html, text, full: html, text
36668c67d3b1b7ef9da683a2473ab1dea6be6709 Mon Sep 7 11:52:39 2026 -0700
otto: a daily check that every otto job is still running, refs #38101
Otto jobs are silent when the source has published nothing, which is the
design and also why a job that stops running is invisible. #38280 is the
case that prompted this: a daily job whose source URL had disappeared ran
about 1,100 times over three years without a word.
The monitor asks one question per job, did it run when it was supposed to,
and answers it from a run stamp, meaning something the job leaves behind
whether or not the data changed. Where a job leaves nothing it is reported
as blind rather than as passing, so the gap stays visible.
A late job is not automatically a bug, so a late job with a source URL gets
that URL fetched. A dead source has to fail twice in a row before it becomes
a ticket. A live source means the failure was something else, and that files
the same day.
Filing is off unless --file is given, and the script is silent when every
job is on time.
ottoMonitorStamps.tsv carries the per-job run stamp. It comes from the
survey in /hive/groups/browser/redmineNotes/38101/claude/.
- lines changed 3, context: html, text, full: html, text
383ff66de9a1a37635a1d079cccdeb9842056524 Tue Sep 8 09:22:26 2026 -0700
ottoMonitor: correct the job counts in the README and the survey header, refs #38101
The README's "WHAT IT ASKS" section said fifteen jobs write a log or a named
file, nine leave only a directory mtime, and eight leave nothing. Those add to
32, but the monitor watches 40. The real split is twenty-four, eight and
eight, which is what ottoMonitor.py -v reports. The old "fifteen" counted only
the external jobs with a positive stamp and dropped the nine internal ones that
also write a per-run file. The old "nine" came from counting the "dir mtime"
notes in ottoMonitorStamps.tsv, which catches ottoGitVsHive, but that job is
blind, not directory-mtime. The same off-by-one is fixed in the traps section.
The two surveys cover all 47 jobs in otto.crontab while the monitor watches 40,
so the README now says where the other seven went: they are the Cell Browser
jobs, marked monitor=no in ottoOwners.tsv.
ottoFailureSignatures.tsv opened by saying the 18 internal jobs were not
surveyed yet, three sections above a fully surveyed section for exactly those
18 jobs. The sentence was left over from the draft before that section was
added.
Also drop a dead clause in sourceIsUp(). "code == 226 or code == 0 and False"
reduces to "code == 226", because and binds tighter than or, so it read as if a
curl that could not connect were handled specially when it was not. Behavior
is unchanged: curl() returns 0 when curl itself failed, and 0 means the source
did not answer.
These are the two items the 2026-09-08 code review asked for, both in files
that tell a future editor to read them before touching a stamp glob.
- lines changed 36, context: html, text, full: html, text
52357be3947a4643b735276094b8d01da8e3f8f0 Tue Sep 8 10:30:02 2026 -0700
ottoMonitor: check the last closed grace window, and let a job be owned by whoever is on duty
Three changes, all from comments on the ticket.
Lou: civic has no individual owner, so it belongs to the otto person. The owner
column of ottoOwners.tsv now accepts ottoOnDuty, which the monitor resolves from
the ottoOnDuty header at run time, so the rotation stays a one-line edit. The
ticket body says the job has no individual owner, and the same person is not
added as a watcher twice when the owner is also the person on duty.
The grace window is now measured back from the last scheduled time whose window
has already closed, instead of forward from the latest scheduled time. Written
the other way, a daily job scheduled fewer than graceHours before the monitor's
own 12:15 run could never be reported late, because every check landed inside a
fresh window. Six of the forty jobs were in that hole: clinGen, genArkPushRR,
grcIncidentDb, liftRequest, omim and pubtatorDbSnp.
Max: a uniprot run can take days, and how long depends on the size of the
release. Its stamp is created by a > redirect when the run starts, so the grace
does not have to cover the run length, and a fresh stamp does not mean the run
worked. That limit is now written down in the stamps table and the README, with
the live case: the uniprot run dies after about 37 minutes on a missing lxml and
has produced no output since January 2025, while the monitor reads it as on
time.
refs #38101
- src/hg/utils/otto/ottoMonitor/ottoMonitorStamps.tsv
- lines changed 60, context: html, text, full: html, text
36668c67d3b1b7ef9da683a2473ab1dea6be6709 Mon Sep 7 11:52:39 2026 -0700
otto: a daily check that every otto job is still running, refs #38101
Otto jobs are silent when the source has published nothing, which is the
design and also why a job that stops running is invisible. #38280 is the
case that prompted this: a daily job whose source URL had disappeared ran
about 1,100 times over three years without a word.
The monitor asks one question per job, did it run when it was supposed to,
and answers it from a run stamp, meaning something the job leaves behind
whether or not the data changed. Where a job leaves nothing it is reported
as blind rather than as passing, so the gap stays visible.
A late job is not automatically a bug, so a late job with a source URL gets
that URL fetched. A dead source has to fail twice in a row before it becomes
a ticket. A live source means the failure was something else, and that files
the same day.
Filing is off unless --file is given, and the script is silent when every
job is on time.
ottoMonitorStamps.tsv carries the per-job run stamp. It comes from the
survey in /hive/groups/browser/redmineNotes/38101/claude/.
- lines changed 2, context: html, text, full: html, text
d1444e2ca8e21460228433810b1a6703c6520db1 Mon Sep 7 12:02:02 2026 -0700
otto: keep the two surveys beside the monitor that reads them, refs #38101
The monitor's stamp table is the machine-readable half of a survey that
lived only in the redmineNotes directory, which is not a repository. The
reasoning behind each of the forty globs, and the measured curl shape
behind each source URL, were therefore one copy on /hive.
Both surveys now sit beside the script. Read ottoFailureSignatures.tsv
before changing a stamp glob: it says what each job writes and when, which
is the difference between a stamp that tracks every run and one that only
moves when the data changes.
- lines changed 11, context: html, text, full: html, text
52357be3947a4643b735276094b8d01da8e3f8f0 Tue Sep 8 10:30:02 2026 -0700
ottoMonitor: check the last closed grace window, and let a job be owned by whoever is on duty
Three changes, all from comments on the ticket.
Lou: civic has no individual owner, so it belongs to the otto person. The owner
column of ottoOwners.tsv now accepts ottoOnDuty, which the monitor resolves from
the ottoOnDuty header at run time, so the rotation stays a one-line edit. The
ticket body says the job has no individual owner, and the same person is not
added as a watcher twice when the owner is also the person on duty.
The grace window is now measured back from the last scheduled time whose window
has already closed, instead of forward from the latest scheduled time. Written
the other way, a daily job scheduled fewer than graceHours before the monitor's
own 12:15 run could never be reported late, because every check landed inside a
fresh window. Six of the forty jobs were in that hole: clinGen, genArkPushRR,
grcIncidentDb, liftRequest, omim and pubtatorDbSnp.
Max: a uniprot run can take days, and how long depends on the size of the
release. Its stamp is created by a > redirect when the run starts, so the grace
does not have to cover the run length, and a fresh stamp does not mean the run
worked. That limit is now written down in the stamps table and the README, with
the live case: the uniprot run dies after about 37 minutes on a missing lxml and
has produced no output since January 2025, while the monitor reads it as on
time.
refs #38101
- src/hg/utils/otto/ottoMonitor/ottoSourceUrls.tsv
- lines changed 70, context: html, text, full: html, text
d1444e2ca8e21460228433810b1a6703c6520db1 Mon Sep 7 12:02:02 2026 -0700
otto: keep the two surveys beside the monitor that reads them, refs #38101
The monitor's stamp table is the machine-readable half of a survey that
lived only in the redmineNotes directory, which is not a repository. The
reasoning behind each of the forty globs, and the measured curl shape
behind each source URL, were therefore one copy on /hive.
Both surveys now sit beside the script. Read ottoFailureSignatures.tsv
before changing a stamp glob: it says what each job writes and when, which
is the difference between a stamp that tracks every run and one that only
moves when the data changes.
- src/hg/utils/otto/uniprot/README.txt
- lines changed 130, context: html, text, full: html, text
af613a331e6839c6513c3e366abcb67af0fe8386 Wed Sep 9 06:47:14 2026 -0700
UniProt otto: get the monthly update running again and make a stalled run visible
The monthly UniProt job had produced nothing since January 2025. The tracks
served release 2024_06 while the download sitting on disk was at 2026_02, on
every assembly the job builds.
Cause: uniprotToTab appended a personal conda site-packages directory to
sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A
venv's python is only a symlink to the system one, so when hgwdev moved to
python 3.9 the compiled lxml in there stopped loading and every run died at the
parse step. Removed both. The system python3 has lxml from python3-lxml and the
two are upgraded together, so there is nothing left here to go stale. Verified
by parsing real 2026_02 records under python 3.9 with lxml 5.4.
Why nobody noticed for nineteen months:
- doUpdate.sh read $? after an intervening echo, so it captured the echo's exit
code and mailed "Big Uniprot update OK" every month while the job was dying.
It now reads the real exit code, says FAILED, prints the tail of the log and
exits nonzero. A month with no new UniProt release stays silent, which is the
normal otto behaviour, so silence again means "nothing to do".
- The logs were overwritten on every run, so a failure left no trace on disk.
doUpdate.sh now appends one line per run to runLog.txt, which is never
truncated, and keeps a failing log as lastFail.log.
- version.txt in each bigBed directory was rewritten on every run even when the
release string was identical. That is the file the trackDb dataVersion setting
shows, and its date is what people check to decide whether a pipeline is still
alive, so a stalled track could look freshly updated. It is now written only
when the release actually changes.
Also, so this cannot come back:
- doUniprot checks that uniprotToTab can start before the download, instead of
finding out 35 minutes later.
- pylint on hgwdev is itself pinned to pythons that no longer exist, so
"make install" aborted on its first line and could not be used. Replaced with
a syntax check that needs nothing but python3; pylint stays best-effort.
- uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from
/hive/data/outside/otto/uniprot without being in the makefile's copy list.
The tree copy of uniprotToTab was still python 2 from 2021. All are now
listed and in sync, and "make diff" reports drift.
- Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh
and the pslMap -inType/-mapType flags.
refs #38300
- lines changed 44, context: html, text, full: html, text
bee41315e53da6fb5671f96302233c655906dfd5 Wed Sep 9 06:56:49 2026 -0700
UniProt otto: rebuild the venv rather than relying on a system lxml
Correction to the previous commit: hgwdev has no system-wide lxml at all. The
import I tested was resolving to my own ~/.local/lib/python3.9/site-packages,
which cron never sees, because it runs the pipeline as otto.
So the environment is a virtualenv again, but a reproducible one. makeVenv.sh
deletes venv/ and rebuilds it from /usr/bin/python3, installs lxml, opens up the
permissions for otto, and then checks that lxml imports with an empty environment
so we know the venv stands on its own instead of borrowing from whoever ran it.
Built with --copies, so venv/bin/python is a real copy rather than a symlink that
would silently follow a system python upgrade while its compiled modules stayed
behind.
doUpdate.sh activates venv/ again and says to run makeVenv.sh if it is missing or
if the parser will not start. Verified: /usr/bin/python3 without per-user packages
cannot import lxml, the venv can, and after activation the parser runs and
converts real 2026_02 records.
Also shortened the README to how the pipeline is started and how it works, and
trimmed the history out of the code comments, leaving the ticket as the pointer.
refs #38300
- lines changed 6, context: html, text, full: html, text
d033cea2063e9362949baf5b4d8b837597173a0d Thu Sep 10 05:16:05 2026 -0700
Address the code review of the Sep 9 commits
Faceted composite: text that comes from a hub - a metadata column's
description, the name and title of a data type, and the values quoted back in
the "could not load the metadata" row - is put on the page as text rather than
as markup. The three places built their markup from template strings, so a
value carrying angle brackets or a quote was read as HTML: the column
description now goes through the shared htmlEncode() once where the header is
parsed, and the other two build their elements as nodes. The error row reads
better for it as well, since a value with brackets in it used to disappear from
the message that was meant to show it.
The saved UI state keys on the assembly as well as the metadata id. localStorage
is per-origin, so two assemblies whose tracks share a name were sharing one
entry, and a row order dragged on one came back on the other over a different
set of samples. hgTrackUi passes the database down for it. State saved under the
old key is dropped, which costs a facet selection or a page length.
Imprinting: the five subtrack description pages link back to the container as
hgTrackUi?db=$db&g=$parentTrack, without the hgsid. Native trackDb html is
substituted by hgTrackDb as it loads the table, where there is no cart, so
${hgsid} came out empty and the link read 'hgsid=&g=...'. Matches what the
Fiber-seq pages already do. The makeDoc note that described the old form is
updated with the reason.
UniProt otto: README.txt lists all eight things that reach runLog.txt. It had
four, and was missing LOCKED, along with PREFLIGHT-FAIL, END and INTERRUPTED.
refs #36210
refs #37599
refs #38300
- lines changed 1, context: html, text, full: html, text
55a768d2e0ced94dc3ba7ab322b24daa67ea3570 Fri Sep 11 10:09:16 2026 -0700
uniprot otto: resolve the 2bit and chrom.sizes for every kind of assembly
The first GenArk run stopped on hs1 with "expected exactly one chrom.sizes file in
/gbdb/hs1/hubs, found 0". hs1 is served as a hub but keeps its 2bit at
/gbdb/hs1/hs1.2bit and its chrom.sizes at /hive/data/genomes/hs1/chrom.sizes,
exactly where a classic assembly keeps them; only a real GenArk assembly keeps
them in the hub directory.
twoBitFname now keys on isGenArk rather than on being a hub, and a matching
chromSizesFile does the same, so both are resolved the same way everywhere. The
transcript building still uses the bigBed path for any hub assembly, because hs1
has no MySQL tables either, but it asks these two helpers for the supporting files.
Checked that all four resolve to files that exist: hs1 and hg38 to the classic
locations, mPanPan1_v2.0 and GRCz12ab to their hub directories.
Also fixed the README, which documented the option to limit the run as --onlyDbs.
It is --dbs; --onlyDbs is the internal dest name and the run fails with "no such
option".
refs #38300
- src/hg/utils/otto/uniprot/bigPslUniprot.as
- lines changed 1, context: html, text, full: html, text
d2be4681e2c8493bf9dfe755037e76d2b784e9ff Thu Sep 10 05:24:00 2026 -0700
uniprot otto: bigPslUniprot.as was missing the isMain field
The committed .as declared 46 fields while doUniprot writes 47: pslToBigPsl()
appends isMainStr as the last column (doUniprot line 1331, "primary sequence" or
"alternative isoform"), and trackDb/uniprot.ra refers to it twice as
"skipFields isMain". The field has been missing from this file since at least
2021; the copy in /hive/data/outside/otto/uniprot had been fixed by hand and
never committed, which is why runs kept working and nobody noticed.
I broke the running update by rsyncing this stale copy over the working one, and
bedToBigBed stopped with "Expecting 46 words line 1 ... got 47" after the parse
had already finished. Adding the field to the tree copy fixes both the immediate
failure and the drift.
Verified against the input that failed: bedToBigBed now writes 44142 records with
47 fields and both extra indexes, every record has exactly 47 columns, isMain
reads back as 21362 "primary sequence" and 22780 "alternative isoform", and all
eleven fields named by uniprot.ra's labelFields, mouseOver and skipFields resolve
in the result.
refs #38300
- src/hg/utils/otto/uniprot/doUniprot
- lines changed 41, context: html, text, full: html, text
af613a331e6839c6513c3e366abcb67af0fe8386 Wed Sep 9 06:47:14 2026 -0700
UniProt otto: get the monthly update running again and make a stalled run visible
The monthly UniProt job had produced nothing since January 2025. The tracks
served release 2024_06 while the download sitting on disk was at 2026_02, on
every assembly the job builds.
Cause: uniprotToTab appended a personal conda site-packages directory to
sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A
venv's python is only a symlink to the system one, so when hgwdev moved to
python 3.9 the compiled lxml in there stopped loading and every run died at the
parse step. Removed both. The system python3 has lxml from python3-lxml and the
two are upgraded together, so there is nothing left here to go stale. Verified
by parsing real 2026_02 records under python 3.9 with lxml 5.4.
Why nobody noticed for nineteen months:
- doUpdate.sh read $? after an intervening echo, so it captured the echo's exit
code and mailed "Big Uniprot update OK" every month while the job was dying.
It now reads the real exit code, says FAILED, prints the tail of the log and
exits nonzero. A month with no new UniProt release stays silent, which is the
normal otto behaviour, so silence again means "nothing to do".
- The logs were overwritten on every run, so a failure left no trace on disk.
doUpdate.sh now appends one line per run to runLog.txt, which is never
truncated, and keeps a failing log as lastFail.log.
- version.txt in each bigBed directory was rewritten on every run even when the
release string was identical. That is the file the trackDb dataVersion setting
shows, and its date is what people check to decide whether a pipeline is still
alive, so a stalled track could look freshly updated. It is now written only
when the release actually changes.
Also, so this cannot come back:
- doUniprot checks that uniprotToTab can start before the download, instead of
finding out 35 minutes later.
- pylint on hgwdev is itself pinned to pythons that no longer exist, so
"make install" aborted on its first line and could not be used. Replaced with
a syntax check that needs nothing but python3; pylint stays best-effort.
- uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from
/hive/data/outside/otto/uniprot without being in the makefile's copy list.
The tree copy of uniprotToTab was still python 2 from 2021. All are now
listed and in sync, and "make diff" reports drift.
- Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh
and the pslMap -inType/-mapType flags.
refs #38300
- lines changed 12, context: html, text, full: html, text
bee41315e53da6fb5671f96302233c655906dfd5 Wed Sep 9 06:56:49 2026 -0700
UniProt otto: rebuild the venv rather than relying on a system lxml
Correction to the previous commit: hgwdev has no system-wide lxml at all. The
import I tested was resolving to my own ~/.local/lib/python3.9/site-packages,
which cron never sees, because it runs the pipeline as otto.
So the environment is a virtualenv again, but a reproducible one. makeVenv.sh
deletes venv/ and rebuilds it from /usr/bin/python3, installs lxml, opens up the
permissions for otto, and then checks that lxml imports with an empty environment
so we know the venv stands on its own instead of borrowing from whoever ran it.
Built with --copies, so venv/bin/python is a real copy rather than a symlink that
would silently follow a system python upgrade while its compiled modules stayed
behind.
doUpdate.sh activates venv/ again and says to run makeVenv.sh if it is missing or
if the parser will not start. Verified: /usr/bin/python3 without per-user packages
cannot import lxml, the venv can, and after activation the parser runs and
converts real 2026_02 records.
Also shortened the README to how the pipeline is started and how it works, and
trimmed the history out of the code comments, leaving the ticket as the pointer.
refs #38300
- lines changed 6, context: html, text, full: html, text
efe79575f0b546414196fb3b8a8871bf5f7611fc Wed Sep 9 07:16:04 2026 -0700
UniProt otto: stop curl writing its progress bar into the run log
The gene2refseq download wrote a few hundred lines of percentages into
lastRun.log, which buries the lines that matter. --no-progress-meter drops the
bar but keeps errors, unlike the plain -s that was removed here earlier for
exactly that reason, and --fail turns an HTTP error into a curl error instead of
piping NCBI's error page into zcat and failing with 'not in gzip format'.
Not yet installed to /hive: a run is in progress. refs #38300
- lines changed 11, context: html, text, full: html, text
af670dcecc6911452f3e1a5c38cfea1ef0067978 Wed Sep 9 07:31:07 2026 -0700
UniProt otto: submit the parasol batch on hgwdev, drop the ssh to ku
The protein-to-transcript BLAST batch was submitted with
ssh ku "cd <workdir> && para make jobList"
ku has been decommissioned for about two years. It still resolves in DNS, but
"ssh ku" is an immediate "No route to host", so a run would have died at the
mapping stage, which comes only after days of XML parsing.
hgwdev is the parasol head node now, so "para make" runs here with no ssh hop.
Verified with a one-job batch using the same bare-command jobList this script
writes: it lands on a compute node with its cwd set to the batch directory, the
same as the old ku jobs did. Also checked from a compute node that tclsh and the
blast-2.2.16 blastall/formatdb that mapUniprot_doBlast needs are still there.
The cluster name is gone rather than redirected. doUniprot no longer reads
/cluster/bin/scripts/cluster.txt, a file written in 2017 that still says "ku" and
that nothing else in the tree read, and makeUniProtPsl.sh no longer takes a head
node argument, so its positional parameters shift down by one.
refs #38300
- lines changed 49, context: html, text, full: html, text
5783988b48ed0722ce2398b50bc1a422911c1fa7 Fri Sep 11 06:02:22 2026 -0700
uniprot otto: teach the gene-model search about GenArk assemblies
findBestGeneTable only knew how to look for MySQL tables, so on a GenArk assembly
it found nothing and fell through to BLAT, which then died on a 2bit path that
does not exist for a hub assembly. GenArk keeps its gene models as bigBed files
in the hub instead.
Search order, best first:
catGenes Comparative Annotation Toolkit, shipped as a contrib collection
(track hprcCatGenes, contrib/hprc2annot/catGenes.bb)
ncbiRefSeq the GenArk RefSeq gene track, bbi/*.ncbiRefSeq.bb
ncbiGene the annotation the submitter sent to GenBank with the assembly
augustus ab initio, present nearly everywhere, so it is the last resort
Matched by glob rather than by constructed name: files under bbi/ carry the full
asmId including its assembly-name suffix, while the hub directory is named with
the short accession. A dbDb row is recognised as GenArk by nibPath starting with
"hub:", cached so the lookup happens once per db.
Checked against the assemblies that are actually blocking the update: the twelve
GCF ones resolve to ncbiRefSeq, GRCz12ab and calJac240_pri have no RefSeq or
submitted annotation and fall to augustus, and on an HPRC assembly that carries
both, catGenes wins over augustus as intended. Classic assemblies are untouched -
genArkHubDir returns None for them and hg38/hg19/mm39/panPan3/rn6 still resolve
exactly as before through the MySQL path.
This is the search only. Building from these still needs the transcript fasta and
PSL to come from the hub rather than from MySQL. refs #38300
- lines changed 54, context: html, text, full: html, text
dcf390fefe68c32f26bd32717bd858190e8875a0 Fri Sep 11 06:06:58 2026 -0700
uniprot otto: build the transcript files for a GenArk assembly from the hub
A GenArk assembly has no MySQL database, so everything makeTranscriptFiles did
had to come from somewhere else. It all exists in the hub:
bigGenePredToGenePred gene bigBed -> transcripts.gp
genePredToFakePsl -chromSize=<hub> -> transcripts.psl
getRnaPred -genomeSeqs=<2bit> -> transcripts.fa
Both of those options exist precisely so the tools do not have to go through
chromInfo, so no database is involved anywhere in the chain.
Also made hub-aware, in the same call chain:
- chrom.sizes for the bigPsl and bigBed steps, which used
/hive/data/genomes/<db>/chrom.sizes and there is no such directory
- getTransIds, which read the transcript IDs out of a MySQL table; the fasta we
just built holds them all, so read them from there as the refGene case does
- writeMapDesc, which read ncbiRefSeqVersion.txt; date the models by the gene
bigBed we actually read instead
Helpers genArkTwoBit and genArkChromSizes glob for the single matching file and
abort if there is not exactly one, since the hub names those with the short
accession while the files under bbi/ carry the full asmId.
Verified on bonobo mPanPan1_v2.0: 95418 transcripts, identical ID sets across the
fasta, genePred and PSL; getTransIds returns those 95418 without touching MySQL;
writeMapDesc dates the models 2025-06-02, which matches the date embedded in the
build's own ncbiRefSeq GTF filename. hg38 still reads its real
ncbiRefSeqVersion.txt and is otherwise untouched.
Still to do: the outputs. /gbdb/<db>/uniprot and a trackDb .ra stanza are not how
a GenArk assembly is served. refs #38300
- lines changed 82, context: html, text, full: html, text
953e29496b012cc36ee7933ce94cefea2f66ec38 Fri Sep 11 09:21:22 2026 -0700
uniprot otto: align proteins with miniprot instead of BLAT or Augustus
Where an assembly has no gene models worth mapping through, the pipeline used
"blat -q=prot -t=dnax", which is slow enough that hs1 was excluded from the whole
job over it. On GenArk assemblies the alternative was Augustus, which is an ab
initio prediction, so mapping UniProt through it stacks its errors on top of ours.
Aligning the proteins straight to the genome avoids both.
miniprot 0.18 built from github into
/hive/data/outside/otto/uniprot/bin/miniprot, linked against system libraries
only and runnable by otto.
miniprotProteins() replaces blatProteinsKeepBest(). miniprot reads fasta rather
than 2bit, so the genome is unpacked to a temp file and removed again once the
alignment is done. Its GFF3 needs some care before gff3ToGenePred will take it:
the ##PAF meta lines and the capitalised Rank/Identity attributes have to go, and
alignments are named MP000001 with the UniProt accession hidden in Target=, so the
ids are rewritten to the accession. The uniquifying suffix that GFF3 requires is
stripped again afterwards, leaving the bare accession as qName, which is what the
rest of the pipeline keys on.
Augustus is dropped from the GenArk gene source list entirely, so the order is now
catGenes, ncbiRefSeq, ncbiGene, then miniprot. The classic "blat" fallback becomes
miniprot too.
Checked on real data: 300 bonobo UniProt proteins against one 227 Mb chromosome
give 138 alignments over 102 distinct proteins, pslCheck reports 138 checked and 0
failed, the temp genome fasta is cleaned up, and the aligner version is recorded in
the mapping stats. GRCz12ab and calJac240_pri, the two assemblies that had nothing
but Augustus, now resolve to miniprot.
refs #38300
- lines changed 38, context: html, text, full: html, text
d886333094fd66bac7a872f685c3464a6d667244 Fri Sep 11 09:52:30 2026 -0700
uniprot otto: run miniprot on the cluster, with the RAM and CPUs it actually needs
para's default RAM per job is the node's RAM divided by its CPU count, which for a
16 CPU job is far less than miniprot wants, and without -cpu parasol would pack
more of these onto a node than it has cores for. Both are now passed.
The RAM figure is measured rather than guessed. Peak RSS is linear in genome size
at about 11 GB per Gb of sequence (2.60 GB for 227 Mb, 4.69 GB for 424 Mb, 8.93 GB
for 811 Mb) and is flat in the thread count, because the index is built once and
shared: 2.49 GB at -t 1 against 2.60 GB at -t 16 on the same sequence. So the
reservation is sized off the genome alone, with headroom, and floored at 8g.
Zebrafish comes out at 21g and marmoset at 42g.
Note that -cpu is a scheduling reservation, not a limit: the job still sees every
core on the node. It stops parasol oversubscribing the machine, and pairing it
with miniprot's own -t is what makes the two agree.
Checked with a real one-job batch at -cpu=16 -ram=42g: accepted, ran on a compute
node, and the binary is reachable from there.
refs #38300
- lines changed 80, context: html, text, full: html, text
edf45d7b82651acd7bb9cf00cc492e9b19936f1f Fri Sep 11 09:58:18 2026 -0700
uniprot otto: write a GenArk contrib collection instead of /gbdb symlinks
A GenArk assembly is not served out of /gbdb/<db>/, so the symlinks and the
trackDb .ra stanza that classic assemblies get do not apply to it. Its files
belong in a contrib collection.
For a GenArk assembly the pipeline now writes
contrib/uniprot/<accession>/ with the bigBeds and a trackDb.txt, and skips three
things that only make sense for a classic assembly: the /gbdb/<db>/uniprot
symlinks, the version.txt symlink, and the goldenPath archive copy, whose download
path does not exist for a hub assembly.
The trackDb comes from the same template the archive hub uses, with three changes:
track names must not carry the release or they would change every month and break
saved sessions; the data files sit beside trackDb.txt rather than in a per-release
subdirectory; and dataVersion has to be the literal release string because a
contrib trackDb cannot read a /gbdb path.
It deliberately stops there. Nothing is symlinked into the GenArk build
directories and no hub.txt is touched: installing the collection is a separate,
deliberate step with "genark addContrib uniprot", not something a monthly data
update should do on its own.
Checked by writing a collection for one assembly and running the genark tool's
--dry-run over it: all 16 bigDataUrls resolve to files that are present and
non-empty, the html path resolves to the shared docs page, and the tool reports it
would wire the block into alpha.hub.txt only. The shared docs page is the trackDb
description with the archive-hub paragraph and the /gbdb download links removed,
since neither exists for a GenArk assembly.
refs #38300
- lines changed 42, context: html, text, full: html, text
89e1e4c6ebefe29b7219fc98417b73b6a1755f05 Fri Sep 11 10:07:02 2026 -0700
uniprot otto: handle hub assemblies that are not GenArk, and bring hs1 back in
Not every assembly with a "hub:" nibPath is a GenArk assembly. hs1 is served as a
hub from /gbdb/hs1/hubs but keeps its track files in /gbdb/hs1/ exactly like a
classic assembly, so treating every hub as GenArk put its gene model search in the
wrong place and would have sent its output to a contrib collection it does not
belong in.
Split the two questions. genArkHubDir still answers "is this served as a hub, and
from where", and a new isGenArk asks the narrower question the output routing
actually cares about: is this under /gbdb/genark with the sharded layout. The
/gbdb symlinks, the version.txt symlink, the goldenPath archive and the contrib
collection now all key on isGenArk, so hs1 gets the classic treatment.
The gene model search now takes several globs per source and looks through both the
hub directory and /gbdb/<db>/, which covers both layouts: a GenArk assembly keeps
its models under bbi/ with CAT in a contrib collection, hs1 keeps them in
/gbdb/hs1/<trackName>/.
That makes hs1 work, so it comes out of notAutoDbs. It was excluded for having "no
good gene model" and because the BLAT protein search took forever; it actually has
CAT genes, 234903 of them in catLiftOffGenesV1, which is the best source we look
for, and BLAT is gone anyway.
Resolutions checked: hs1 -> catGenes from /gbdb/hs1/catLiftOffGenesV1, not GenArk;
mPanPan1_v2.0 -> ncbiRefSeq from its hub bbi/, GenArk; GRCz12ab -> miniprot, GenArk;
hg38 and panPan3 unchanged on the MySQL path.
refs #38300
- lines changed 19, context: html, text, full: html, text
55a768d2e0ced94dc3ba7ab322b24daa67ea3570 Fri Sep 11 10:09:16 2026 -0700
uniprot otto: resolve the 2bit and chrom.sizes for every kind of assembly
The first GenArk run stopped on hs1 with "expected exactly one chrom.sizes file in
/gbdb/hs1/hubs, found 0". hs1 is served as a hub but keeps its 2bit at
/gbdb/hs1/hs1.2bit and its chrom.sizes at /hive/data/genomes/hs1/chrom.sizes,
exactly where a classic assembly keeps them; only a real GenArk assembly keeps
them in the hub directory.
twoBitFname now keys on isGenArk rather than on being a hub, and a matching
chromSizesFile does the same, so both are resolved the same way everywhere. The
transcript building still uses the bigBed path for any hub assembly, because hs1
has no MySQL tables either, but it asks these two helpers for the supporting files.
Checked that all four resolve to files that exist: hs1 and hg38 to the classic
locations, mPanPan1_v2.0 and GRCz12ab to their hub directories.
Also fixed the README, which documented the option to limit the run as --onlyDbs.
It is --dbs; --onlyDbs is the internal dest name and the run fails with "no such
option".
refs #38300
- lines changed 16, context: html, text, full: html, text
155514c1985d0fdc25f4a33f4e1786f35b47b09d Fri Sep 11 11:11:49 2026 -0700
uniprot otto: make CAT transcript names unique before deriving the fasta and PSL
The hs1 run got through a 52 minute BLAST batch with no crashes and then stopped in
pslMap with
Error: inPsl RBMY1F-1 tSize (1887) != mapPsl RBMY1F-1 qSize (1718)
CAT names its transcripts after the source gene, so paralogs share a name. The hs1
CAT set has 777 duplicated names among 234903 transcripts, and RBMY1F-1 is three Y
chromosome paralogs of 1810, 1718 and 1887 bases. The protein-to-transcript
alignment and the transcript-to-genome alignment then disagree about which
transcript the name refers to and pslMap refuses to map. This is a property of CAT,
not of hub assemblies: the GenArk ncbiRefSeq sets use accessions and have no
duplicates at all.
Uniquify the names on the genePred before the fasta and the PSL are derived from
it, so the two can never disagree.
Verified against the real hs1 CAT data: 234903 rows and 234903 distinct names, no
duplicates in the genePred, fasta or PSL, the three RBMY1F-1 paralogs now separate,
and every one of the 234903 PSL rows has a qSize equal to the length of its own
fasta sequence, which is the invariant pslMap enforces. On bonobo ncbiRefSeq it is
a no-op: 95418 rows, no name changed.
refs #38300
- lines changed 4, context: html, text, full: html, text
f7a079f0d9b56429eb44109f1d05be2ab759294a Fri Sep 11 12:13:22 2026 -0700
uniprot otto: the duplicate-name count reported zero every time
The line that was meant to say how many CAT transcript names had been made unique
subtracted the distinct name count of the file it had just uniquified, which is
equal to its row count by construction, so the answer was always zero and the
message never appeared. The rename itself was working: the hs1 run has 3769 rows
carrying a -dup suffix and all 234903 names distinct.
Count the names in the input instead, and say which of the two numbers is which.
Verified against the real hs1 files: 234903 rows, 231134 distinct names before,
so it now reports 3769 renamed.
refs #38300
- lines changed 12, context: html, text, full: html, text
947dcb525b76d579bd6ab425cf71a221d9bab347 Fri Sep 11 12:50:09 2026 -0700
uniprot otto: give the unfiltered gene-track case a map source
hs1 cleared pslMap after the duplicate-name fix, ran the full 91 minute alignment,
and then died writing the bigPsl with
KeyError: 'default' at mapSource = accToMapSource["default"]
buildSelectFile returns an empty dict when no UniProt cross-reference matches the
ids in the gene track, and pslToBigPsl looks up accToMapSource["default"] for every
protein it writes. That branch was unreachable before: every gene table
findBestGeneTable could return was handled somewhere else, with augustus and the
direct protein alignment each setting their own default. catGenes and ncbiGene are
the first to reach it.
It now returns {"default":"best"}, the same answer the augustus case gives, which
is the honest description: the alignment is the best match we found, we just cannot
name the transcript evidence behind it.
Hardened the lookup as well. protMapSource on the supported path only ever gets
per-accession entries and never a "default" key, so any accession missing from it
would have raised the same KeyError on hg38. It now falls back to "best" rather
than throwing away hours of cluster work over one protein.
Also reworded the log line, which said "No supported gene track found" while a
perfectly good gene track was in use; the missing thing is a cross-reference that
matches its ids.
refs #38300
- lines changed 7, context: html, text, full: html, text
75e828960283291546d2c1a27845e2cf3823adcd Mon Sep 14 05:29:02 2026 -0700
uniprot otto: the miniprot cluster job needs absolute paths
GRCz12ab failed with the parasol job crashing four times, return 1, no output.
The wrapper I wrote ran
miniprot -t 16 --gff protToGenome/GRCz12ab/.../genome.fa fasta/7955.fa > $1
and a parasol job runs with its working directory set to the batch directory, not
to the directory the pipeline runs in, so neither input existed from the job's
point of view. The BLAST batch next door gets away with relative paths because it
cds into its own workdir and its jobList is written relative to that; this batch
directory sits a level deeper and its paths were relative to the otto root.
Every path in the wrapper, the jobList command and the output check is now
absolute.
Verified on the cluster against the real 1.48 Gb zebrafish genome: successful
batch, a 195 MB GFF with 93518 mRNA records.
refs #38300
- lines changed 12, context: html, text, full: html, text
b8320ab5d2dd4d520742faedc2def3144595c450 Mon Sep 14 05:49:34 2026 -0700
uniprot otto: writeMapDesc crashed on a hub assembly with no gene models
GRCz12ab got through miniprot on the cluster and then died writing its lift info:
os.path.getmtime(geneBb) -> stat on None
The hub branch dates the gene models by the bigBed that was read, but an assembly
that has no gene models is precisely the one that reaches miniprot, and there is no
bigBed to date. The miniprot case was already handled further down the chain, so
the fix is only to test it first.
Checked every combination that can occur: GRCz12ab and calJac240_pri, hub
assemblies with no models, report "direct"; hs1 dates its CAT models 2022-03-15;
mPanPan1_v2.0 dates its GenArk RefSeq 2025-06-02; hg38 still reads its real
ncbiRefSeqVersion.txt and rn6 still queries trackVersion.
refs #38300
- lines changed 66, context: html, text, full: html, text
da869954bf88c67e25a8b68cfd49ad50c5cec7d0 Mon Sep 14 05:58:57 2026 -0700
uniprot otto: optionally work on several taxa at once
The assemblies were processed strictly one after another, and that leaves the
cluster mostly idle. Each assembly submits its BLAST batch, waits for it to drain,
and then spends the best part of an hour in the single-threaded pslReps that
follows before the next assembly submits anything. Measured on the hs1 batch: 332
hours of CPU finished in 30 minutes of wall clock, a speedup of about 660, and then
58 minutes of one core concatenating and filtering 34335 PSL files. With 119
assemblies in the plan and a 1568 CPU cluster, that ordering costs about a day.
--taxonThreads=N runs N taxa at a time. It defaults to 1, so nothing changes unless
it is asked for; 4 to 6 is a reasonable range.
Taxa, not assemblies. The assemblies of one taxon share fasta/<taxId>.fa, which is
rebuilt at the start of each taxon, so two threads on one taxon would race on it.
Below that everything is per-assembly - protToGenome/<db>, bigBed/<db>, the cluster
batch directory - so separate taxa do not share files. os.makedirs calls are now
exist_ok, since two taxa starting together can both find a directory missing.
A failing worker is escalated, not swallowed. run() reports a failed command with
sys.exit(), which raises SystemExit; in a worker thread that would kill only that
thread and leave the run looking successful, which is the failure mode this
pipeline has a long history of. runTaxa catches BaseException per taxon, names each
one that failed, and aborts the run at the end.
Checked: a worker calling sys.exit aborts the run, a worker raising an ordinary
exception aborts the run, both after the other taxa have still been attempted, an
all-good set returns normally, --taxonThreads=1 keeps the old sequential order, and
--dbs still selects which taxa run.
refs #38300
- lines changed 31, context: html, text, full: html, text
99a26061f6cf8d3ee709f3468dda88af74e4049d Mon Sep 14 06:17:21 2026 -0700
uniprot otto: convert miniprot alignments to PSL properly
GRCz12ab cleared miniprot and then died in the annotation lift:
Error: inPsl Q98TT6 tSize (336) != mapPsl Q98TT6 qSize (339)
The lift maps annotations that are given in protein coordinates, so the mapping PSL
has to have the protein as its query, with qSize three times the protein length.
Routing miniprot's output through gff3ToGenePred and genePredToFakePsl does not give
that: it makes the query the transcript implied by the alignment, so qSize comes out
as the aligned CDS length. Measured over 93518 zebrafish alignments, that was wrong
for 95% of them - 79% out by exactly one codon, the trailing stop, and 16% out by
other amounts where miniprot aligned only part of the protein.
pafToPsl cannot do it either: it rejects miniprot's CIGAR, which is splice aware and
uses operators it does not know.
But every CDS line of a miniprot GFF carries its own "Target=<acc> <start> <end>"
giving the protein range that block covers, so the alignment can be reconstructed
exactly. New miniprotToPsl does that.
Two things it has to get right. On the minus strand miniprot lists blocks in protein
order, which is descending genomic order, while a PSL lists them ascending and puts
the block starts on the reverse complemented query, with qStart and qEnd still in
forward coordinates. And a block whose genomic span is shorter than its protein range
implies, where miniprot placed a frameshift, is clamped to the genome, with the
overall ranges then derived from the blocks rather than from the protein ranges.
Verified against real bonobo alignments: pslCheck reports 65 checked, 0 failed; qSize
is three times the protein length for every row; qName, strand and tStart match
miniprot exactly for all 65; and tEnd matches for 43, is short by exactly 3 on 12,
and those 12 are the ones carrying a stop_codon feature, which is correct since the
protein has no stop codon. 26 of the rows are on the minus strand.
refs #38300
- lines changed 1, context: html, text, full: html, text
a77bab60dc03de789c241d97692ba1f672dfddab Mon Sep 14 07:30:12 2026 -0700
uniprot otto: use the miniprot in /cluster/bin/x86_64
max built the current miniprot into /cluster/bin/x86_64/miniprot, which is the
right home for it: shared, on the PATH the cluster nodes already use, and
maintained with the rest of the kent binaries rather than by this pipeline.
Dropped the copy I had built under the otto directory, so there is only one.
Both were 0.18-r281; checked that otto can run the system one.
refs #38300
- src/hg/utils/otto/uniprot/doUpdate.sh
- lines changed 64, context: html, text, full: html, text
af613a331e6839c6513c3e366abcb67af0fe8386 Wed Sep 9 06:47:14 2026 -0700
UniProt otto: get the monthly update running again and make a stalled run visible
The monthly UniProt job had produced nothing since January 2025. The tracks
served release 2024_06 while the download sitting on disk was at 2026_02, on
every assembly the job builds.
Cause: uniprotToTab appended a personal conda site-packages directory to
sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A
venv's python is only a symlink to the system one, so when hgwdev moved to
python 3.9 the compiled lxml in there stopped loading and every run died at the
parse step. Removed both. The system python3 has lxml from python3-lxml and the
two are upgraded together, so there is nothing left here to go stale. Verified
by parsing real 2026_02 records under python 3.9 with lxml 5.4.
Why nobody noticed for nineteen months:
- doUpdate.sh read $? after an intervening echo, so it captured the echo's exit
code and mailed "Big Uniprot update OK" every month while the job was dying.
It now reads the real exit code, says FAILED, prints the tail of the log and
exits nonzero. A month with no new UniProt release stays silent, which is the
normal otto behaviour, so silence again means "nothing to do".
- The logs were overwritten on every run, so a failure left no trace on disk.
doUpdate.sh now appends one line per run to runLog.txt, which is never
truncated, and keeps a failing log as lastFail.log.
- version.txt in each bigBed directory was rewritten on every run even when the
release string was identical. That is the file the trackDb dataVersion setting
shows, and its date is what people check to decide whether a pipeline is still
alive, so a stalled track could look freshly updated. It is now written only
when the release actually changes.
Also, so this cannot come back:
- doUniprot checks that uniprotToTab can start before the download, instead of
finding out 35 minutes later.
- pylint on hgwdev is itself pinned to pythons that no longer exist, so
"make install" aborted on its first line and could not be used. Replaced with
a syntax check that needs nothing but python3; pylint stays best-effort.
- uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from
/hive/data/outside/otto/uniprot without being in the makefile's copy list.
The tree copy of uniprotToTab was still python 2 from 2021. All are now
listed and in sync, and "make diff" reports drift.
- Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh
and the pslMap -inType/-mapType flags.
refs #38300
- lines changed 20, context: html, text, full: html, text
bee41315e53da6fb5671f96302233c655906dfd5 Wed Sep 9 06:56:49 2026 -0700
UniProt otto: rebuild the venv rather than relying on a system lxml
Correction to the previous commit: hgwdev has no system-wide lxml at all. The
import I tested was resolving to my own ~/.local/lib/python3.9/site-packages,
which cron never sees, because it runs the pipeline as otto.
So the environment is a virtualenv again, but a reproducible one. makeVenv.sh
deletes venv/ and rebuilds it from /usr/bin/python3, installs lxml, opens up the
permissions for otto, and then checks that lxml imports with an empty environment
so we know the venv stands on its own instead of borrowing from whoever ran it.
Built with --copies, so venv/bin/python is a real copy rather than a symlink that
would silently follow a system python upgrade while its compiled modules stayed
behind.
doUpdate.sh activates venv/ again and says to run makeVenv.sh if it is missing or
if the parser will not start. Verified: /usr/bin/python3 without per-user packages
cannot import lxml, the venv can, and after activation the parser runs and
converts real 2026_02 records.
Also shortened the README to how the pipeline is started and how it works, and
trimmed the history out of the code comments, leaving the ticket as the pointer.
refs #38300
- lines changed 10, context: html, text, full: html, text
28b1395f3e814f53fb8b0d75904f09469504563e Wed Sep 9 07:13:08 2026 -0700
UniProt otto: report a held lock file as its own case, not as a failure
A run that is still going, or one that crashed and left
/hive/data/outside/uniProt/current/doUniprot.lock behind, made the next cron run
print the full failure report. Now it says so in one line and logs LOCKED, so a
long run in progress does not look like a broken pipeline, while a stale lock is
still mentioned to whoever gets the mail.
refs #38300
- lines changed 6, context: html, text, full: html, text
989ad01358d595f619021742182be67c01dd56fd Wed Sep 9 07:38:59 2026 -0700
UniProt otto: log an interrupted run instead of leaving a dangling START
Killing a run left "START" in runLog.txt with no line after it, which reads
exactly like a run that is still going. doUpdate.sh now traps INT/TERM/HUP, logs
INTERRUPTED, and removes doUniprot.lock, which doUniprot's own atexit handler
does not get to run on a signal and which would otherwise block the next run.
Note the shell only runs the trap once the foreground doUniprot has exited, so
this fires when the whole process group is killed, which is what pkill does and
what actually happens in practice.
refs #38300
- src/hg/utils/otto/uniprot/makeUniProtPsl.sh
- lines changed 5, context: html, text, full: html, text
af613a331e6839c6513c3e366abcb67af0fe8386 Wed Sep 9 06:47:14 2026 -0700
UniProt otto: get the monthly update running again and make a stalled run visible
The monthly UniProt job had produced nothing since January 2025. The tracks
served release 2024_06 while the download sitting on disk was at 2026_02, on
every assembly the job builds.
Cause: uniprotToTab appended a personal conda site-packages directory to
sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A
venv's python is only a symlink to the system one, so when hgwdev moved to
python 3.9 the compiled lxml in there stopped loading and every run died at the
parse step. Removed both. The system python3 has lxml from python3-lxml and the
two are upgraded together, so there is nothing left here to go stale. Verified
by parsing real 2026_02 records under python 3.9 with lxml 5.4.
Why nobody noticed for nineteen months:
- doUpdate.sh read $? after an intervening echo, so it captured the echo's exit
code and mailed "Big Uniprot update OK" every month while the job was dying.
It now reads the real exit code, says FAILED, prints the tail of the log and
exits nonzero. A month with no new UniProt release stays silent, which is the
normal otto behaviour, so silence again means "nothing to do".
- The logs were overwritten on every run, so a failure left no trace on disk.
doUpdate.sh now appends one line per run to runLog.txt, which is never
truncated, and keeps a failing log as lastFail.log.
- version.txt in each bigBed directory was rewritten on every run even when the
release string was identical. That is the file the trackDb dataVersion setting
shows, and its date is what people check to decide whether a pipeline is still
alive, so a stalled track could look freshly updated. It is now written only
when the release actually changes.
Also, so this cannot come back:
- doUniprot checks that uniprotToTab can start before the download, instead of
finding out 35 minutes later.
- pylint on hgwdev is itself pinned to pythons that no longer exist, so
"make install" aborted on its first line and could not be used. Replaced with
a syntax check that needs nothing but python3; pylint stays best-effort.
- uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from
/hive/data/outside/otto/uniprot without being in the makefile's copy list.
The tree copy of uniprotToTab was still python 2 from 2021. All are now
listed and in sync, and "make diff" reports drift.
- Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh
and the pslMap -inType/-mapType flags.
refs #38300
- lines changed 6, context: html, text, full: html, text
af670dcecc6911452f3e1a5c38cfea1ef0067978 Wed Sep 9 07:31:07 2026 -0700
UniProt otto: submit the parasol batch on hgwdev, drop the ssh to ku
The protein-to-transcript BLAST batch was submitted with
ssh ku "cd <workdir> && para make jobList"
ku has been decommissioned for about two years. It still resolves in DNS, but
"ssh ku" is an immediate "No route to host", so a run would have died at the
mapping stage, which comes only after days of XML parsing.
hgwdev is the parasol head node now, so "para make" runs here with no ssh hop.
Verified with a one-job batch using the same bare-command jobList this script
writes: it lands on a compute node with its cwd set to the batch directory, the
same as the old ku jobs did. Also checked from a compute node that tclsh and the
blast-2.2.16 blastall/formatdb that mapUniprot_doBlast needs are still there.
The cluster name is gone rather than redirected. doUniprot no longer reads
/cluster/bin/scripts/cluster.txt, a file written in 2017 that still says "ku" and
that nothing else in the tree read, and makeUniProtPsl.sh no longer takes a head
node argument, so its positional parameters shift down by one.
refs #38300
- src/hg/utils/otto/uniprot/makeVenv.sh
- lines changed 47, context: html, text, full: html, text
bee41315e53da6fb5671f96302233c655906dfd5 Wed Sep 9 06:56:49 2026 -0700
UniProt otto: rebuild the venv rather than relying on a system lxml
Correction to the previous commit: hgwdev has no system-wide lxml at all. The
import I tested was resolving to my own ~/.local/lib/python3.9/site-packages,
which cron never sees, because it runs the pipeline as otto.
So the environment is a virtualenv again, but a reproducible one. makeVenv.sh
deletes venv/ and rebuilds it from /usr/bin/python3, installs lxml, opens up the
permissions for otto, and then checks that lxml imports with an empty environment
so we know the venv stands on its own instead of borrowing from whoever ran it.
Built with --copies, so venv/bin/python is a real copy rather than a symlink that
would silently follow a system python upgrade while its compiled modules stayed
behind.
doUpdate.sh activates venv/ again and says to run makeVenv.sh if it is missing or
if the parser will not start. Verified: /usr/bin/python3 without per-user packages
cannot import lxml, the venv can, and after activation the parser runs and
converts real 2026_02 records.
Also shortened the README to how the pipeline is started and how it works, and
trimmed the history out of the code comments, leaving the ticket as the pointer.
refs #38300
- src/hg/utils/otto/uniprot/makefile
- lines changed 27, context: html, text, full: html, text
af613a331e6839c6513c3e366abcb67af0fe8386 Wed Sep 9 06:47:14 2026 -0700
UniProt otto: get the monthly update running again and make a stalled run visible
The monthly UniProt job had produced nothing since January 2025. The tracks
served release 2024_06 while the download sitting on disk was at 2026_02, on
every assembly the job builds.
Cause: uniprotToTab appended a personal conda site-packages directory to
sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A
venv's python is only a symlink to the system one, so when hgwdev moved to
python 3.9 the compiled lxml in there stopped loading and every run died at the
parse step. Removed both. The system python3 has lxml from python3-lxml and the
two are upgraded together, so there is nothing left here to go stale. Verified
by parsing real 2026_02 records under python 3.9 with lxml 5.4.
Why nobody noticed for nineteen months:
- doUpdate.sh read $? after an intervening echo, so it captured the echo's exit
code and mailed "Big Uniprot update OK" every month while the job was dying.
It now reads the real exit code, says FAILED, prints the tail of the log and
exits nonzero. A month with no new UniProt release stays silent, which is the
normal otto behaviour, so silence again means "nothing to do".
- The logs were overwritten on every run, so a failure left no trace on disk.
doUpdate.sh now appends one line per run to runLog.txt, which is never
truncated, and keeps a failing log as lastFail.log.
- version.txt in each bigBed directory was rewritten on every run even when the
release string was identical. That is the file the trackDb dataVersion setting
shows, and its date is what people check to decide whether a pipeline is still
alive, so a stalled track could look freshly updated. It is now written only
when the release actually changes.
Also, so this cannot come back:
- doUniprot checks that uniprotToTab can start before the download, instead of
finding out 35 minutes later.
- pylint on hgwdev is itself pinned to pythons that no longer exist, so
"make install" aborted on its first line and could not be used. Replaced with
a syntax check that needs nothing but python3; pylint stays best-effort.
- uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from
/hive/data/outside/otto/uniprot without being in the makefile's copy list.
The tree copy of uniprotToTab was still python 2 from 2021. All are now
listed and in sync, and "make diff" reports drift.
- Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh
and the pslMap -inType/-mapType flags.
refs #38300
- lines changed 8, context: html, text, full: html, text
bee41315e53da6fb5671f96302233c655906dfd5 Wed Sep 9 06:56:49 2026 -0700
UniProt otto: rebuild the venv rather than relying on a system lxml
Correction to the previous commit: hgwdev has no system-wide lxml at all. The
import I tested was resolving to my own ~/.local/lib/python3.9/site-packages,
which cron never sees, because it runs the pipeline as otto.
So the environment is a virtualenv again, but a reproducible one. makeVenv.sh
deletes venv/ and rebuilds it from /usr/bin/python3, installs lxml, opens up the
permissions for otto, and then checks that lxml imports with an empty environment
so we know the venv stands on its own instead of borrowing from whoever ran it.
Built with --copies, so venv/bin/python is a real copy rather than a symlink that
would silently follow a system python upgrade while its compiled modules stayed
behind.
doUpdate.sh activates venv/ again and says to run makeVenv.sh if it is missing or
if the parser will not start. Verified: /usr/bin/python3 without per-user packages
cannot import lxml, the venv can, and after activation the parser runs and
converts real 2026_02 records.
Also shortened the README to how the pipeline is started and how it works, and
trimmed the history out of the code comments, leaving the ticket as the pointer.
refs #38300
- lines changed 1, context: html, text, full: html, text
99a26061f6cf8d3ee709f3468dda88af74e4049d Mon Sep 14 06:17:21 2026 -0700
uniprot otto: convert miniprot alignments to PSL properly
GRCz12ab cleared miniprot and then died in the annotation lift:
Error: inPsl Q98TT6 tSize (336) != mapPsl Q98TT6 qSize (339)
The lift maps annotations that are given in protein coordinates, so the mapping PSL
has to have the protein as its query, with qSize three times the protein length.
Routing miniprot's output through gff3ToGenePred and genePredToFakePsl does not give
that: it makes the query the transcript implied by the alignment, so qSize comes out
as the aligned CDS length. Measured over 93518 zebrafish alignments, that was wrong
for 95% of them - 79% out by exactly one codon, the trailing stop, and 16% out by
other amounts where miniprot aligned only part of the protein.
pafToPsl cannot do it either: it rejects miniprot's CIGAR, which is splice aware and
uses operators it does not know.
But every CDS line of a miniprot GFF carries its own "Target=<acc> <start> <end>"
giving the protein range that block covers, so the alignment can be reconstructed
exactly. New miniprotToPsl does that.
Two things it has to get right. On the minus strand miniprot lists blocks in protein
order, which is descending genomic order, while a PSL lists them ascending and puts
the block starts on the reverse complemented query, with qStart and qEnd still in
forward coordinates. And a block whose genomic span is shorter than its protein range
implies, where miniprot placed a frameshift, is clamped to the genome, with the
overall ranges then derived from the blocks rather than from the protein ranges.
Verified against real bonobo alignments: pslCheck reports 65 checked, 0 failed; qSize
is three times the protein length for every row; qName, strand and tStart match
miniprot exactly for all 65; and tEnd matches for 43, is short by exactly 3 on 12,
and those 12 are the ones carrying a stop_codon feature, which is correct since the
protein has no stop codon. 26 of the rows are on the minus strand.
refs #38300
- src/hg/utils/otto/uniprot/miniprotToPsl
- lines changed 180, context: html, text, full: html, text
99a26061f6cf8d3ee709f3468dda88af74e4049d Mon Sep 14 06:17:21 2026 -0700
uniprot otto: convert miniprot alignments to PSL properly
GRCz12ab cleared miniprot and then died in the annotation lift:
Error: inPsl Q98TT6 tSize (336) != mapPsl Q98TT6 qSize (339)
The lift maps annotations that are given in protein coordinates, so the mapping PSL
has to have the protein as its query, with qSize three times the protein length.
Routing miniprot's output through gff3ToGenePred and genePredToFakePsl does not give
that: it makes the query the transcript implied by the alignment, so qSize comes out
as the aligned CDS length. Measured over 93518 zebrafish alignments, that was wrong
for 95% of them - 79% out by exactly one codon, the trailing stop, and 16% out by
other amounts where miniprot aligned only part of the protein.
pafToPsl cannot do it either: it rejects miniprot's CIGAR, which is splice aware and
uses operators it does not know.
But every CDS line of a miniprot GFF carries its own "Target=<acc> <start> <end>"
giving the protein range that block covers, so the alignment can be reconstructed
exactly. New miniprotToPsl does that.
Two things it has to get right. On the minus strand miniprot lists blocks in protein
order, which is descending genomic order, while a PSL lists them ascending and puts
the block starts on the reverse complemented query, with qStart and qEnd still in
forward coordinates. And a block whose genomic span is shorter than its protein range
implies, where miniprot placed a frameshift, is clamped to the genome, with the
overall ranges then derived from the blocks rather than from the protein ranges.
Verified against real bonobo alignments: pslCheck reports 65 checked, 0 failed; qSize
is three times the protein length for every row; qName, strand and tStart match
miniprot exactly for all 65; and tEnd matches for 43, is short by exactly 3 on 12,
and those 12 are the ones carrying a stop_codon feature, which is correct since the
protein has no stop codon. 26 of the rows are on the minus strand.
refs #38300
- src/hg/utils/otto/uniprot/notifyRun.sh
- lines changed 88, context: html, text, full: html, text
bf15116918c332d150006ee7b26eff5cc90cc5b9 Wed Sep 9 08:19:45 2026 -0700
UniProt otto: a notifier for hand-started catch-up runs
A run started by hand can take days, and until it ends there is nothing to tell
you where it is short of going and looking. notifyRun.sh watches lastRun.log,
works out which of the nine stages the pipeline has reached, and mails a note on
every transition, with a "still doing X" note every twelve hours so a long TrEMBL
parse does not go quiet for three days. When the process disappears it mails the
outcome, read out of runLog.txt, so a killed or failed run is reported and not
just an absence of mail.
The cron run does not need it, doUpdate.sh already mails through otto's MAILTO.
This is for watching a catch-up run in between.
setsid nohup ./notifyRun.sh you@ucsc.edu > notifyRun.log 2>&1 < /dev/null &
refs #38300
- lines changed 16, context: html, text, full: html, text
fcaa1479c3dbb46e3b3b25865432158156cf2dff Wed Sep 9 08:22:31 2026 -0700
UniProt otto: let notifyRun.sh post to a Slack incoming webhook too
Mail is the reliable channel for a run that takes days, but it does not
necessarily land on a phone. With a Slack incoming webhook in ~/.hg.conf as
slack.webhook=..., each stage note is also posted to that channel. Without the
key, nothing changes and notifications go by mail only.
The URL stays out of the command line, where ps would show it to everyone on the
machine, and out of stdout and stderr: curl runs with -s and a failed post logs
only that it failed. Checked that an absent webhook is a silent no-op, that an
unreachable one warns without killing the watch, and that the URL appears nowhere
in the output.
refs #38300
- lines changed 5, context: html, text, full: html, text
51b9776d380b32ed640c81917ec2d9a3528d4b06 Wed Sep 9 08:58:55 2026 -0700
UniProt otto: notifyRun.sh could not see that the TrEMBL parse had started
doUniprot's run() logs its "Running: <cmd>" line after os.system returns, not
before, so the log says a command has finished, never that it is running. The
stage ladder keyed the TrEMBL stage off that line, which meant it only noticed
TrEMBL once TrEMBL was over: a three-day parse would have been reported as
"parsing the SwissProt XML" throughout, including in every twelve-hour heartbeat.
Ask the process table for a running uniprotToTab --trembl instead, and keep the
log check as the fallback for after the process is gone. Later stages still
override it, so the ladder is unchanged otherwise. Caught on the live run, which
had the flag in its command line while the notifier still said SwissProt.
refs #38300
- lines changed 9, context: html, text, full: html, text
66db3fb64ca38726355efb68c340c93fd83282bb Mon Sep 14 05:33:23 2026 -0700
uniprot otto: the run watcher could never see a run end
notifyRun.sh decided whether a run was still going with
pgrep -f "doUniprot run"
which matches far more than the pipeline: the shell that launched the watcher
carries that string in its own command line and stays alive for as long as the
watcher does, and so does every status command anyone types. The test was
therefore true forever. The watcher started on 9 September sat in its loop through
a failure on 11 September and three days of nothing, never reached the code that
reports the outcome, and never sent the mail it exists to send. Its log was empty
the whole time, which looked exactly like healthy silence.
Both process searches are now restricted to the user the pipeline runs as, otto by
default and overridable as the fourth argument. Measured against the live process
list: the old pattern matched 5 processes where there is 1 run, the new one matches
exactly 1.
refs #38300
- src/hg/utils/otto/uniprot/trackDb.template.txt
- lines changed 224, context: html, text, full: html, text
af613a331e6839c6513c3e366abcb67af0fe8386 Wed Sep 9 06:47:14 2026 -0700
UniProt otto: get the monthly update running again and make a stalled run visible
The monthly UniProt job had produced nothing since January 2025. The tracks
served release 2024_06 while the download sitting on disk was at 2026_02, on
every assembly the job builds.
Cause: uniprotToTab appended a personal conda site-packages directory to
sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A
venv's python is only a symlink to the system one, so when hgwdev moved to
python 3.9 the compiled lxml in there stopped loading and every run died at the
parse step. Removed both. The system python3 has lxml from python3-lxml and the
two are upgraded together, so there is nothing left here to go stale. Verified
by parsing real 2026_02 records under python 3.9 with lxml 5.4.
Why nobody noticed for nineteen months:
- doUpdate.sh read $? after an intervening echo, so it captured the echo's exit
code and mailed "Big Uniprot update OK" every month while the job was dying.
It now reads the real exit code, says FAILED, prints the tail of the log and
exits nonzero. A month with no new UniProt release stays silent, which is the
normal otto behaviour, so silence again means "nothing to do".
- The logs were overwritten on every run, so a failure left no trace on disk.
doUpdate.sh now appends one line per run to runLog.txt, which is never
truncated, and keeps a failing log as lastFail.log.
- version.txt in each bigBed directory was rewritten on every run even when the
release string was identical. That is the file the trackDb dataVersion setting
shows, and its date is what people check to decide whether a pipeline is still
alive, so a stalled track could look freshly updated. It is now written only
when the release actually changes.
Also, so this cannot come back:
- doUniprot checks that uniprotToTab can start before the download, instead of
finding out 35 minutes later.
- pylint on hgwdev is itself pinned to pythons that no longer exist, so
"make install" aborted on its first line and could not be used. Replaced with
a syntax check that needs nothing but python3; pylint stays best-effort.
- uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from
/hive/data/outside/otto/uniprot without being in the makefile's copy list.
The tree copy of uniprotToTab was still python 2 from 2021. All are now
listed and in sync, and "make diff" reports drift.
- Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh
and the pslMap -inType/-mapType flags.
refs #38300
- src/hg/utils/otto/varChat/varChatOtto.sh
- lines changed 12, context: html, text, full: html, text
042acb30f02d050444afd846b3f026eb33c99ab1 Wed Sep 9 08:38:02 2026 -0700
varChat otto: don't blank the track's version file when the fetch fails
/gbdb/hg38/bbi/varChatVersion.txt symlinks straight to
/hive/data/outside/otto/varChat/version.txt, so the browser reads that file
directly, and wget truncates its -O target before it has anything to write there.
A failed or partial fetch therefore left the track showing an empty version. By
that point the run has already done "mv varChat.hg38.latest.bb varChat.hg38.bb",
so the new data is live and only the version string is gone, and set -e aborting
afterwards does not undo it. VarChat only updates when upstream changes, and
upstream has been on v1.1 - 2025-11-07 for ten months, so a blank could sit there
that long.
Fetch to version.new.txt, require it to be non-empty, and only then move it into
place, keeping the old string with a warning otherwise. Same shape as
mitoMap/checkMitoMapUpdate.sh, which already does this.
Checked all three paths: the real upstream URL updates the file and leaves no temp
behind, an unreachable URL and a URL returning an empty body both keep the
previous string and warn. For contrast, the old one-liner against an unreachable
URL truncated version.txt to zero bytes.
Found while sweeping the otto updaters for the version-stamping problem behind
the uniprot outage. refs #38300
- src/hg/utils/overlapSelect/makefile
- lines changed 3, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/utils/refSeqGet/makefile
- lines changed 2, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/utils/refreshNamedSessionCustomTracks/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/utils/refreshNamedSessionCustomTracks/refreshNamedSessionCustomTracks.c
- lines changed 38, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/sessionCartAudit/sessionCartAudit.py
- lines changed 113, context: html, text, full: html, text
b066328906fe2c8fbbfada16f18e8cc2fb9dc1cf Sat Sep 12 09:58:46 2026 -0700
sessionCartAudit: match a catalog row that names a whole cart variable, refs #37979
peel() only ever considered suffixes that begin after a separator, so a catalog
row anchored on a fixed prefix could never match the variable it describes: by
the time the walk reached a separator, the hgta_ that anchors
hgta_fs.check.<db>.<table>.<field> was gone. A plain literal row was hit just
as hard, since dbRIP.genoRegion could only be tested as its own tail,
"genoRegion". 4,299 hgta_ names were reported as covered by nothing but a
catch-all because of it.
The #37838 catalog already says which kind of row it is and the audit was
throwing that away. A row whose separator is "." or "_" names a suffix that
follows a track name; anything else names the whole cart variable, with the
separator in front - "" for hgTables and the old per-dataset variables,
cgs_<track>_ for chromGraph. trackVarNames() now returns the separator with
the name, the whole-variable rows are compiled apart, and peel() tries the
whole name before walking suffixes, so the longer and correct match wins.
Two shorthands the catalog already uses are read rather than expanded by hand:
a comma list is several variables sharing one description, and a trailing * is
a family. A row that is prose rather than one pattern can be matched by
nothing, so --check names it instead of letting it count for nothing; there is
one today. The bare wildcard lists are sorted, because they were built by
walking a set and two runs of a published report diffed for no reason.
Over 6,631 saved sessions this moves 2,016 names: 1,990 out of the catch-all
bucket, 25 out of the bare-track-name bucket (filter text boxes whose stored
value is empty, which the visibility heuristic had been reading as track
names), and tfbsConsSitesCutoff out of unknown. Nothing leaves the catalogued
buckets. 1,208 hgta_fs.check names now match the row that describes them
rather than <wigTrack>.<wigVar> or <filterName>Type.
- src/hg/utils/tdbQuery/makefile
- lines changed 1, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/utils/urlCommandCatalog/urlCommandCatalog.py
- lines changed 10, context: html, text, full: html, text
1270c4fd6763378548918470aa2a4b799c7c930d Sat Sep 12 06:59:17 2026 -0700
urlCommandCatalog: catalog hgc's aliTrack parameter, refs #37923
aliTrackParam() puts &aliTrack=<track> on the alignment links and hgc reads it
back with cartUsualString to find the track it was called on. aliTable cannot
serve that purpose: it is the table name from the assembly the alignments came
from, and that name usually exists on the assembly being viewed as well, so it
cannot tell a quickLifted alignment track from a native one. Added to hgc
under #38249; it is in hgc's excludeVars, so it does not reach a saved session.
- src/hg/visiGene/hgVisiGene/hgVisiGene.c
- lines changed 4, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/hg/visiGene/hgVisiGene/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/wikiPlot/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/inc/cheapcgi.h
- lines changed 6, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- lines changed 14, context: html, text, full: html, text
cf9f4cb7f55c7beb8ad5f11118656a60770a71a5 Thu Sep 10 01:02:36 2026 -0700
Move the extra-HTTP-header list into cheapcgi, and write the header only once
Follow-on to the cgiPrintContentType() refactor.
cart.c owned the mechanism for adding headers ahead of the content type: a
global slPair list plus addHttpHeaders() to print it. That put it in hg/lib,
out of reach of the CGIs and library code that do not use a cart, even though
nothing about it is cart-specific. It now lives next to cgiPrintContentType()
in lib/cheapcgi.c, behind cgiAddHttpHeader(name, value) instead of a bare
global, and cgiPrintContentType() writes the queued headers itself. The one
caller, hgTracks/mainMain.c, reads the same but no longer reaches into cart.h
for it. cspWriteResponseHeader() stays in hg/lib where it belongs, since it
needs hg.conf; cartWriteHeaderAndCont() calls it directly now, the way the
other ten callers already do.
cgiPrintContentType() also writes at most once per process now. A second
content type cannot reach the browser as a header - it lands in the page body
as text - so the later caller is always the mistaken one. cart.c had a private
cartDidContentType flag for exactly this, covering only the flows that went
through the cart; the guard is now in the one function every flow shares, and
cartDidContentType is gone. Its public equivalent, cgiDidContentType(), is
what cartWriteHeaderAndCont() checks so it does not write a second cookie.
Verified: make libs, make cgi and the lib test suite are clean, hgTracks still
emits Cache-Control: no-store, and hgTracks, hgc and hgTables each emit exactly
one Content-Type on both their html and their text paths.
- lines changed 5, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/inc/common.mk
- lines changed 31, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/inc/userApp.mk
- lines changed 5, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/jkOwnLib/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/lib/cheapcgi.c
- lines changed 11, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- lines changed 33, context: html, text, full: html, text
cf9f4cb7f55c7beb8ad5f11118656a60770a71a5 Thu Sep 10 01:02:36 2026 -0700
Move the extra-HTTP-header list into cheapcgi, and write the header only once
Follow-on to the cgiPrintContentType() refactor.
cart.c owned the mechanism for adding headers ahead of the content type: a
global slPair list plus addHttpHeaders() to print it. That put it in hg/lib,
out of reach of the CGIs and library code that do not use a cart, even though
nothing about it is cart-specific. It now lives next to cgiPrintContentType()
in lib/cheapcgi.c, behind cgiAddHttpHeader(name, value) instead of a bare
global, and cgiPrintContentType() writes the queued headers itself. The one
caller, hgTracks/mainMain.c, reads the same but no longer reaches into cart.h
for it. cspWriteResponseHeader() stays in hg/lib where it belongs, since it
needs hg.conf; cartWriteHeaderAndCont() calls it directly now, the way the
other ten callers already do.
cgiPrintContentType() also writes at most once per process now. A second
content type cannot reach the browser as a header - it lands in the page body
as text - so the later caller is always the mistaken one. cart.c had a private
cartDidContentType flag for exactly this, covering only the flows that went
through the cart; the guard is now in the one function every flow shares, and
cartDidContentType is gone. Its public equivalent, cgiDidContentType(), is
what cartWriteHeaderAndCont() checks so it does not write a second cookie.
Verified: make libs, make cgi and the lib test suite are clean, hgTracks still
emits Cache-Control: no-store, and hgTracks, hgc and hgTables each emit exactly
one Content-Type on both their html and their text paths.
- lines changed 53, context: html, text, full: html, text
c2a6ef817930149ad6f55818fa0196d99c47e02b Fri Sep 11 10:40:11 2026 -0700
cheapcgi: skip a CGI pair with no =value instead of aborting, refs #38335
Both query string parsers looked for the '=' across the whole rest of the
string rather than inside the current pair. A pair with no '=' in it
therefore ran into the pair after it and took its value. "g-catV2&db=hg38"
was stored as one variable named "g-catV2&db", so db was lost with no
warning, and that corrupt name was copied on into the cart. The same pair
at the end of the string had no '=' left to find and aborted the whole
request, which is what the "Mangled CGI input string g-catV2" entries in the
hgw1 logs were.
Both parsers now find the end of the pair first, keeping the existing
separator precedence ('&', then ';' for DAS), and skip a pair with no '='.
A mixed "a=1;b=2&c=3" still parses the way it did.
Adds lib/tests/cgiParseTest, which runs 18 query strings through both
parsers. It covers the empty pair of #38185 as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 36, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/errAbort.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- lines changed 8, context: html, text, full: html, text
58104a98358604975dac0a9350ca91d2d25c501d Thu Sep 10 05:02:45 2026 -0700
hUserAbort shows its message to the user instead of turning into a 500
hUserAbort() reports an error caused by user input, so the message is written
to be read by the user. It was only reaching them when a CGI had already
pushed a warn handler of its own. The apiKey and bot checks call it from
main() before that happens, and the default handler then writes to stderr and
nothing else, unless hg.conf sets showEarlyErrors - off by default, and off on
the RR. Apache turns the empty response into a 500, which is what
hubApi/hubApi.c works around by pre-validating the apiKey itself.
hVaUserAbort() now turns doContentType on for the rest of the process when it
is running as a CGI, so the default handler emits the Content-Type line and
the message. It stays off for a program that never called cgiSpoof(), and it
is inert inside an errCatch, which pushes its own warn handler - so a caller
that catches the abort to write its own response (hubApi's JSON) is unchanged.
Fixes the va_list handling in defaultVaWarn() while in there. It read args
three times but only the second and third read from a va_copy: the first
vfprintf consumed args itself, so the two reads after it saw a spent va_list
and the copy sent to the browser lost every %s and %d. It printed
"Bad thing: [br]" where the message was "Bad thing: %s<br>". Every read now
takes its own copy, and the buffer is filled with vsnprintf rather than
vsprintf.
No XSS: on this path defaultVaWarn replaces < and > with [ and ] across the
whole formatted message, args included. The other handlers that can report an
hUserAbort - earlyWarningHandler and cartEarlyWarningHandler via
htmlVaEncodeErrorText, htmlVaWarn, webVaWarn - all run the arguments through
vaHtmlDyStringPrintf, which html-encodes & < > / " and '. No caller passes
user data as the format string.
- src/lib/fa.c
- lines changed 1, context: html, text, full: html, text
055cc335c485aa94f48b440b0105eae2e1ecacb4 Thu Sep 10 09:41:34 2026 -0700
lib/fa.c: grow the FASTA read buffer to hold the line being added, refs #38320
faMixedSpeedReadNext decided to grow its buffer when bufIx + lineSize no
longer fit, but then asked expandFaFastBuf to reach only lineSize.
expandFaFastBuf stops doubling as soon as it meets that size, so it could
hand back a buffer smaller than bufIx + lineSize. The copy loop wrote the
whole line regardless.
Ask for bufIx + lineSize + 1, which covers the line and the terminating NUL
written after the loop. The other two calls each add one byte to a buffer
that is exactly full, so doubling always covers them and they are unchanged.
Doubling usually leaves enough room, which is why this took so long to show
up. It needs a record whose lines vary a lot in length. faToTwoBit aborted
on such a file with a corrupted heap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/htmshell.c
- lines changed 6, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/lib/makefile
- lines changed 6, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/lib/osunix.c
- lines changed 93, context: html, text, full: html, text
924c63fc1be1909f8ab2de587a968dfe1955f499 Tue Sep 8 07:19:24 2026 -0700
lib: replace eatExcessDotDotInPath with eatExcessDotsInPath, and resolve a root ".."
simplifyPathToDir now canonicalizes with eatExcessDotsInPath instead of the
older eatExcessDotDotInPath, which is removed. simplifyPathToDir was its only
caller. The old routine scanned for the literal string "/../" and so gave
several answers that do not match realpath(3):
../../a became a (the second .. ate the first)
x/./../y became x/y (.. ate the ".", not the "x")
../.. became "" (an empty path, not the parent)
a/./b stayed a/./b (single dots were never removed)
h/../../../etc/passwd became etc/passwd (the escaping .. were eaten)
The last one is the reason to bother. A caller that wants to know whether a
path stays inside a directory cannot tell from the old result, because a path
that climbs out comes back looking like it stayed in.
Two changes to eatExcessDotsInPath came out of this.
It now drops a ".." at the root of an absolute path, so /../a gives /a and
/.. gives /, as realpath(3) does. Because an absolute path can then never
hold a "..", the guard is exactly "nothing consumed and not absolute".
It also returns "." rather than "" for a non-empty relative path that reduces
to nothing. Callers join the result with "%s/%s", where an empty string names
the file system root instead of the current directory. Without this,
"tdbQuery -root=." looked for /tagTypes.tab and died. The in-place write is
safe because a non-empty input always leaves room for one byte.
The DEBUG selftest is rewritten. Two of its assertions asserted the old wrong
answers for /.. and /../a, a third followed from them, and three asserted the
empty-string result now replaced by ".". Added cases for each item above.
Verified: all 38 selftest cases pass against the built library, and 48089
exhaustively enumerated paths over the alphabets "/.a" and "/.ab" match an
independent model with no ASan or UBSan report. tdbQuery and raSqlQuery give
byte-identical output to the master build across every root form tried,
except two that the master build got wrong.
The other caller of eatExcessDotsInPath is resolveDotDots, which hgTrackUi
uses to canonicalize a fileUrl before checking it against a hub's base
directory. Neither change loosens that check: a path that used to
canonicalize to /../secret now gives /secret, and neither is under a hub base
directory.
refs #37263
- src/lib/tests/cgiCookieTest.c
- lines changed 72, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/tests/cgiParseTest.c
- lines changed 95, context: html, text, full: html, text
c2a6ef817930149ad6f55818fa0196d99c47e02b Fri Sep 11 10:40:11 2026 -0700
cheapcgi: skip a CGI pair with no =value instead of aborting, refs #38335
Both query string parsers looked for the '=' across the whole rest of the
string rather than inside the current pair. A pair with no '=' in it
therefore ran into the pair after it and took its value. "g-catV2&db=hg38"
was stored as one variable named "g-catV2&db", so db was lost with no
warning, and that corrupt name was copied on into the cart. The same pair
at the end of the string had no '=' left to find and aborted the whole
request, which is what the "Mangled CGI input string g-catV2" entries in the
hgw1 logs were.
Both parsers now find the end of the pair first, keeping the existing
separator precedence ('&', then ';' for DAS), and skip a pair with no '='.
A mixed "a=1;b=2&c=3" still parses the way it did.
Adds lib/tests/cgiParseTest, which runs 18 query strings through both
parsers. It covers the empty pair of #38185 as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/tests/expected/cgiCookieTest
- lines changed 44, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/tests/expected/cgiParseTest
- lines changed 72, context: html, text, full: html, text
c2a6ef817930149ad6f55818fa0196d99c47e02b Fri Sep 11 10:40:11 2026 -0700
cheapcgi: skip a CGI pair with no =value instead of aborting, refs #38335
Both query string parsers looked for the '=' across the whole rest of the
string rather than inside the current pair. A pair with no '=' in it
therefore ran into the pair after it and took its value. "g-catV2&db=hg38"
was stored as one variable named "g-catV2&db", so db was lost with no
warning, and that corrupt name was copied on into the cart. The same pair
at the end of the string had no '=' left to find and aborted the whole
request, which is what the "Mangled CGI input string g-catV2" entries in the
hgw1 logs were.
Both parsers now find the end of the pair first, keeping the existing
separator precedence ('&', then ';' for DAS), and skip a pair with no '='.
A mixed "a=1;b=2&c=3" still parses the way it did.
Adds lib/tests/cgiParseTest, which runs 18 query strings through both
parsers. It covers the empty pair of #38185 as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/tests/expected/faSpeedReadTest
- lines changed 24, context: html, text, full: html, text
28538f8d9721d6b6503d16b4c6e7db1350cb7c53 Thu Sep 10 09:41:44 2026 -0700
lib/tests: add faSpeedReadTest, covering the FASTA read buffer growth, refs #38320
Eleven cases, each a set of line lengths. Each one writes a FASTA file,
reads it back with faMixedSpeedReadNext, and checks the name, the size,
every base and the terminating NUL.
The test frees the buffer before each case, so the growth starts from the
same place every time. That is what makes a shape reproduce a given sequence
of buffer sizes, and so what makes these shapes mean anything. Freeing also
means a write past the end of the buffer has to survive a free before the
next case can print.
The shapes are the one from the ticket, a variant whose overrun was small
enough to go unnoticed, the edges of the initial 65536 buffer, the first
doubling, a short line before a long one, a record spanning several
doublings, an evenly wrapped control, a single short line, and a two record
file so the second record is checked for anything the first left behind.
Against the code before the previous commit the test reports a short read
and differing bases on the first case, and exits non zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/tests/expected/pathSimplifyTest
- lines changed 41, context: html, text, full: html, text
9e597a8f98e918d3c118943293cc7f45ec37768a Tue Sep 8 10:12:21 2026 -0700
lib/tests: add pathSimplifyTest, covering path canonicalization
Covers eatSlashSlashInPath and eatExcessDotsInPath, which had no test.
The test has two halves. The first prints the canonical form of 41 named
paths and the makefile diffs that against expected/, so a change in behavior
shows up as a diff. Those paths include the cases from #37263: a ".." that
climbs out of a relative path, a "." next to a "..", and a ".." at the root
of an absolute path.
The second half enumerates every path over the alphabets "/.a" up to length 9
and "/.ab" up to length 7, 51369 in all, and compares the answer against a
reference written inside the test. The reference builds a stack of components
instead of walking one buffer with two pointers, so the two are unlikely to
share a mistake. Only the count and the number of disagreements go in the
expected file, which keeps it small while the check stays broad.
A test that cannot fail is worth nothing, so this was checked against the
version of osunix.c from before the fix: it reports 32 differing named lines
and 1949 disagreements with the reference there, and none against the current
code.
refs #37263
- src/lib/tests/faSpeedReadTest.c
- lines changed 177, context: html, text, full: html, text
28538f8d9721d6b6503d16b4c6e7db1350cb7c53 Thu Sep 10 09:41:44 2026 -0700
lib/tests: add faSpeedReadTest, covering the FASTA read buffer growth, refs #38320
Eleven cases, each a set of line lengths. Each one writes a FASTA file,
reads it back with faMixedSpeedReadNext, and checks the name, the size,
every base and the terminating NUL.
The test frees the buffer before each case, so the growth starts from the
same place every time. That is what makes a shape reproduce a given sequence
of buffer sizes, and so what makes these shapes mean anything. Freeing also
means a write past the end of the buffer has to survive a free before the
next case can print.
The shapes are the one from the ticket, a variant whose overrun was small
enough to go unnoticed, the edges of the initial 65536 buffer, the first
doubling, a short line before a long one, a record spanning several
doublings, an evenly wrapped control, a single short line, and a two record
file so the second record is checked for anything the first left behind.
Against the code before the previous commit the test reports a short read
and differing bases on the first case, and exits non zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/tests/makefile
- lines changed 8, context: html, text, full: html, text
9e597a8f98e918d3c118943293cc7f45ec37768a Tue Sep 8 10:12:21 2026 -0700
lib/tests: add pathSimplifyTest, covering path canonicalization
Covers eatSlashSlashInPath and eatExcessDotsInPath, which had no test.
The test has two halves. The first prints the canonical form of 41 named
paths and the makefile diffs that against expected/, so a change in behavior
shows up as a diff. Those paths include the cases from #37263: a ".." that
climbs out of a relative path, a "." next to a "..", and a ".." at the root
of an absolute path.
The second half enumerates every path over the alphabets "/.a" up to length 9
and "/.ab" up to length 7, 51369 in all, and compares the answer against a
reference written inside the test. The reference builds a stack of components
instead of walking one buffer with two pointers, so the two are unlikely to
share a mistake. Only the count and the number of disagreements go in the
expected file, which keeps it small while the check stays broad.
A test that cannot fail is worth nothing, so this was checked against the
version of osunix.c from before the fix: it reports 32 differing named lines
and 1949 disagreements with the reference there, and none against the current
code.
refs #37263
- lines changed 8, context: html, text, full: html, text
28538f8d9721d6b6503d16b4c6e7db1350cb7c53 Thu Sep 10 09:41:44 2026 -0700
lib/tests: add faSpeedReadTest, covering the FASTA read buffer growth, refs #38320
Eleven cases, each a set of line lengths. Each one writes a FASTA file,
reads it back with faMixedSpeedReadNext, and checks the name, the size,
every base and the terminating NUL.
The test frees the buffer before each case, so the growth starts from the
same place every time. That is what makes a shape reproduce a given sequence
of buffer sizes, and so what makes these shapes mean anything. Freeing also
means a write past the end of the buffer has to survive a free before the
next case can print.
The shapes are the one from the ticket, a variant whose overrun was small
enough to go unnoticed, the edges of the initial 65536 buffer, the first
doubling, a short line before a long one, a record spanning several
doublings, an evenly wrapped control, a single short line, and a two record
file so the second record is checked for anything the first left behind.
Against the code before the previous commit the test reports a short read
and differing bases on the first case, and exits non zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 8, context: html, text, full: html, text
c2a6ef817930149ad6f55818fa0196d99c47e02b Fri Sep 11 10:40:11 2026 -0700
cheapcgi: skip a CGI pair with no =value instead of aborting, refs #38335
Both query string parsers looked for the '=' across the whole rest of the
string rather than inside the current pair. A pair with no '=' in it
therefore ran into the pair after it and took its value. "g-catV2&db=hg38"
was stored as one variable named "g-catV2&db", so db was lost with no
warning, and that corrupt name was copied on into the cart. The same pair
at the end of the string had no '=' left to find and aborted the whole
request, which is what the "Mangled CGI input string g-catV2" entries in the
hgw1 logs were.
Both parsers now find the end of the pair first, keeping the existing
separator precedence ('&', then ';' for DAS), and skip a pair with no '='.
A mixed "a=1;b=2&c=3" still parses the way it did.
Adds lib/tests/cgiParseTest, which runs 18 query strings through both
parsers. It covers the empty pair of #38185 as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- lines changed 8, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/tests/pathSimplifyTest.c
- lines changed 148, context: html, text, full: html, text
9e597a8f98e918d3c118943293cc7f45ec37768a Tue Sep 8 10:12:21 2026 -0700
lib/tests: add pathSimplifyTest, covering path canonicalization
Covers eatSlashSlashInPath and eatExcessDotsInPath, which had no test.
The test has two halves. The first prints the canonical form of 41 named
paths and the makefile diffs that against expected/, so a change in behavior
shows up as a diff. Those paths include the cases from #37263: a ".." that
climbs out of a relative path, a "." next to a "..", and a ".." at the root
of an absolute path.
The second half enumerates every path over the alphabets "/.a" up to length 9
and "/.ab" up to length 7, 51369 in all, and compares the answer against a
reference written inside the test. The reference builds a stack of components
instead of walking one buffer with two pointers, so the two are unlikely to
share a mistake. Only the count and the number of disagreements go in the
expected file, which keeps it small while the check stays broad.
A test that cannot fail is worth nothing, so this was checked against the
version of osunix.c from before the fix: it reports 32 differing named lines
and 1949 disagreements with the reference there, and none against the current
code.
refs #37263
- src/lib/textOut.c
- lines changed 9, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/lib/verbose.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/oneShot/dateXmlCgi/dateXmlCgi.c
- lines changed 3, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/parasol/paraHub/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/parasol/paraNode/makefile
- lines changed 1, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/submodules/submoduleSetup
- lines changed 8, context: html, text, full: html, text
cac1fca3406536a9f42965bbbebc101b62d8d8e3 Wed Sep 9 18:01:54 2026 -0700
submoduleSetup: anchor on kent/src so it works from either caller's cwd
Every path in submoduleSetup is relative to kent/src, but its two callers
invoke it from different directories: src/makefile runs
./submodules/submoduleSetup from src, while userApps/fetchKentSource.sh cds
into src/submodules first and runs ./submoduleSetup.
Since the zlib-ng change this broke the userApps source build. From
src/submodules the guard "[ ! -e submodules/zlib-ng/Makefile ]" is true
because that path cannot exist there, so the configure block ran and its
redirect to submodules/zlib-ng-configure.log failed on a missing directory:
./submoduleSetup: line 41: submodules/zlib-ng-configure.log:
No such file or directory
Error: zlib-ng configure failed, see submodules/zlib-ng-configure.log
make: *** [Makefile:28: fetchSource] Error 1
The htslib serial-probe block added for the AVX2 -j race has the same path
problem, but its guard is not negated, so from src/submodules it silently
tested a path that could not exist and never ran at all -- leaving the
userApps build exposed to the very race that block exists to prevent.
Fix both by cd'ing to the script's own parent directory up front, so the
caller's cwd no longer matters. Verified from both call sites: the
userApps path now configures zlib-ng and pre-generates htscodecs.mk with
correct probe flags (HTS_CFLAGS_AVX2 = -mavx2 -mpopcnt), and a re-run from
src is a quiet no-op.
refs #38125
- src/utils/genark/genark
- lines changed 78, context: html, text, full: html, text
679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI
Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.
New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.
Two additions serve every plot type, not just this one:
- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.
- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.
Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.
trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.
pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.
genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.
refs #35415
- lines changed 96, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- lines changed 16, context: html, text, full: html, text
6b0035d19769346baffe193ef9419c269d46f8d8 Wed Sep 9 06:09:41 2026 -0700
hprc2annot: pcLAI column 10 is the ancestry centroid, not a segment coordinate
Reading the pcLAI authors' own format description
(github.com/AI-sandbox/hprc-pclai, README "Output format (BED)") while building
the same annotation as a native hg38 track showed that column 10 of the source
BED had been described wrongly here. It is not the PCA coordinate of a longer
ancestry segment the window belongs to; the authors call it the centroid, the
discretized pcLAI ancestry of the window written as the PCA centroid of its
ancestry cluster. That is why it only ever takes four values -- four clusters,
not four long shared segments. The old reading also implied a segmentation step
the method does not have: pcLAI predicts one coordinate per window, and the
blocks visible in the display are runs of windows with similar predictions.
Field renamed pcaSegment -> centroid in pclai.as with the description corrected,
and the mouseOver, the detailsScript exportFields and the description page
follow. The README also settles that windows are a fixed 1000 SNPs rather than a
fixed number of bases, and that thickStart is specified to equal chromStart, so
the occasional thickStart == chromStart-1 the converter works around is a bug in
their files rather than something we misread.
A field name and its description live inside each bigBed, so editing pclai.as
does nothing to a built collection. hprc2annotRewriteAs.sh re-emits a built
bigBed with the current .as -- no re-download, no column change, item count
checked across the round trip, and safe to re-run, unlike hprc2annotFixBed.sh.
All 460 pclai.bb were rewritten with it. Worth knowing: those files had been
built from an older pclai.as than the tree and nothing had noticed, so this is
the tool to run after any .as description edit.
genark: the "...Url" inside a detailsScript value must not be rebased the way
bigDataUrl is. hgc resolves a relative detailsScript Url against the track's own
bigDataUrl when it builds the details page, and bigDataUrl has already been
rebased, so the prefix landed twice: the pcLAI scatterplot had been asking for
contrib/hprc2annot/contrib/hprc2annot/pclaiRefPanel.json and quietly getting
nothing on every GenArk hub. In this layout the panel file is symlinked beside
the .bb, so relative-to-the-.bb is the bare file name; rebaseBeside() does that
and is idempotent, so addContrib can be re-run.
refs #35415
- lines changed 9, context: html, text, full: html, text
2096509ab88749e811913cd2f07239f4972d4f18 Wed Sep 9 07:21:43 2026 -0700
genark: make addContrib's release tier a positional argument, not --tier
- src/utils/hubtools/hubtools
- lines changed 1701, context: html, text, full: html, text
a8694a3b22d43f0536c02101e9f3b56b5339b4dc Wed Sep 9 05:47:17 2026 -0700
hubtools: add "import igv" and "splitHap", and the bTaeGut7 zebra finch hub
import igv builds a hub from an IGV session XML. Every Track element becomes a
track, in session order, with the IGV display attributes translated to trackDb
settings. Files the browser can read over the network are linked where they are;
bed, gff, gtf, wig and bedGraph are downloaded and converted, which needs
chrom.sizes and gets them from --chromSizes, from the UCSC assembly, or from a
bigWig of the session itself, the only source there is for a custom assembly.
The BED cleaner exists because real files are not to spec: reversed start/end,
scores over 1000, "#rrggbb" colours, names past 255 characters, and columns that
are not the BED field they sit in, such as trf writing the repeat motif where
thickStart belongs.
splitHap turns a hub built on a diploid assembly into one hub with a genome per
haplotype, reading both assemblies' chrom.sizes and chromAlias from GenArk and
sending each record to whichever assembly has its sequence. It writes
splitHap.report.txt with the records per track per haplotype, the sequences
neither assembly has, and the records reaching past a sequence end, and checks
every track as it goes: records read must equal records matched plus records
with no sequence, and every match must produce an output record or a drop. A
track that does not add up stops the run rather than being written up as a
finding.
Two conversion fixes that came out of the zebra finch data. GFF3 requires unique
IDs, but an annotation of a phased assembly often gives both haplotypes the same
ID; gff3ToGenePred then merges the two copies into one transcript spanning two
chromosomes and discards it, which was losing 31 of 182 retrocopies. IDs that
occur on more than one sequence are now made unique per sequence first. And a
feature name is now taken from the first non-numeric attribute, so a
RepeatMasker GFF gives Motif:Tgut716A rather than the running number in ID=.
genark addContrib gains --tier alpha|beta|public. It edits only betaGenArk.txt
and publicGenArk.txt; beta.hub.txt and public.hub.txt are generated from those
lists and shipped by quickPush.pl, so writing them by hand would push content
outside the normal flow and lose it at the next clade build. The default alpha
tier leaves the lists untouched, so re-running an install cannot demote a
collection that is already promoted.
doc/contrib/bTaeGut7 and trackDb/contrib/bTaeGut7 are the zebra finch
telomere-to-telomere hub built with the above, from the IGV session the authors
ship with the annotations on GenomeArk (Formenti et al, Cell 2026, PMID
42561917). 21 tracks in 6 collections plus 3 standalone, 27 description pages,
and a makeDoc recording where every record went.
- src/utils/qa/weeklybld/buildEnv.csh
- lines changed 2, context: html, text, full: html, text
386bd6292e071d16bc62d6135cfe503e21c3a462 Mon Sep 14 10:14:43 2026 -0700
v504 preview2 (automated)
- src/utils/qa/weeklybld/buildHgCentralSql.csh
- lines changed 1, context: html, text, full: html, text
bc4639b87acf68232656130b2972293975e11933 Thu Sep 10 12:43:45 2026 -0700
genark: tolerate a missing or stale genarkOrg table, refs #38327
genarkGetOrgHash() aborted when the central database had no genarkOrg
table. A mirror's hgcentral has never had one: buildHgCentralSql.csh did
not list the table, so hgcentral.sql on hgdownload carries neither its rows
nor its schema. Add an sqlTableExists check, and add genarkOrg to the list
of tables that hgcentral.sql replaces entirely.
genarkMakeDbDb() defaulted genome to "Other" for an accession with no
genarkOrg row, but left organism NULL. hgConvert prints organism, so the
Convert page read "Genome: (null)". Default both. The copy of genarkOrg
on the RR is 20,754 rows behind hgwdev, so this is visible on
genome.ucsc.edu today for 23 of the 855 GenArk assemblies that appear in
liftOverChain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/utils/ts/README
- lines changed 18, context: html, text, full: html, text
26009fa434f2595ded075145fd322dc85dc6d518 Tue Sep 8 08:57:37 2026 -0700
ts: give each ticket sandbox its own trackDb cache
Every user's CGIs write their trackDb cache into one shared cacheTrackDbDir.
That is wrong for a park in two ways. The cached image is a shared-memory dump
whose layout is tied to TRACKDB_VERSION in the binaries that wrote it, so a park
built from a branch that touches the cache shares a directory with the live
sandbox and with everybody else. And the writer leaves a name.txt beside each
image and opens it with mustOpen, so whoever wrote it first owns the file and
the next writer's CGI dies on a permission error - after the image itself has
landed, so the next request succeeds and the failure reads as intermittent.
Each ticket now gets NNNNN/trackDbCache inside its own sandbox. Unlike the udc
cache this is derived from the code rather than data, so it is treated the
opposite way: freeze clears it, since the binaries it belongs to have just been
replaced, remove takes it with the sandbox, and conf retrofits it onto an
instance frozen before this existed.
refs #37867
- src/utils/ts/ts
- lines changed 35, context: html, text, full: html, text
26009fa434f2595ded075145fd322dc85dc6d518 Tue Sep 8 08:57:37 2026 -0700
ts: give each ticket sandbox its own trackDb cache
Every user's CGIs write their trackDb cache into one shared cacheTrackDbDir.
That is wrong for a park in two ways. The cached image is a shared-memory dump
whose layout is tied to TRACKDB_VERSION in the binaries that wrote it, so a park
built from a branch that touches the cache shares a directory with the live
sandbox and with everybody else. And the writer leaves a name.txt beside each
image and opens it with mustOpen, so whoever wrote it first owns the file and
the next writer's CGI dies on a permission error - after the image itself has
landed, so the next request succeeds and the failure reads as intermittent.
Each ticket now gets NNNNN/trackDbCache inside its own sandbox. Unlike the udc
cache this is derived from the code rather than data, so it is treated the
opposite way: freeze clears it, since the binaries it belongs to have just been
replaced, remove takes it with the sandbox, and conf retrofits it onto an
instance frozen before this existed.
refs #37867
- src/utils/uniprotToTab
- lines changed 183, context: html, text, full: html, text
af613a331e6839c6513c3e366abcb67af0fe8386 Wed Sep 9 06:47:14 2026 -0700
UniProt otto: get the monthly update running again and make a stalled run visible
The monthly UniProt job had produced nothing since January 2025. The tracks
served release 2024_06 while the download sitting on disk was at 2026_02, on
every assembly the job builds.
Cause: uniprotToTab appended a personal conda site-packages directory to
sys.path, and doUpdate.sh sourced a virtualenv, both built for python 3.6. A
venv's python is only a symlink to the system one, so when hgwdev moved to
python 3.9 the compiled lxml in there stopped loading and every run died at the
parse step. Removed both. The system python3 has lxml from python3-lxml and the
two are upgraded together, so there is nothing left here to go stale. Verified
by parsing real 2026_02 records under python 3.9 with lxml 5.4.
Why nobody noticed for nineteen months:
- doUpdate.sh read $? after an intervening echo, so it captured the echo's exit
code and mailed "Big Uniprot update OK" every month while the job was dying.
It now reads the real exit code, says FAILED, prints the tail of the log and
exits nonzero. A month with no new UniProt release stays silent, which is the
normal otto behaviour, so silence again means "nothing to do".
- The logs were overwritten on every run, so a failure left no trace on disk.
doUpdate.sh now appends one line per run to runLog.txt, which is never
truncated, and keeps a failing log as lastFail.log.
- version.txt in each bigBed directory was rewritten on every run even when the
release string was identical. That is the file the trackDb dataVersion setting
shows, and its date is what people check to decide whether a pipeline is still
alive, so a stalled track could look freshly updated. It is now written only
when the release actually changes.
Also, so this cannot come back:
- doUniprot checks that uniprotToTab can start before the download, instead of
finding out 35 minutes later.
- pylint on hgwdev is itself pinned to pythons that no longer exist, so
"make install" aborted on its first line and could not be used. Replaced with
a syntax check that needs nothing but python3; pylint stays best-effort.
- uniprotToTab, pslProtCnv, trackDb.template.txt and README.txt ran from
/hive/data/outside/otto/uniprot without being in the makefile's copy list.
The tree copy of uniprotToTab was still python 2 from 2021. All are now
listed and in sync, and "make diff" reports drift.
- Brought the two live-only fixes into the tree: mkdir -p in makeUniProtPsl.sh
and the pslMap -inType/-mapType flags.
refs #38300
- lines changed 11, context: html, text, full: html, text
bee41315e53da6fb5671f96302233c655906dfd5 Wed Sep 9 06:56:49 2026 -0700
UniProt otto: rebuild the venv rather than relying on a system lxml
Correction to the previous commit: hgwdev has no system-wide lxml at all. The
import I tested was resolving to my own ~/.local/lib/python3.9/site-packages,
which cron never sees, because it runs the pipeline as otto.
So the environment is a virtualenv again, but a reproducible one. makeVenv.sh
deletes venv/ and rebuilds it from /usr/bin/python3, installs lxml, opens up the
permissions for otto, and then checks that lxml imports with an empty environment
so we know the venv stands on its own instead of borrowing from whoever ran it.
Built with --copies, so venv/bin/python is a real copy rather than a symlink that
would silently follow a system python upgrade while its compiled modules stayed
behind.
doUpdate.sh activates venv/ again and says to run makeVenv.sh if it is missing or
if the parser will not start. Verified: /usr/bin/python3 without per-user packages
cannot import lxml, the venv can, and after activation the parser runs and
converts real 2026_02 records.
Also shortened the README to how the pipeline is started and how it works, and
trimmed the history out of the code comments, leaving the ticket as the pointer.
refs #38300
- src/webBlat/webBlat.c
- lines changed 2, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- src/weblet/counter/counter.c
- lines changed 1, context: html, text, full: html, text
cb99f0b11bdeee5dfa76064d38b6410da0f4a709 Thu Sep 10 00:55:21 2026 -0700
Centralize CGI Content-Type printing in one cgiPrintContentType() helper
Around 90 places across the tree hand-rolled the CGI response header, each
with its own spelling: "Content-Type:" or "Content-type:", \n or \r\n, and
the terminating blank line written as part of the same string, as a separate
puts("\n") (which emits two newlines, so a stray blank line led the body) or
as printf("\r\n\r\n") (two blank lines). A handful forgot the terminator
entirely and relied on a following header to supply it.
cgiPrintContentType() in lib/cheapcgi.c now writes the Content-Type line and
the blank line that ends the header. Header lines are not ordered, so the
callers that also send Status, Set-Cookie, Content-Disposition, Content-Length
or X-Sendfile write those first and call this last to close the header; that
keeps it to a single helper rather than a print-the-line / end-the-header pair
that a caller can half-use. cart.c's existing httpHeaders list already worked
this way.
Only the CGI response path is touched. The dyStringPrintf("Content-type: ...")
calls that build outgoing HTTP *requests* (genomeSpace, oauthLogin, eapMetaSync,
edwWebAuthLogin, ga4ghToBed) are unrelated and left alone.
Also fills out the apiKey error message in botDelay.c to say where to create a
key and that keys are server-specific.
No behavior change on the wire beyond dropping those stray blank lines and
adding the missing newline after Retry-After.
- lines changed: 48127
- files changed: 730
