Commits for braney
switch to files view, user index
v504_preview to v504_preview2 (2026-09-07 to 2026-09-14) v504
Show details
d0ed51d077718213c0b843a89fd6f36ac32f1804 Fri Sep 4 11:39:28 2026 -0700
- quickLift: lift alignments, and let psl and bigPsl tracks into the hub
Every type quickLift handled so far is one set of coordinates on one genome, so
the lift is a call to remapBlockedBed. An alignment carries coordinates on both
sides at once, with a block start on each, so moving the genome side means
splitting and trimming blocks while keeping the other side lined up with them.
quickLiftPsl does that with pieces that were already in the tree: pick the chain
with liftOverChainForRange, the same one the bed path picks, turn it into a
mapping alignment with chainToPsl plus pslSwap, then pslTransMap. The mapping
alignment is kept per chain, since rebuilding it for every item costs nothing at
gene zoom and a great deal zoomed out.
Two things pslTransMap does had to be undone. It recounts match and mismatch off
the blocks, which would read every lifted alignment as a perfect match and draw
every item at full shade, so quickLiftPslCounts puts the original counts back,
scaled by how much of the alignment survived. And it leaves a protein alignment
in nucleotide space, which the base alignment view rejects, so
quickLiftPslBackToProtein returns the query side to protein units when the lift
did not split a codon.
quickLiftChainHash hands out the chains for a reference range for callers that
collected their items some other way. validateOneTdb now accepts psl and bigPsl,
and bigPsl joins bigBed and bigWig in the bigDataUrl fill-in.
refs #38249
- src/hg/inc/quickLift.h - lines changed 18, context: html, text, full: html, text
- src/hg/lib/quickLift.c - lines changed 168, context: html, text, full: html, text
544f7054d812254a0883a412b98a08dcd78f1b84 Fri Sep 4 11:39:41 2026 -0700
- hgTracks: draw quickLifted psl and bigPsl tracks
pslTrack.c had no quickLift path at all. It now reads the alignments out of the
assembly the track came from through quickLiftSql and maps them onto the
reference. The chromFilter clause and the sort-and-filter tail moved into
helpers so the two loaders cannot drift apart.
bigBedTrack.c built the alignment out of the source interval but labelled it with
the reference sequence name and never lifted it, so a quickLifted bigPsl track
drew its items in the wrong place. It now takes the real source name out of the
bigBed, lifts, counts a failure the way the bed path does, and rewrites the
coordinate fields so a $chromStart in a mouseOver reports where the item is drawn.
cds.c asked the assembly on screen for the CDS and for the sequence the alignment
is to. Both belong to the assembly the alignment came from. This was already
the wrong answer for a quickLifted track on any target, and on a hub-backed
target it fails outright, because the name of such an assembly is not a database.
The table-exists answer is now remembered per assembly rather than once for the
process, since one page can hold both native and lifted alignment tracks.
The mRNA filter reads the same source assembly, for the same reason: the
accessions and the gbCdnaInfo ids are the source assembly's.
refs #38249
- src/hg/hgTracks/bigBedTrack.c - lines changed 45, context: html, text, full: html, text
- src/hg/hgTracks/pslTrack.c - lines changed 114, context: html, text, full: html, text
ee190bc09015bf1d10254691824b0a38f59e1aff Fri Sep 4 11:39:56 2026 -0700
- hgc: show the lifted alignment on a quickLifted psl or bigPsl details page
The details page and its alignment views all read the assembly on screen, so a
quickLifted alignment track either reported the position it had before the lift
or, for the base alignment views, could not find its item at all.
Finding the track was the first problem. aliTable is the table name from the
assembly the alignments came from, and that name usually belongs to a real table
on the assembly being viewed as well, so it cannot tell the two apart. The
alignment links now carry aliTrack, the track hgc was called on, built the same
way hgcAnchorSomewhere builds its table parameter. A hub track's trackDb only
reaches trackHash if its hub is attached in cartDoMiddle, which was already being
done for the two htcBigPsl commands and now covers the cdna and protein ones too.
quickLiftAliInfo resolves the trackDb, assembly, table and chain file behind an
aliTable and aliTrack pair, custom tracks included. quickLiftFindPsl finds the
alignment a link points at: only the reference position is known there and the
lift does not run backwards, so it reads every alignment of the accession out of
the other assembly, lifts them, and keeps the one that lands on that position.
htcCdnaAli and htcCdnaAliInWindow use that, and read the CDS and the query
sequence from the source assembly. htcBigPslAli and htcBigPslAliInWindow read
the source intervals through quickLiftGetIntervals and match on the lifted
position rather than on the raw interval. Several name tests that decide what
kind of alignment this is now skip the hub prefix, so the guard against a
translated alignment in a window is not bypassed on a lifted xeno track.
Separately, and not specific to quickLift: genericBigPslClick read through an
empty alignment list, which is what happens whenever nothing in the window
matches the item asked for.
refs #38249
42cfb8cd4a22209d8183ffcad97e5ca5e9ae070b Fri Sep 4 12:47:47 2026 -0700
- chainNetDbLoad: add chainLoadRange, every chain in a range with its blocks
chainLoadIdRange loads one chain by id, and the bigChain reader takes -1 to mean
every chain in the range, but there was no SQL equivalent of the latter. This is
it: one query for the headers, one for the links, and each link handed to the
chain it belongs to.
The blocks are sorted rather than just reversed. A bin indexed range query does
not return rows in position order, and anything that walks a chain's blocks
expects them ascending, which is why chainLinkAddResult sorts them too.
refs #38249
- src/hg/inc/chainNetDbLoad.h - lines changed 5, context: html, text, full: html, text
- src/hg/lib/chainNetDbLoad.c - lines changed 56, context: html, text, full: html, text
e70c7f68f61a3477631bd364c31f99c157528fc8 Fri Sep 4 12:48:02 2026 -0700
- quickLift: lift chains, and let chain and bigChain tracks into the hub
A chain is an alignment between the assembly the track came from and some other
species, so lifting one composes two alignments and leaves the user with chains
between the assembly on screen and that species.
quickLiftChain does it by the same route the psl work uses: chainToPsl, then
quickLiftPsl, then chainFromPsl, which is the inverse chainToPsl never had. It
does not modify the chain handed to it. That matters more than it sounds: every
chain loader leaves the header describing the whole chain while loading only the
blocks in range, so the header has to be corrected for the conversion, but the
callers still want the original. The details page reports it, and the track
sorts on it, which is what puts a lifted chain in the same row as the native one.
quickLiftSourceRanges hands back the ranges in the other assembly that the window
maps to, for callers whose items cannot be had from a query quickLiftSql knows
how to make. It works those ranges out by intersecting the window with each
chain block rather than taking the whole block, which matters here because one
block can be enormous: hg19 and hg38 run identical for 12.8Mb on chr7, and
asking for all of it turned a 39 chain window into a 1892 chain one.
quickLiftIsOwnChainTrack keeps quickLift's own chain track out of all this. That
stanza carries quickLiftUrl and quickLiftDb like any lifted track and is loaded
by bigChainLoadItems like any bigChain, but its data is already in reference
coordinates. The giveaway is that its bigDataUrl is the quickLift chain file.
validateOneTdb now accepts chain and bigChain. netAlign is still refused: a net
is a hierarchy of gaps rather than a plain alignment and wants its own thought.
refs #38249
- src/hg/inc/quickLift.h - lines changed 28, context: html, text, full: html, text
- src/hg/lib/quickLift.c - lines changed 195, context: html, text, full: html, text
fb55d9fe44a703a042d8ac6b586764819170f539 Fri Sep 4 12:48:13 2026 -0700
- hgTracks: draw quickLifted chain and bigChain tracks
quickLiftChainLoadItems is reached from both chainLoadItems and
bigChainLoadItems. It reads the chains out of the assembly the track came from,
maps them onto the reference, and hangs the lifted blocks on the item directly,
so unlike the native loaders it does not leave the components for loadLinks to
fetch afterwards.
The link file comes from linkDataUrl when the track names one. Deriving it from
bigDataUrl, which is what bigChainGetLinkFile does, misses the tracks that do not
follow that naming: hg19's chainHs1 calls it hg19.chainHs1Link.bb, so nothing
loaded and the track came up empty.
The item label reports where the whole chain starts on the other species, taken
before the lift, since the lifted alignment only covers the window.
refs #38249
- src/hg/hgTracks/chainTrack.c - lines changed 146, context: html, text, full: html, text
8fb78633bc427f68f8d2c6b1a75d8093883a0b80 Fri Sep 4 12:48:23 2026 -0700
- hgc: show the lifted chain on a quickLifted chain track's details page
chainLoadItemInRange is the one place all four chain detail consumers get their
chain, so lifting there covers the details page, the base by base alignment, its
translated variant, and quickLift's own difference page at once.
quickLiftChainInRange loads every chain over the source ranges and matches on id
rather than asking for the one id. The chain's sequence name in the other
assembly is not known at that point, and both by-id loaders abort when the id is
not inside the range they were handed.
The page used to end by saying the fields above describe the entire chain rather
than the part in the window. For a lifted chain that is not true, since only the
part around the window is ever worked out, so it now says that instead. Native
chain tracks keep the original sentence.
refs #38249
5fc426954da9ceb7dc42b9760858bbd1189760e1 Fri Sep 4 13:08:14 2026 -0700
- quickLift: lift MAF blocks, and let bigMaf and wigMaf tracks into the hub
Most of this was already written. mafSubset does the part that looked hard,
which is recomputing every row's start and size when columns are taken away, so
what was left was deciding where to cut.
quickLiftMafs cuts a block at every chain block boundary. Inside one chain block
the two assemblies run in step, so the columns carry over untouched and only the
first row's coordinates change. Across a boundary the reference either loses
bases or gains them, and either way the block can no longer be one contiguous run
on the reference, which is the one thing a MAF block has to be.
The reference row is named with the assembly name minus any hub prefix, since
that is the name the maf drawing code builds when it goes looking for it.
A minus strand chain turns the block over, so every row is turned over with it and
the forward start comes from the far end of the run. That path is written but has
not been exercised: no minus strand quickLift chain wins in a window I could
find.
validateOneTdb accepts bigMaf and wigMaf. Plain maf is left out on purpose:
those tracks are drawn by mafTrack.c, which has no quickLift path, so offering
them would hand back a track read from the wrong assembly.
refs #38249
- src/hg/lib/quickLift.c - lines changed 81, context: html, text, full: html, text
f2e67daf98177a40eea3e1d52364b52c4abf4728 Fri Sep 4 13:08:29 2026 -0700
- hgTracks: draw quickLifted bigMaf and wigMaf tracks
quickLiftLoadMafs is reached from both places a maf track loads its blocks. It
reads them over the ranges in the other assembly through the existing readers and
maps them onto the reference.
A lifted maf cannot use its summary. The summary names a table or a file in the
assembly the track came from, so it cannot be queried with reference coordinates,
and for a hub track the setting comes back rewritten as a path under the hub
besides, which is how it was failing: a range query against
../trash/quickLift/NNN/multiz100waySummary. inSummaryMode now says no for a
lifted track and the real blocks are read instead.
That costs time at a wide window, since reading the summary is exactly the work a
maf track avoids that way. Measured on multiz100way, hg19 lifted onto hg38: at
900 kb, where both read real blocks, native is 1.52s and lifted is 1.09s, so
nothing about the lift is slow. At 2.5 Mb, where native reads the summary,
native is 1.05s and lifted is 7.0s. Lifting the summary is the fix and wants its
own pass: a summary row is a coordinate range and a score, so mapping one is
liftOverRemapRange, but it is read in the drawing functions rather than in a
loader.
refs #38249
- src/hg/hgTracks/mafTrack.h - lines changed 7, context: html, text, full: html, text
- src/hg/hgTracks/wigMafTrack.c - lines changed 50, context: html, text, full: html, text
cec5ead054791f2a6601f308a56d280c08bd8ef7 Fri Sep 4 13:32:36 2026 -0700
- quickLift: do not take the lift path on half a pair of settings
A hub can set quickLiftDb without setting quickLiftUrl, and nothing filters hub
trackDb settings. The alignment loaders gated on quickLiftDb alone, so such a
stanza took the lift path with no chain file and hgTracks died in
endsWith(NULL, ".bb") from bigChainGetLinkFile, by way of quickLiftLoadChains.
Verified: SIGSEGV in strlen from common.c:1653, page truncated mid-HTML. This is
reachable on a production browser now that bigChain and bigMaf are liftable,
because those carry their own bigDataUrl and so need no trustTrackDb.
quickLiftIsLifted requires both halves, and every gate now uses it, which also
settles the two different predicates that were in use for the same question.
quickLiftLoadChains returns an empty list for a NULL file as well, so the older
bed and genePred callers are covered whatever their gate does.
quickLiftSql now checks that a row has at least as many columns as the loader is
going to read. The native psl loader has always checked this, and the quickLift
path replacing it did not, so a table of the wrong type walked off the end of the
row; the psl caller now states the 21 columns it needs.
quickLiftMafs held a maf component name in a fixed buffer through safecpy, which
aborts rather than truncates, so a long name from a hub took hgTracks down. It
clones instead.
htcBigPslAliInWindow used a trackDb pointer its lookup can leave NULL, which its
sibling htcBigPslAli already checked for. And aliTrackParam formats a URL
parameter into a fixed buffer with safef, which aborts on a long one.
refs #38249
- src/hg/hgTracks/chainTrack.c - lines changed 2, context: html, text, full: html, text
- src/hg/hgTracks/mafTrack.h - lines changed 5, context: html, text, full: html, text
- src/hg/hgTracks/pslTrack.c - lines changed 2, context: html, text, full: html, text
- src/hg/hgTracks/wigMafTrack.c - lines changed 2, context: html, text, full: html, text
- src/hg/lib/quickLift.c - lines changed 29, context: html, text, full: html, text
15e8651da02e09bbd37dbb404a966ed8c7d75167 Fri Sep 4 13:32:47 2026 -0700
- hgc: show the lifted alignment on a quickLifted maf track's details page
The page read the track's own file with reference coordinates and then went
looking for a reference row named for the assembly on screen. Both kinds of maf
track failed, differently: bigMaf aborted with "Couldn't find hg38.chr7 in maf"
out of mafFindComponent, and wigMaf drew a page with a title and nothing under
it, which is the worse of the two.
quickLiftClickMafs reads the blocks over the ranges in the other assembly, on
connections to that assembly, and maps them onto the reference, the same way the
track loader does.
refs #38249
02710f0107a6b154d9e5689165941ce724cb4c6a Fri Sep 4 13:38:54 2026 -0700
- quickLift: fix the seams a second review pass found
quickLiftPslBackToProtein left two things wrong. pslTransMap can hand back
strand[0] == '-', because it reverse complements the input when the two
alignments disagree about the strand of the sequence they share, and forcing
strand[1] to '+' on top of that produced "-+". A protein psl is only ever "++"
or "+-", and pslShow reads strand[0] == '-' as "reverse complement the query", so
it would have reverse complemented a protein as though it were DNA. It now turns
the alignment over so the minus lands on the target side. qBaseInsert is in
nucleotides like everything else being divided, so it comes down too, and it
joins the divisibility guard: without it the result failed pslCheck and the
number was printed verbatim on the details page.
Adding that back-conversion made a comment in pslTrack.c false. The lift no
longer always returns an untranslated alignment, so the drawing code has to ask
rather than assume, the way bigBedTrack.c already did. A quickLifted protein psl
track was drawing every block at a third of its length. No such track exists on
hg19 or hg38 today, so this was latent.
The normalized score on the chain details page was read from the assembly on
screen. Where that assembly has no such table the page died; where it has a
table of the same name, which is the common case for a self or a well known
chain track, it silently returned some other assembly's chain and printed a blank
score. It now reads the assembly the chain came from, on a connection to it.
Two smaller things: htcBigPslAli guarded its connection with trackHubDatabase
alone, but a GenArk accession does not start with hub_, so it matches the guard
genericClickHandlerPlus already uses; and the table name tests in cds.c now skip
the hub prefix the way the ones in hgc.c were changed to, so a lifted refSeqAli
reaches its special case.
The chain item label took its start from the source chain and its strand
character from the lifted one. Both now come from the source chain.
refs #38249
- src/hg/hgTracks/chainTrack.c - lines changed 2, context: html, text, full: html, text
- src/hg/hgTracks/pslTrack.c - lines changed 4, context: html, text, full: html, text
- src/hg/lib/quickLift.c - lines changed 11, context: html, text, full: html, text
353a34ac7e7638457db3f55e57452069113d860b Sun Sep 6 13:34:28 2026 -0700
- Add a bigNet track type, a net of alignments in a bigBed, refs #20824
Track hubs have had no way to show a real net. The usual stand-in is a
net rendered as a maf, which loses the level structure that makes a net
useful for establishing orthologous sequence. bigNet holds the netAlign
columns in a bigBed, so a hub can carry the net itself.
The format is bed6+20: the target in chrom/chromStart/chromEnd, the query
sequence in name, the query strand in strand, then level and the rest of
the netAlign fields. The trackDb line is
type bigNet <targetDb> <chainTrack>
mirroring type netAlign. chainTrack is the plain trackDb name of the
bigChain track in the same hub; hgc adds the hub prefix itself.
chainNetLoadRangeHub() builds a chainNet from a bigBed range query and
hands it to the same helpToNet() the SQL path uses, so the nesting is
rebuilt the same way. netDraw picks its loader off tg->isBigBed and the
drawing code below that is untouched. genericNetClick does the same for
the details page and follows the named chain track for the alignment.
Also bounds the level walk in helpToNet() by help->maxDepth. It could
read one past the end of the levels array.
netToBigNet converts a net file to bedToBigBed input. It writes the tab
line itself rather than calling bigNetTabOut, because autoSql prints a
double with %g and that drops digits off a chain score.
- src/hg/hgTracks/netTrack.c - lines changed 11, context: html, text, full: html, text
- src/hg/hgTracks/simpleTracks.c - lines changed 5, context: html, text, full: html, text
- src/hg/htdocs/FAQ/FAQformat.html - lines changed 1, context: html, text, full: html, text
- src/hg/htdocs/goldenPath/help/api.html - lines changed 1, context: html, text, full: html, text
- src/hg/htdocs/goldenPath/help/bigNet.html - lines changed 176, context: html, text, full: html, text
- src/hg/htdocs/goldenPath/help/trackDb/changes.html - lines changed 12, context: html, text, full: html, text
- src/hg/htdocs/goldenPath/help/trackDb/trackDbDoc.html - lines changed 1, context: html, text, full: html, text
- src/hg/htdocs/goldenPath/help/trackDb/trackDbHub.v3.html - lines changed 20, context: html, text, full: html, text
- src/hg/htdocs/goldenPath/help/trackDb/trackDbLibrary.shtml - lines changed 12, context: html, text, full: html, text
- src/hg/htdocs/goldenPath/help/trackDb/trackDbSettings.json - lines changed 16, context: html, text, full: html, text
- src/hg/htdocs/goldenPath/help/trackDb/trackDbSettings.yaml - lines changed 22, context: html, text, full: html, text
- src/hg/hubApi/apiUtils.c - lines changed 1, context: html, text, full: html, text
- src/hg/inc/chainNetDbLoad.h - lines changed 4, context: html, text, full: html, text
- src/hg/lib/chainNetDbLoad.c - lines changed 80, context: html, text, full: html, text
- src/hg/lib/trackDbCustom.c - lines changed 1, context: html, text, full: html, text
- src/hg/makeDb/trackDb/tagTypes.tab - lines changed 6, context: html, text, full: html, text
- src/hg/utils/hubCheck/hubCheck.c - lines changed 1, context: html, text, full: html, text
- src/hg/utils/netToBigNet/makefile - lines changed 3, context: html, text, full: html, text
- src/hg/utils/netToBigNet/netToBigNet.c - lines changed 151, context: html, text, full: html, text
- src/hg/utils/netToBigNet/tests/expected/simpleTest.bigNet - lines changed 14, context: html, text, full: html, text
- src/hg/utils/netToBigNet/tests/input/hg38.chrom.sizes - lines changed 1, context: html, text, full: html, text
- src/hg/utils/netToBigNet/tests/input/hg38.mm39.test.net - lines changed 15, context: html, text, full: html, text
- src/hg/utils/netToBigNet/tests/makefile - lines changed 22, context: html, text, full: html, text
f29b65452a1cca8a5bc62f007100b313e589d036 Sun Sep 6 15:11:36 2026 -0700
- quickLift: lift a bigNet track, refs #20824
A net is not an ordinary track to lift. It is an alignment of two assemblies,
so only its target side moves to the new reference; the query side names a third
assembly and is carried across untouched. And the browser draws a net by
recursion, so what really has to survive the lift is the tree: a row's level only
means anything relative to the row above it.
chainNetLoadRangeQuickLift() maps each row's target range with
quickLiftIntervalsToBedClip, which is the same code every other quickLift track
uses, so a net row lands where a bed of the same span would. The surviving rows
then go to the same helpToNet() the unlifted path uses, so there is one tree
builder and not two. The row collection either path does is now cnlHelperNew and
cnlHelperAddBigNet.
validateOneTdb lets bigNet into a quickLift hub. Its chain track is not offered,
so a lifted net's details page has no chain to follow and says so rather than
looking for a track this assembly does not have. bigNetLoadOne lifts the same
way for that page, unclipped, so it reports the item's whole extent.
bigNetFromInterval passes -1 as the cached chromId. bbiCachedChromLookup leaves
the buffer alone when the id matches the one before it, so a cache that outlives
the buffer hands back stale bytes.
Measured against the standalone liftOver tool, on hg19 chr22's mouse net lifted
to hg38: 36 of 36 source rows in the sampled window land on the same coordinates,
counting the one liftOver will not take whole, whose two ends it does place
exactly where the browser puts them. Against hg38's own mouse net, computed
independently, 1.02% of the drawn pixels differ.
- src/hg/hgTracks/netTrack.c - lines changed 4, context: html, text, full: html, text
- src/hg/inc/chainNetDbLoad.h - lines changed 16, context: html, text, full: html, text
- src/hg/lib/chainNetDbLoad.c - lines changed 98, context: html, text, full: html, text
2bd879990be58c3a533783a25f74b3b8e937dda4 Sun Sep 6 15:46:38 2026 -0700
- docent: make rm37389 assert what 12d4ad442f7 said it already asserted
12d4ad442f7 renamed rm37389 out of .xfail and its message said "the script now
also asks for a phrase out of the hub's own description file". It did not:
.../regress/{rm37389.xfail.docent.yaml => rm37389.docent.yaml} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
The rename was the whole commit. So the script has been asserting `text:
"Description"` and nothing else, while its own comment described a second check
that was not in the file. This adds it.
"Description" was not doing nothing -- measured on the crash1 hgc page, it occurs
exactly once in the visible text and it is the heading printTrackHtml writes above
the block, which goes away with the block. So this is a tightening rather than a
rescue. What the phrase adds is that it can only be on the page if the
description file was really fetched from the GenArk hub, which is the thing
#38275 broke.
The comment is rewritten to match, and is shorter. The bisect scaffolding is
gone -- the ruled-out list was worth writing while the cause was unknown and is
noise now that it is known. Both causes are kept, since a reader who sees only
one of them cannot understand why the script is worded as it is.
First commit on this branch since it was merged. The branch was 99 commits
behind and fully contained in master, so it was fast-forwarded to 79aa96ab96e
first. Note that the nightly runs from a clone of origin/master, so this does
not run nightly until the branch is merged.
refs #38252
- src/hg/utils/docent/tests/regress/rm37389.docent.yaml - lines changed 44, context: html, text, full: html, text
357d4dbeca6b3bbb59f60185f6b833d74fd74fbc Sun Sep 6 15:51:56 2026 -0700
- bigNet: four fixes from the code review, refs #20824
A quickLifted net's details page lifted the row unclipped so it could report
the whole extent, but the image lifts clipped. An item too big for the chains
loaded in the window is drawn clipped and was then unfindable on click, which
puts a box on screen that says it is not there. Try the unclipped lift, fall
back to the clipped one, and say plainly when the numbers describe only the
part that could be placed.
quickLiftGetIntervals can return one source row twice, through two chains whose
padded query ranges overlap. helpToNet cannot tell two identical parents apart:
the second inherits no children and then draws as one solid box over the first
one's gaps. A level, a target range and a chain id name a row in a net, so that
is enough to recognize the repeat and drop it. Preventive -- no duplicate was
observed in the window measured.
The sentence explaining why a lifted net has no alignment to show printed
quickLiftDb twice, and a hub can set quickLiftUrl and leave quickLiftDb unset,
so it could be handed a null. One printf, and it reads correctly either way.
Free the per-row bed in both lift loops. It is about ninety thousand of them on
a whole chromosome, which is more than a CGI should be asked to shrug off.
Rendering is unchanged: the unlifted net still draws pixel for pixel like the
native netAlign track at three widths, every lifted figure but the details page
is pixel-identical to the one built before these fixes, and the 36 of 36
agreement with the standalone liftOver tool is unchanged.
- src/hg/lib/chainNetDbLoad.c - lines changed 26, context: html, text, full: html, text
5f47cb8c328b76c41a9a21599477bfacf4d91a37 Sun Sep 6 16:57:16 2026 -0700
- makefiles: let the compiler write the header dependencies
The tree had 72 hand-written "foo.o: bar.h" lines across 15 makefiles, so
almost every object was rebuilt only when its own .c file changed. Editing a
header left every other object that included it holding the old layout, and the
crash landed somewhere the change never touched.
The %.o: %.c rule in inc/common.mk now passes -MMD -MP. The compiler writes
foo.d beside foo.o listing the headers that compile really read, and an
-include reads them back. Nine makefiles keep a compile rule of their own,
because they add -DGBROWSE, -DGFSERVER_HUGE or -DCGI_BIN=; each got ${DEPGEN}
too. lib and hg/lib are the only two that build objects into a subdirectory,
and each reads its own subdirectory .d files at the foot of its own file.
Touching hg/hgTracks/wigCommon.h used to rebuild 2 objects. It now rebuilds
11, which is every .c file in that directory that includes the header.
A .d file holds rules, and make takes its default goal from the first rule it
reads, included files and all. common.mk is read before a makefile's own
rules, so the include has to save $(.DEFAULT_GOAL) and set it back afterwards;
without that, make in lib built adjacency.o and stopped.
Fifteen link rules in directories the build enters named a library on the
command line without depending on it. Each now lists it. hg/hgPhyloPlace,
hg/visiGene/hgVisiGene and hg/orthoMap were the three whose target is a real
file and could go stale.
make clean still leaves the .d files. A shared clean:: rule in common.mk would
first mean converting about 320 single-colon clean: rules, since make refuses
to mix the two forms on one target. A leftover .d cannot break a build: -MP
writes an empty target for each header, so a deleted or renamed one does not
leave make asking for a file no rule can build.
refs #36621
- src/hg/affyTransciptome/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/encode/docId/lib/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/hgPhyloPlace/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/hgTracks/makefile - lines changed 2, context: html, text, full: html, text
- src/hg/hgTracks/makefile.hgRenderTracks - lines changed 1, context: html, text, full: html, text
- src/hg/liftAgp/makefile - lines changed 2, context: html, text, full: html, text
- src/hg/makeDb/genbank/common.mk - lines changed 1, context: html, text, full: html, text
- src/hg/makeDb/hgTomRough/makefile - lines changed 2, context: html, text, full: html, text
- src/hg/mouseStuff/whyConserved/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/orthoMap/makefile - lines changed 2, context: html, text, full: html, text
- src/hg/utils/makeTrackIndex/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/utils/refreshNamedSessionCustomTracks/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/visiGene/hgVisiGene/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/wikiPlot/makefile - lines changed 1, context: html, text, full: html, text
- src/parasol/paraHub/makefile - lines changed 1, context: html, text, full: html, text
- src/parasol/paraNode/makefile - lines changed 1, context: html, text, full: html, text
7e88fd5b88b421a27e9677737bf357fbe29c0e6c Sun Sep 6 17:16:14 2026 -0700
- makefiles: take out the hand-written header dependencies
The compiler records these now, so the lines are only a second copy that can
drift. They already had: hg/hgTables named 23 objects as depending on
hgTables.h, and thirteen more include it. Touching that header rebuilt 23
objects before this and rebuilds 36 after, and nothing that rebuilt before
stops rebuilding.
43 lines go, in hg/hgTables, hg/hgTracks, hg/blastToPsl, hg/genePredToMafFrames,
hg/pslDiff and hg/utils/refSeqGet, along with four extraHeaders variables that
were the same thing said another way.
A hand-written line stays wherever the header is generated, by autoSql, by
stringify or by a sed rule. On the first build there is no .d file yet, so
nothing else makes the generator run before the compile that needs its output.
That leaves hg/hgGeneRing, hg/visiGene/vgLoadMahoney, hg/qaPushQ, hg/lib and
hg/hgGateway, and trims extraHeaders down to the generated headers in
hg/pslCDnaFilter and hg/utils/overlapSelect. Both of those were cleaned and
rebuilt from a tree with no usage.h or algo.h in it to check the ordering still
holds. The C++ ones in optimalLeaf and hg/lib/straw stay as well, since -MMD
never sees a .cc file.
The genbank tree keeps its eleven lines untouched. It is built by no kent
target, and it does not compile at all today: mgcStatusTbl.c fails
-Werror=format. Nothing there could be verified, so nothing there was changed.
inc/userApp.mk now says extraHeaders is for generated headers, so the ordinary
ones do not come back.
refs #36621
- src/hg/blastToPsl/makefile - lines changed 4, context: html, text, full: html, text
- src/hg/genePredToMafFrames/makefile - lines changed 5, context: html, text, full: html, text
- src/hg/hgHubConnect/hooks/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/hgTables/makefile - lines changed 25, context: html, text, full: html, text
- src/hg/hgTracks/makefile - lines changed 7, context: html, text, full: html, text
- src/hg/makeDb/hgGoldGapGl/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/pslCDnaFilter/makefile - lines changed 3, context: html, text, full: html, text
- src/hg/pslDiff/makefile - lines changed 2, context: html, text, full: html, text
- src/hg/utils/overlapSelect/makefile - lines changed 3, context: html, text, full: html, text
- src/hg/utils/refSeqGet/makefile - lines changed 2, context: html, text, full: html, text
- src/hg/utils/tdbQuery/makefile - lines changed 1, context: html, text, full: html, text
d11a21c48cdc2cebe0a97779b510df8317b980cd Mon Sep 7 06:29:43 2026 -0700
- bigNet: gate the track type behind an hg.conf flag, refs #20824
Add the boolean hg.conf setting bigNet, default FALSE, so the type ships
dark and a machine turns it on with bigNet=on.
trackHubBigNetEnabled() in hg/lib/trackHub.c is the one read; the four
places that accept or advertise the type ask it. validateOneTrack drops
bigNet from the hub track type allowlist, validateOneTdb drops it from the
types quickLift will lift, hubCheck drops it from VALID_TRACK_TYPES and
from the message listing the valid types, and hubApi does not add it to
supportedTypes. netToBigNet, netTrack.c, chainNetDbLoad.c and the hgc
details code are unchanged, since they cannot be reached once a bigNet
track will not load.
With the flag off hgTracks does not abort the hub. It draws the track row
as a bigWarn bar reading "Unsupported type 'bigNet ...'" and the rest of
the hub loads normally.
Register the flag in hgConfCatalog.py with role="gate" so the sunset
report tracks it.
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 8, context: html, text, full: html, text
- src/hg/utils/hubCheck/hubCheck.c - lines changed 13, context: html, text, full: html, text
daa607de5aa0404e3b70f7752e52741ca6491810 Mon Sep 7 10:55:38 2026 -0700
- Merge branch 'makeDeps36621': generated makefile header dependencies
inc/common.mk passes -MMD -MP and reads the generated .d files back, so the
compiler records the header dependencies instead of a person. The second
commit then removes the 43 hand-written foo.o: bar.h lines that are now
redundant. Touching hg/hgTracks/wigCommon.h rebuilds 11 objects instead of 2.
hg/hgTables named 23 objects as depending on hgTables.h when 36 include it.
refs #36621
dbbb5418bed8155095c2dd7e6210e3ae909cd4a4 Mon Sep 7 11:17:57 2026 -0700
- Merge branch 'bigNet20824': bigNet track type, quickLift support, and its hg.conf gate
refs #20824
b18f2f9facbf4fede2ba522909535f133726c400 Mon Sep 7 11:20:43 2026 -0700
- bigNet: declare the bigBed types chainNetDbLoad.h uses, refs #20824
bigNetFromInterval takes a struct bbiFile and a struct bigBedInterval, and
the header named them without including bigBed.h. Any file that reached
chainNetDbLoad.h without bigBed.h ahead of it failed -Werror with "declared
inside parameter list", which is where hg/lib/liftOver.c ended up once the
generated header dependencies started rebuilding it.
- src/hg/inc/chainNetDbLoad.h - lines changed 4, context: html, text, full: html, text
992aeef92fea7be25a2acd916578898649212a32 Mon Sep 7 11:31:07 2026 -0700
- quickLift: gate the alignment lift behind an hg.conf flag, refs #38249
Add browser.quickLiftAlignments, default FALSE, so the alignment lift ships
dark and a machine turns it on with browser.quickLiftAlignments=on. It sits
beside browser.quickLift, the gate on the rest of the feature.
quickLiftAlignmentsEnabled() in hg/lib/quickLift.c is the one read, and
validateOneTdb in hg/lib/trackHub.c is the one place that asks it, before an
alignment track may enter a quickLift hub. That is the only door:
quickLiftUrl and quickLiftDb, the pair every lift path keys off, are written
by the quickLift hub writer and by nothing else, so with the flag off an
alignment track never gets them and the lifting, drawing and details code
behind them cannot be reached. pslTrack.c, chainTrack.c, wigMafTrack.c,
bigBedTrack.c and hgc.c are unchanged.
With the flag off hgConvert lists psl, bigPsl, chain, bigChain, maf, bigMaf
and wigMaf tracks in its "type is not supported by QuickLift" table, which is
what it did before this work. A hub built while the flag was on keeps working
after it is turned off, since its stanzas are already in the hub file in
trash, so this holds the feature back from people who have not used it rather
than switching off a session that has.
Read the hg.conf half with a literal cfgOptionBooleanDefault rather than
cartOrCfgOption so harvestHgConf.py can see it; a cart variable of the same
name still overrides it. Register the flag in hgConfCatalog.py with
role="gate" so the sunset report tracks it, and turn it on in
confs/hgwdev.hg.conf.
- src/hg/lib/quickLift.c - lines changed 17, context: html, text, full: html, text
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 19, context: html, text, full: html, text
e9ed25747e3d8ab10963306b826f7cedc5e71897 Mon Sep 7 11:35:30 2026 -0700
- Merge branch 'quickLiftAlign38249' -- alignment tracks in quickLift, refs #38249
# Conflicts:
# src/hg/lib/trackHub.c
- src/hg/inc/chainNetDbLoad.h - lines changed 29, context: html, text, full: html, text
36668c67d3b1b7ef9da683a2473ab1dea6be6709 Mon Sep 7 11:52:39 2026 -0700
- otto: a daily check that every otto job is still running, refs #38101
Otto jobs are silent when the source has published nothing, which is the
design and also why a job that stops running is invisible. #38280 is the
case that prompted this: a daily job whose source URL had disappeared ran
about 1,100 times over three years without a word.
The monitor asks one question per job, did it run when it was supposed to,
and answers it from a run stamp, meaning something the job leaves behind
whether or not the data changed. Where a job leaves nothing it is reported
as blind rather than as passing, so the gap stays visible.
A late job is not automatically a bug, so a late job with a source URL gets
that URL fetched. A dead source has to fail twice in a row before it becomes
a ticket. A live source means the failure was something else, and that files
the same day.
Filing is off unless --file is given, and the script is silent when every
job is on time.
ottoMonitorStamps.tsv carries the per-job run stamp. It comes from the
survey in /hive/groups/browser/redmineNotes/38101/claude/.
- src/hg/utils/otto/ottoMonitor/README - lines changed 86, context: html, text, full: html, text
- src/hg/utils/otto/ottoMonitor/ottoMonitor.py - lines changed 431, context: html, text, full: html, text
- src/hg/utils/otto/ottoMonitor/ottoMonitorStamps.tsv - lines changed 60, context: html, text, full: html, text
d1444e2ca8e21460228433810b1a6703c6520db1 Mon Sep 7 12:02:02 2026 -0700
- otto: keep the two surveys beside the monitor that reads them, refs #38101
The monitor's stamp table is the machine-readable half of a survey that
lived only in the redmineNotes directory, which is not a repository. The
reasoning behind each of the forty globs, and the measured curl shape
behind each source URL, were therefore one copy on /hive.
Both surveys now sit beside the script. Read ottoFailureSignatures.tsv
before changing a stamp glob: it says what each job writes and when, which
is the difference between a stamp that tracks every run and one that only
moves when the data changes.
- src/hg/utils/otto/ottoMonitor/README - lines changed 13, context: html, text, full: html, text
- src/hg/utils/otto/ottoMonitor/ottoFailureSignatures.tsv - lines changed 78, context: html, text, full: html, text
- src/hg/utils/otto/ottoMonitor/ottoMonitorStamps.tsv - lines changed 2, context: html, text, full: html, text
- src/hg/utils/otto/ottoMonitor/ottoSourceUrls.tsv - lines changed 70, context: html, text, full: html, text
3d96881a7939c2a0f32d4dcdf9f7f6dc7951a18c Mon Sep 7 12:02:30 2026 -0700
- Merge branch 'otto38101': daily otto failure monitor, refs #38101
c4bcca06ef1a06c434c9136a79459f1512cd0606 Tue Sep 8 07:41:02 2026 -0700
- docent: expect: can assert the color a track's row was drawn in, and a test for #36212
A bug about color leaves the page identical -- same rows, same height, same item
names, same tooltips -- so every check expect: had was blind to it. `color:` reads
the pixels instead: it names the color a row is mostly drawn in (`is:`), or one it
must not be (`not:`), with `part: label` for the center label rather than the
items, `at:`/`frac:`/`x:` for one item rather than the whole row, and a list form
so one step can state a whole color matrix and a failure name every row that came
out wrong.
hgTracks draws the whole view into one png and shows each row as a CSS-offset
slice of it, so a row's pixels are that slice drawn into a canvas at its offset.
The clipping box is the img's own div.sliceDiv, not the table cell: the center
label and the data are two slices inside one td_data_<key>, and measuring the cell
runs the canvas past the end of this row and into the next track's, which reads
that track's color as part of this one. The side labels are a separate png and are
never included, since "what color is this row" must not be answered by the label
text. White is background; everything else counts, black included, because a track
with no color of its own draws black items. is:/not: take r,g,b or #rrggbb and no
CSS color names, because trackDb's `color 0,255,0` is not CSS green.
tests/colorchecks covers all of it and tests/colorchecks.xfail aims all six forms
wrong in one step, so the failure has to name all six.
tests/regress/rm36212.xfail is the bug it was written for: a stanza that sets both
`itemRgb on` and `color` draws its items in the color setting instead of in the
file's own RGB column, because bedItemRgb() (hg/cgilib/bedCart.c) tests for the
presence of `color` before it tests for an explicit `itemRgb on`, so the explicit
setting is never reached. It is an xfail because the bug is live on the RR, on beta
and on genome-test.
It is also the first script in that directory that has been watched both to fail on
a build with the bug and to pass on a build with the fix -- the three-line reorder
built into parked #36212 and the same script pointed at that port. The fixture is
~/public_html/docentFixtures/itemRgbHub/, four tracks over four copies of one bed9
file whose items all carry a pure blue itemRgb column.
refs #36212, refs #37892
- src/hg/utils/docent/README.md - lines changed 27, context: html, text, full: html, text
- src/hg/utils/docent/docent.js - lines changed 139, context: html, text, full: html, text
- src/hg/utils/docent/tests/README.txt - lines changed 18, context: html, text, full: html, text
- src/hg/utils/docent/tests/colorchecks.docent.yaml - lines changed 56, context: html, text, full: html, text
- src/hg/utils/docent/tests/colorchecks.xfail.docent.yaml - lines changed 30, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/README.txt - lines changed 19, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36212.xfail.docent.yaml - lines changed 81, context: html, text, full: html, text
26009fa434f2595ded075145fd322dc85dc6d518 Tue Sep 8 08:57:37 2026 -0700
- ts: give each ticket sandbox its own trackDb cache
Every user's CGIs write their trackDb cache into one shared cacheTrackDbDir.
That is wrong for a park in two ways. The cached image is a shared-memory dump
whose layout is tied to TRACKDB_VERSION in the binaries that wrote it, so a park
built from a branch that touches the cache shares a directory with the live
sandbox and with everybody else. And the writer leaves a name.txt beside each
image and opens it with mustOpen, so whoever wrote it first owns the file and
the next writer's CGI dies on a permission error - after the image itself has
landed, so the next request succeeds and the failure reads as intermittent.
Each ticket now gets NNNNN/trackDbCache inside its own sandbox. Unlike the udc
cache this is derived from the code rather than data, so it is treated the
opposite way: freeze clears it, since the binaries it belongs to have just been
replaced, remove takes it with the sandbox, and conf retrofits it onto an
instance frozen before this existed.
refs #37867
59e2bbf2d3a99e5f18436a756e1e5c8b64308e33 Tue Sep 8 09:21:02 2026 -0700
- Copy a track collection's hub file when the program that writes it asks for a copy, instead of on every session load.
cartCopyLocalHubs ran on every session load, in every CGI and in four more places in
hgSession. For each customComposite-<db> cart variable it copied the collection's hub
file to a fresh trash name and registered the copy in hgcentral.hubStatus to get a hub
id. Nothing removed the old row. 81% of the 3.1 million rows in hubStatus on the RR are
these dead registrations, and each load also took the central_hubStatus advisory lock
and wrote to a MyISAM table, which serializes every concurrent load of a shared session
that carries a collection.
The copy itself is needed. A saved session's hub file lives under sessionDataDir and
every load of that session names it, so hgCollection must not write it in place. It was
just being made for every load, and almost no load is followed by an edit. hgCollection
is the only program that writes one of these files, so it now asks for its own copy:
main() calls cartRequestLocalHubCopy() before it opens the cart, and cartNew() makes the
copy for it.
That has to be a property of the program rather than of the request, and the copy has to
be made at cart open, above hubConnectLoadHubs. Copying the file gives the hub a new id,
the hubs are loaded during cart open, and trackList carries that id in every track name,
so a copy made any later leaves printTrackDbListToHub looking for the collection under
an id trackList does not have and the hub file comes out with a header and no tracks. A
condition that instead tries to work out whether this particular request will write
cannot be correct either: "cmd" is not in hgTracks' excludeVars, so it persists in the
cart and hgCollection dispatches on the cart rather than on the CGI variable, a settings
file can carry it, and a command-line run has no SCRIPT_NAME to test.
Three other parts. copyLocalHubs skips a hub the cart already owns, so repeated edits
reuse one file and one hub id rather than renumbering the hub on every drag; the test
requires a plain file, because saveTrackFile leaves the trash name behind as a symbolic
link to the durable copy and writing through that link would rewrite the saved session's
own file. copyLocalHubs also screens the path with isServerUserFilePath before opening
it, the way getHubName does, and selects on customComposite-<db> with the dash so that
the names it acts on are the ones fileNameCartVarPrefixes screens. saveTrackFile copies
when the source is a local hub outside trash that is not under this session's own
directory, so loading one session and saving it under a new name gives the new session
its own file instead of a reference into the first one's directory. pathIsUnderDir is no
longer static in trashDir.c.
All of it is behind the hg.conf gate collectionHubCopyOnWrite, default off, which
reproduces the old behavior exactly. Retiring the gate is not uniform: the body of
cartCopyLocalHubsOnSessionLoad is the old behavior and that function and its five
callers go away with the gate, while the other three tests lose only the gate term.
Measured on hgcentraltest with one binary. With the gate off, five loads of a session
carrying a collection add five hubStatus rows and three later edits add none; with it
on, the five loads add none and the first edit adds one. Saving a loaded session costs
two rows off and one on. The resulting hub file is byte identical either way apart from
the hub id, and the file the saved session names is unchanged by md5 and mtime,
including after an edit made straight after a save, and including when the cart names a
trash symbolic link into session storage.
refs #38273
- src/hg/hgCollection/hgCollection.c - lines changed 6, context: html, text, full: html, text
- src/hg/hgSession/hgSession.c - lines changed 8, context: html, text, full: html, text
- src/hg/lib/sessionData.c - lines changed 39, context: html, text, full: html, text
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 14, context: html, text, full: html, text
383ff66de9a1a37635a1d079cccdeb9842056524 Tue Sep 8 09:22:26 2026 -0700
- ottoMonitor: correct the job counts in the README and the survey header, refs #38101
The README's "WHAT IT ASKS" section said fifteen jobs write a log or a named
file, nine leave only a directory mtime, and eight leave nothing. Those add to
32, but the monitor watches 40. The real split is twenty-four, eight and
eight, which is what ottoMonitor.py -v reports. The old "fifteen" counted only
the external jobs with a positive stamp and dropped the nine internal ones that
also write a per-run file. The old "nine" came from counting the "dir mtime"
notes in ottoMonitorStamps.tsv, which catches ottoGitVsHive, but that job is
blind, not directory-mtime. The same off-by-one is fixed in the traps section.
The two surveys cover all 47 jobs in otto.crontab while the monitor watches 40,
so the README now says where the other seven went: they are the Cell Browser
jobs, marked monitor=no in ottoOwners.tsv.
ottoFailureSignatures.tsv opened by saying the 18 internal jobs were not
surveyed yet, three sections above a fully surveyed section for exactly those
18 jobs. The sentence was left over from the draft before that section was
added.
Also drop a dead clause in sourceIsUp(). "code == 226 or code == 0 and False"
reduces to "code == 226", because and binds tighter than or, so it read as if a
curl that could not connect were handled specially when it was not. Behavior
is unchanged: curl() returns 0 when curl itself failed, and 0 means the source
did not answer.
These are the two items the 2026-09-08 code review asked for, both in files
that tell a future editor to read them before touching a stamp glob.
- src/hg/utils/otto/ottoMonitor/README - lines changed 10, context: html, text, full: html, text
- src/hg/utils/otto/ottoMonitor/ottoFailureSignatures.tsv - lines changed 2, context: html, text, full: html, text
- src/hg/utils/otto/ottoMonitor/ottoMonitor.py - lines changed 3, context: html, text, full: html, text
53a8c1ac1513a14c299dc27940a45d97e6e5456f Tue Sep 8 09:33:22 2026 -0700
- bedItemRgb: let an explicit "itemRgb on" beat the presence of a "color" setting
A stanza could once say both "itemRgb on" and "color" and get both: items drawn
from the file's own RGB column, labels drawn in the color setting. Since 2025 the
color setting wins outright and the two tracks in Gerardo's test hub, one with
"color" alone and one with both settings, render identically.
The cause is the order of the tests in bedItemRgb(), not a missing feature. The
"color" test is only about whether to turn itemRgb on by DEFAULT, but it sat in the
same early return as the "itemRgb off" test, above the test for an explicit
"itemRgb on" -- so that test was unreachable for any stanza that set a color, and
an explicit setting could be overridden by the mere presence of one.
5448842337e added the color rule while a later block still honoured an explicit
setting; 88d620e6c82 folded the two tests together and dropped that block;
c54077c4044 added it back, but below the color test.
Moving the color test below both explicit tests restores the old behavior. Only one
of the four cases changes: "itemRgb on" plus "color" now returns TRUE. "itemRgb
off" still returns FALSE, "color" alone still suppresses the default, and a stanza
that says neither still follows the alwaysItemRgb hg.conf default. The label keeps
taking its color from the color setting either way, since that comes from
colorFromCart() rather than from here.
Measured before and after with a four-track hub whose items all carry a pure blue
itemRgb column and whose color settings are pure green: the both-settings track
went from green items to blue items, with its center label green throughout. The
other three tracks are unchanged. A Docent regression test asserts all four rows,
kent/src/hg/utils/docent/tests/regress/rm36212.xfail.docent.yaml.
refs #36212
- src/hg/cgilib/bedCart.c - lines changed 9, context: html, text, full: html, text
924c63fc1be1909f8ab2de587a968dfe1955f499 Tue Sep 8 07:19:24 2026 -0700
- lib: replace eatExcessDotDotInPath with eatExcessDotsInPath, and resolve a root ".."
simplifyPathToDir now canonicalizes with eatExcessDotsInPath instead of the
older eatExcessDotDotInPath, which is removed. simplifyPathToDir was its only
caller. The old routine scanned for the literal string "/../" and so gave
several answers that do not match realpath(3):
../../a became a (the second .. ate the first)
x/./../y became x/y (.. ate the ".", not the "x")
../.. became "" (an empty path, not the parent)
a/./b stayed a/./b (single dots were never removed)
h/../../../etc/passwd became etc/passwd (the escaping .. were eaten)
The last one is the reason to bother. A caller that wants to know whether a
path stays inside a directory cannot tell from the old result, because a path
that climbs out comes back looking like it stayed in.
Two changes to eatExcessDotsInPath came out of this.
It now drops a ".." at the root of an absolute path, so /../a gives /a and
/.. gives /, as realpath(3) does. Because an absolute path can then never
hold a "..", the guard is exactly "nothing consumed and not absolute".
It also returns "." rather than "" for a non-empty relative path that reduces
to nothing. Callers join the result with "%s/%s", where an empty string names
the file system root instead of the current directory. Without this,
"tdbQuery -root=." looked for /tagTypes.tab and died. The in-place write is
safe because a non-empty input always leaves room for one byte.
The DEBUG selftest is rewritten. Two of its assertions asserted the old wrong
answers for /.. and /../a, a third followed from them, and three asserted the
empty-string result now replaced by ".". Added cases for each item above.
Verified: all 38 selftest cases pass against the built library, and 48089
exhaustively enumerated paths over the alphabets "/.a" and "/.ab" match an
independent model with no ASan or UBSan report. tdbQuery and raSqlQuery give
byte-identical output to the master build across every root form tried,
except two that the master build got wrong.
The other caller of eatExcessDotsInPath is resolveDotDots, which hgTrackUi
uses to canonicalize a fileUrl before checking it against a hub's base
directory. Neither change loosens that check: a path that used to
canonicalize to /../secret now gives /secret, and neither is under a hub base
directory.
refs #37263
cbb406cd96e1984f11e6ddc00a9b5f9c3c60db2e Tue Sep 8 09:48:40 2026 -0700
- Merge branch 'itemRgb36212': an explicit 'itemRgb on' beats the presence of a 'color' setting, refs #36212
db142d39a8dc282f2fd952aa634c842e3e4c5448 Tue Sep 8 09:48:40 2026 -0700
- Merge branch 'docentTests37892': expect: can assert the color a track's row was drawn in, plus the regression test for #36212 and the rm37389 tightening, refs #37892, refs #36212, refs #38252
9e597a8f98e918d3c118943293cc7f45ec37768a Tue Sep 8 10:12:21 2026 -0700
- lib/tests: add pathSimplifyTest, covering path canonicalization
Covers eatSlashSlashInPath and eatExcessDotsInPath, which had no test.
The test has two halves. The first prints the canonical form of 41 named
paths and the makefile diffs that against expected/, so a change in behavior
shows up as a diff. Those paths include the cases from #37263: a ".." that
climbs out of a relative path, a "." next to a "..", and a ".." at the root
of an absolute path.
The second half enumerates every path over the alphabets "/.a" up to length 9
and "/.ab" up to length 7, 51369 in all, and compares the answer against a
reference written inside the test. The reference builds a stack of components
instead of walking one buffer with two pointers, so the two are unlikely to
share a mistake. Only the count and the number of disagreements go in the
expected file, which keeps it small while the check stays broad.
A test that cannot fail is worth nothing, so this was checked against the
version of osunix.c from before the fix: it reports 32 differing named lines
and 1949 disagreements with the reference there, and none against the current
code.
refs #37263
- src/lib/tests/expected/pathSimplifyTest - lines changed 41, context: html, text, full: html, text
- src/lib/tests/pathSimplifyTest.c - lines changed 148, context: html, text, full: html, text
52357be3947a4643b735276094b8d01da8e3f8f0 Tue Sep 8 10:30:02 2026 -0700
- ottoMonitor: check the last closed grace window, and let a job be owned by whoever is on duty
Three changes, all from comments on the ticket.
Lou: civic has no individual owner, so it belongs to the otto person. The owner
column of ottoOwners.tsv now accepts ottoOnDuty, which the monitor resolves from
the ottoOnDuty header at run time, so the rotation stays a one-line edit. The
ticket body says the job has no individual owner, and the same person is not
added as a watcher twice when the owner is also the person on duty.
The grace window is now measured back from the last scheduled time whose window
has already closed, instead of forward from the latest scheduled time. Written
the other way, a daily job scheduled fewer than graceHours before the monitor's
own 12:15 run could never be reported late, because every check landed inside a
fresh window. Six of the forty jobs were in that hole: clinGen, genArkPushRR,
grcIncidentDb, liftRequest, omim and pubtatorDbSnp.
Max: a uniprot run can take days, and how long depends on the size of the
release. Its stamp is created by a > redirect when the run starts, so the grace
does not have to cover the run length, and a fresh stamp does not mean the run
worked. That limit is now written down in the stamps table and the README, with
the live case: the uniprot run dies after about 37 minutes on a missing lxml and
has produced no output since January 2025, while the monitor reads it as on
time.
refs #38101
- src/hg/utils/otto/ottoMonitor/README - lines changed 17, context: html, text, full: html, text
- src/hg/utils/otto/ottoMonitor/ottoMonitor.py - lines changed 36, context: html, text, full: html, text
- src/hg/utils/otto/ottoMonitor/ottoMonitorStamps.tsv - lines changed 11, context: html, text, full: html, text
67700f0c574a5b49df7e49eff54e16221ff7e6ea Tue Sep 8 15:38:12 2026 -0700
- cart: accept a session file path spelled through a symlinked config directory
The file-name check added in 9ad04e0a0b0 compares a path against trashDir(),
sessionDataDir, sessionDataDirOld and myVariantsDataDir as plain strings. Those
directories are often reached through a symlink, and sessionData.c stores the
resolved spelling whenever the trash file it is saving is already a relative
symlink, so a saved session can hold either spelling. On the RR /userdata is a
symlink to /shared/userdata and 583 saved multi-region sessions hold the
resolved form, so hgTracks dropped multiRegionsBedUrl and said "No BED or BED
URL specified" rather than drawing the saved view.
Accept a path under a trusted directory or under whatever that directory
resolves to. Only the configured directory is resolved. Resolving the value
itself is not an option, because a trash file is deliberately a symlink into
session storage.
Use a stack buffer rather than realpath(dir, NULL): that allocates with the
system malloc, and freeMem() goes through the kent handler stack, which under
pushCarefulMemHandler() reads a block header that is not there. hgc, hgTables,
hgVai, hgLogin and hgLinkIn all install that handler before loading a cart.
refs #38303, refs #37623
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
71475dc7c4a4a603ed7334eb4d27fbda8abdbdec Tue Sep 8 17:54:23 2026 -0700
- cart: leave the trash directory out of the symlink-resolving path check
Code review of 67700f0c574 found that resolving trashDir() does more than widen
a check. trashDir() is the relative "../trash", so its resolved spelling is a
different string, and sessionDataPathFromTrash() substitutes exactly the
relative spelling. A path accepted as trash in its resolved spelling therefore
rewrites to itself, and the callers treat the two as two files:
saveTrackFile() opens the old one for reading and truncates the new one for
writing, then unlinks the old name and points it at itself, and
sessionDataSaveTrashFile() unlinks the file and then aborts when link() fails.
Since ctfile_<db>, customComposite-<db> and hubQuickLift-<db> reach
saveTrackFile() from a cart value, a session save could have destroyed another
session's file.
Resolving a relative directory also resolves it against the working directory of
the process, so the answer moved with the caller: on the command line trashDir()
is $JKTRASH or ".", which made anything under $TMPDIR or under cgi-bin count as
trash.
Put isTrashPath() back on the plain string compare and keep the resolving
version for sessionDataDir, sessionDataDirOld and myVariantsDataDir, which are
the configured absolute directories the #38303 sessions are stored under. Only
resolve an absolute directory, so the result no longer depends on the working
directory. All 297,487 saved sessions in the September dumps of the RR, hgwbeta
and hgwdev centrals were checked: none holds a resolved trash path, so nothing
loses the widening.
refs #38303, refs #38304, refs #37623
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
236b28263edfc3b0cb9da780d65d2b647a0d98a6 Wed Sep 9 08:47:28 2026 -0700
- docent: rm35920 was reading hg38's own ultras track, not the fixture hub's
The nightly went red on 2026-09-08 at rm35920's tooltip check. The cause was not
the bug the script is about. hg38 has a native `ultras` track under the
unusualcons superTrack, our copy of the reporter's hub called its track `ultras`
too, and a name resolves to `img_data_<name>` before it resolves to a hub row's
`hub_<n>_<name>`. The exact native id won every time: `track: {ultras: pack}`
turned on the native track and its superTrack, and `mouseover: {track: ultras}`
read the native row. The native items are named uc.N as well, so
`expect: {tip: "uc.1"}` passed on native data and nothing warned. The script
looked green for as long as it existed and tested nothing.
The fixture at ~/public_html/docentFixtures/Auto-generated_hub/ now calls its
tracks rm35920Ultras and rm35920UltraZoos, and declares `visibility pack` itself,
so the `track:` step is gone -- a hub track's cart name carries the per-run
hub_<n>_ prefix, which `track:` cannot write, so that step only ever moved native
tracks. README.txt gains the rule: a fixture hub must never name a track anything
a native assembly might also call it.
With the collision gone, what the two malformed tracks do on genome-test, hgwbeta
and the RR alike, measured 2026-09-09:
rm35920Ultras bigBed 12 + over a 4-field file -- row drawn, EMPTY. No crash,
no dialog, no garbage. #19984's auto-detection does not apply
when the declared type carries a number, so hgTracks trusts
the 12 and drops every row.
rm35920UltraZoos bigBed 4 + over a 3-field file -- items drawn with an EMPTY
name, byte for byte what a correctly declared `type bigBed 3`
over the same file produces, measured against a probe hub.
That is the garbage string gone, and it is #31771's fix.
So the tooltip assertion cannot be restored: the hub track has no named item to
hover, on any server. The script now asserts the crash half (both rows drawn, no
jsEmbedded) and clicks an ultraZoos item by position -- `at:` rather than `item:`,
because the fixed behavior leaves those boxes nameless -- and asserts the click
reaches a real hgc detail page. Checked against a deliberately wrong expectation
so it is not passing vacuously.
refs #35920, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/README.txt - lines changed 10, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm35920.docent.yaml - lines changed 58, context: html, text, full: html, text
fe722461f833a6ccf990003f22be9fe9e8962ebf Wed Sep 9 08:47:44 2026 -0700
- docent: promote rm36212 out of .xfail, the fix is on master
The nightly reported "this was supposed to fail, and it passed" on 2026-09-09.
cbb406cd96e (an explicit `itemRgb on` beats the presence of a `color` setting)
reached origin/master and so genome-test, and the script's color assertions now
hold there: itemRgbAndColor draws its items 0,0,255 from the file's own RGB
column while its center label stays 0,255,0.
It ships in v504, so the script still fails on the RR and on hgwbeta until that
release goes out. The header records that, and the three-server measurement is
kept with the dates rather than deleted, since it is the before half of the only
before-and-after this directory has.
refs #36212, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36212.docent.yaml - lines changed 11, context: html, text, full: html, text
a93b9b879a141a4eeea0cf6505add4d8e7acbc97 Wed Sep 9 09:09:35 2026 -0700
- Merge branch 'docentTests37892': two red docent regression scripts, refs #38252
The nightly went red on 2026-09-09 with two failures.
rm35920 was reading hg38's own ultras track rather than the fixture hub's. The
fixture called its track `ultras`, hg38 has a native `ultras` under unusualcons,
and an exact `img_data_ultras` beats a hub row's `hub_<n>_ultras`, so the script
had been asserting a tooltip off native data for as long as it existed. The
fixture's tracks are renamed, the `track:` step is gone, and the assertions now
match what the fix actually produces. Filed #38310 off what that exposed: a hub
track that declares more bigBed fields than the file has draws an empty row with
no warning.
rm36212 passed while marked .xfail, because cbb406cd96e reached master and so
genome-test. Promoted.
README.txt gains the rule the first one is about: a fixture hub must never name a
track anything a native assembly might also call it.
refs #35920, refs #36212, refs #38252, refs #38310
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
00811fcbcfd1c9a317b07477ff46a14dba046160 Wed Sep 9 12:39:44 2026 -0700
- bigBed: use the file's own field count when the type line asks for more, refs #38310
A `type bigBed N` larger than the number of fields the file holds left the
track drawing nothing at all, and its details page reporting a disagreement
instead of the item. Fall back on the count in the file's header, which is
the count hubCheck already requires the type line to match.
The bound has to be the file's total field count and not its definedFieldCount.
Fifty-two tracks legitimately declare more bed fields than their header calls
defined, forty-eight of them the hs1 T2T_Encode_LOPeaks narrowPeak set, and
those are untouched.
Three tracks are in the over-declared state today, all `type bigBed 4` over a
three-field file: hg38 setDups, and the KAPA_HyperExome and
nexterarapidcapture subtracks of hg19 exomeProbesets. Their item boxes render
pixel-identically and their details pages, which failed before, now work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgTracks/bigBedTrack.c - lines changed 5, context: html, text, full: html, text
- src/hg/hgc/bigBedClick.c - lines changed 5, context: html, text, full: html, text
e2d86e5b1db3e603f503f58db38009fa31840dd5 Wed Sep 9 12:37:09 2026 -0700
- hgTracks: don't read past the end of exonFrames on a transcript's last exon
The exon mouseover works out the codon phase at each end of an exon. The end
phase is the frame of the next exon along the transcript. On the last exon of
a forward-strand transcript there is no next exon, and the index has reached
the number of exons, so the read was one element past the end of the array.
The reverse-strand branch already guarded the same case at its own end of the
transcript.
Nothing the reader sees changes. makeExonFrameText prints an end phase only
when the exon is not the last one, so the value read here was always thrown
away. Measured on two builds from this tree, patched and not: 591 codon-phase
tooltips at chr12:459,900-462,400 are byte for byte the same, while valgrind
reports the invalid read in the unpatched build and none in this one.
refs #38309
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgTracks/simpleTracks.c - lines changed 4, context: html, text, full: html, text
133df533c4df3b1ebfcf3bbf14ab070f7d5c2ff2 Wed Sep 9 12:52:00 2026 -0700
- sessionData: return kent-allocated memory from sessionDataSaveTrashFile
sessionDataSaveTrashFile() returned the result of realpath(path, NULL) when the
trash file it saves is already a relative symlink, which is what the trashCleaner
scripts leave behind. That pointer comes from the system malloc, but all three
callers release it with freeMem() or freez(), which dispatch through kent's own
handler stack. Under pushCarefulMemHandler() the free reads a block header that
was never written and subtracts a garbage size from the running total, so the
next allocation dies with "carefulAlloc: Allocated too much memory".
Nothing reaches it today. hgSession, hgPhyloPlace and snapshotSession are the
callers and none of them installs that handler, so the bug was latent. The fix
is a PATH_MAX stack buffer and a cloneString, the same shape already used by
pathIsUnderDirOrItsTarget() in trashDir.c. PATH_MAX needs an explicit include
of limits.h.
That same branch also dropped the readlink buffer on the floor, because it
replaced newPath with the resolved path instead of with the link target. It is
freed now.
Checked with a probe linking jkhgap under pushCarefulMemHandler: the patched
function resolves the symlink, frees, allocates again and passes
carefulCheckHeap, while the old pattern exits with a negative total.
refs #38318
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/lib/sessionData.c - lines changed 7, context: html, text, full: html, text
68f831e12093cd1d85f814aea73f3fa33b63d322 Wed Sep 9 12:54:14 2026 -0700
- Merge branch 'fieldCount38310': use the bigBed file's own field count when the type line asks for more, refs #38310
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a1c0911798f9e50f0af79db624b8de840af4854f Wed Sep 9 13:02:54 2026 -0700
- hgc: zero the stack refLink in doKnownGene so the details page stops reading uninitialised memory
doKnownGene builds a refLink two ways. When the accession contains NM_ it
loads a real row. Otherwise it uses a plain struct on the stack and fills in
three of the eight fields. A struct declared that way is not zeroed, and two
of the remaining fields are read further down: prKnownGeneInfo tests omimId and
prints an OMIM link built from it, and geneShowPosAndLinksPal passes protAcc to
hGenBankHaveSeq as a string.
No knownGene name on hg38, hg19 or mm39 contains NM_, so every knownGene
details page on those assemblies takes the else branch. The OMIM link appears
and disappears between requests to the same URL and carries a meaningless
number when it does. The protAcc read is worse: run hgc from the command line
on such a page and it segfaults in strlen, reached through checkIfInTable.
Zero the struct before filling it in. The fields that are not set then read as
absent, which is what the else branch already means. Also guard the
hGenBankHaveSeq call on a NULL pepName, because a NULL string argument to
sqlSafef reaches sqlCheckError, whose default level is abort.
Note for anyone moving code between these two functions later: prRefGeneInfo
dereferences rl->product, and doKnownGene does not call it. A zeroed struct
would be a NULL dereference there.
Found with valgrind, refs #38316.
refs #38317
6b2ef5505a6121a2343661bea48ec63d4bdb7477 Wed Sep 9 09:24:32 2026 -0700
- docent: itemXY picked another track's item when this track had none
`mouseover: {track: X, item: "n"}` and `click: {track: X, item: "n"}` both resolve
the name through itemXY, which gathered every map box on the PAGE carrying that
name and then did:
const inBand = cands.filter(h => h.inBand);
const pick = (inBand[0] || cands[0]) || null;
so when track X had no box of its own, `cands[0]` handed back a box belonging to
some other track. Nothing warned. The step passed, the tooltip that came up was
real, and it was the wrong row.
This is not hypothetical. A hub track that declares more bigBed fields than its
file has draws no items at all (#38310). A probe hub with eight tracks over one
file, four of them drawing nothing, reported all eight as drawing uc.1, because
hg38 and the three tracks that do work use that name too. rm35920 had been
reading hg38's native `ultras` rather than its own fixture hub for as long as it
existed, and looked green the whole time. An answer that is wrong but reads as a
pass is worse than a failure.
Candidates are now scoped by MAP NAME, which is how areaXY and itemXY's own error
message already picked a row: hgTracks names each map after the track it draws
(map_data_<key>, map_center_<key>), so the test is exact. The y-band survives only
as the tie-break between several boxes OF THIS TRACK that share a name -- geometry
was never a safe primary test, since a packed row stacks items above and below its
middle and a quickLift target puts them outside the band altogether.
The one case that still gets to decide by geometry is a page where NO map can be
attributed to this key at all, i.e. hgTracks named it something we do not
recognise. `anyMine` keeps the old behavior there rather than turning an
unrecognised name into a hard failure.
When the lookup now fails, the error says where the name actually was, which is
the sentence that would have saved the most time here:
item "uc.1" not found in track "pT5" ... 1 map box(es) in that row, addressable
as: title: "Click to alter the display density of pT5". That name IS on this
page, in hub_195310_pT1, hub_195310_pT2, hub_195310_pT8 -- another track's box
is never used for this one
Checked against that probe hub: the three tracks that really draw uc.1 still pass,
the ones that draw nothing now fail with the message above. Both suites are green,
tests/ (15) and tests/regress/.
refs #38252, refs #38310
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/docent.js - lines changed 62, context: html, text, full: html, text
8cb4edc2f77e57fb03e7a043a287c40d74b6814e Wed Sep 9 13:16:19 2026 -0700
- docent: an xfail for #36888, a bigMethyl track that vanishes when zoomed out
Mark's report is that a bigMethyl track draws its data at base level and has
nothing between its labels two 100x zoom-outs later. No "zoom in" message, no
empty row, no error.
One missing method causes it. bigBedSelectRangeExt (hg/hgTracks/bigBedTrack.c)
asks for bigBedMaxItems()+1 intervals and throws the whole list away when the
count is over the limit, recording it in the trackDb setting bigBedItemsCount.
Every other bigBed type recovers, because commonBigBedMethods installs
track->loadSummary = loadBigBedSummary, which reads that setting and falls back
to the coverage graph. bigMethylMethods (hg/hgTracks/simpleTracks.c) sets
isBigBed and its own loadItems but never loadSummary, so an overflowed window
gets an empty item list and nothing else.
Watched both ways on 2026-09-08, which most of this suite cannot claim:
genome-test (v503, unfixed) draws no img_data_ row and no note; ts park 48090,
v503 plus the one-line loadSummary, draws the row and the note. Both checks
flipped together. The fix is not on master, so this is an .xfail: the day it
reaches genome-test the script passes, `make test` fails because an xfail
passed, and the fix is to drop the .xfail from the name.
The fixture is ours, at ~/public_html/docentFixtures/bigMethyl36888/, a
single-file hub over a 6 MB synthetic bigBed. The ticket's own hub is a 24 GB
file in another user's public_html, and #37490 already lost a fixture that way.
The two windows straddle the 100,000-item limit by a factor of six, so raising
bigBedMaxItems a little cannot quietly defeat the test, and the control step
plus noHas: mean a fixture that stopped resolving fails loudly instead of
satisfying the xfail.
The note is checked with has: on the track's control link, not with text:,
because labelTrackAsDensityTooManyItems appends it to longLabel and hgTracks
draws longLabel into the center-label image. The selector matches both title=
and mouseovertext=, since addMouseover() in hg/js/utils.js moves the text
between them under the showMouseovers hg.conf setting.
refs #36888, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm36888.xfail.docent.yaml - lines changed 80, context: html, text, full: html, text
bb27fdc2f612380ae05efcd1959b4b26297738e5 Wed Sep 9 13:16:38 2026 -0700
- docent: an xfail for #38310, and the second script that needs pixels
A hub track whose `type bigBed N` declares more fields than the file holds drew
its row with no items in it. The fix is written and verified but is not on
master, so no server passes this today. That is the only reason it is an .xfail:
`make test` fails if an xfail passes, so the day the fix reaches genome-test the
suite says there is a test waiting.
The ticket says the row came up empty "with no warning". The row is empty, but
there is a message. hgTracks catches the abort into networkErrMsg and swaps in
bigDrawWarning, which paints the row as a pale yellow bar, 240,240,180
(undefinedYellowColor, hg/hgTracks/simpleTracks.c), with the text inside it. The
text read `invalid signed integer: ""`, the fourth field of a four-field row
array that was never filled in. So the message was there, said nothing useful,
and is drawn INSIDE the png, which is why every text check in this suite is
blind to it and why the original measurement, counting item map boxes in the
HTML, reported silence.
That is why the assertion is a color check, the second one here after rm36212.
Each row is asked for the color it is drawn in: fixed gives a black item box and
a black pack label, dominant 0,0,0; broken gives a full-width bigWarn bar,
dominant 240,240,180. `is: "0,0,0"` states the item is there and
`not: "240,240,180"` states the warning bar is not. `rows:` cannot express this,
because the broken build draws all eight rows.
The second half of the same bug is on the details page: hgc took the declared
count too and aborted, so an item that did draw could not be clicked through.
That is live on the RR today for hg38 setDups and two hg19 exomeProbesets
subtracks, all `bigBed 4` over a three-field file. The step clicks an item and
asserts the item's POSITION, because the aborted page carries the track's
longLabel twice in its own header and a text: check on that alone passes on it.
It clicks bb9 rather than bb12, since "type bigBed 12" is a prefix of
"type bigBed 12 +".
Do not add a mouseover: step here. Before docent's e2b5b26b925, itemXY handed a
track with no box of its own a neighbour's box, so a tooltip check reported uc.1
for rows that drew nothing.
Measured both ways on 2026-09-09. Against genome-test it fails at the color step
naming all five over-declared rows, each 240,240,180 at 95% of the row; against
the #38310 sandbox all eight are 0,0,0 at 100% and the run exits 0. Then the
whole directory was run against that sandbox twice, once with the patched
hgTracks and hgc and once with unpatched controls built from the same tree:
thirty-seven scripts, identical verdicts, except this one. Notes in
/hive/groups/browser/redmineNotes/38310/claude/.
The fixture is ours, at ~/public_html/docentFixtures/bigBedFieldCount/. One
bigBed with four fields, eight tracks over it, one declared type each, so the
only thing that differs between the rows is the number on the type line.
hubCheck rejects five of the eight, correctly; it is the empty row that is the
bug, not the hub.
README.txt gains the section, and with it two rules that apply to any script
here: `rows:` cannot express "this track drew its items", and a drawn item that
cannot be clicked through is half a bug.
refs #38310, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/README.txt - lines changed 24, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm38310.xfail.docent.yaml - lines changed 103, context: html, text, full: html, text
cac1fca3406536a9f42965bbbebc101b62d8d8e3 Wed Sep 9 18:01:54 2026 -0700
- submoduleSetup: anchor on kent/src so it works from either caller's cwd
Every path in submoduleSetup is relative to kent/src, but its two callers
invoke it from different directories: src/makefile runs
./submodules/submoduleSetup from src, while userApps/fetchKentSource.sh cds
into src/submodules first and runs ./submoduleSetup.
Since the zlib-ng change this broke the userApps source build. From
src/submodules the guard "[ ! -e submodules/zlib-ng/Makefile ]" is true
because that path cannot exist there, so the configure block ran and its
redirect to submodules/zlib-ng-configure.log failed on a missing directory:
./submoduleSetup: line 41: submodules/zlib-ng-configure.log:
No such file or directory
Error: zlib-ng configure failed, see submodules/zlib-ng-configure.log
make: *** [Makefile:28: fetchSource] Error 1
The htslib serial-probe block added for the AVX2 -j race has the same path
problem, but its guard is not negated, so from src/submodules it silently
tested a path that could not exist and never ran at all -- leaving the
userApps build exposed to the very race that block exists to prevent.
Fix both by cd'ing to the script's own parent directory up front, so the
caller's cwd no longer matters. Verified from both call sites: the
userApps path now configures zlib-ng and pre-generates htscodecs.mk with
correct probe flags (HTS_CFLAGS_AVX2 = -mavx2 -mpopcnt), and a re-run from
src is a quiet no-op.
refs #38125
- src/submodules/submoduleSetup - lines changed 8, context: html, text, full: html, text
7eb4d115091bd389d5296d9198dbf0f327b35bdd Thu Sep 10 07:10:50 2026 -0700
- docent: promote rm38310 out of .xfail, the #38310 fix reached genome-test
68f831e1209 landed on master 2026-09-09 and genome-test built it overnight, so
the 2026-09-10 nightly went red the way an xfail is supposed to: the script that
was expected to fail passed. Drop the .xfail from the name and rewrite the header
paragraph that said the fix was not on master yet.
This script has now been watched failing on genome-test and passing on the next
build of that same server, which is stronger evidence than the sandbox A/B it
already carried. refs #38310, refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/regress/rm38310.docent.yaml - lines changed 7, context: html, text, full: html, text
4697bbddd881c72cccb85b9ff0aacd769396b9d0 Thu Sep 10 07:34:35 2026 -0700
- docent: record what evidence each regression test has, and count it
A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.
Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.
tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.
The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.
nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.
Full suite run after the change: 37 scripts, all ok. refs #38252
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/tests/docentTest.mk - lines changed 14, context: html, text, full: html, text
- src/hg/utils/docent/tests/proof.js - lines changed 171, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/README.txt - lines changed 44, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/makefile - lines changed 9, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/nightly.sh - lines changed 49, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm27113.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm35333.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm35580.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm35865.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm35920.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36029.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36061.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36212.docent.yaml - lines changed 4, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36331.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36335.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36340.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36387.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36514.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36540.xfail.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36702.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36798.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36805.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36810.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36836.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36888.xfail.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37326.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37388.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37389.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37489.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37491.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37520.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37553.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37562.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37615.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37785.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37805.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37906.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm38108.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm38126.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm38185.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm38272.docent.yaml - lines changed 3, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm38310.docent.yaml - lines changed 4, context: html, text, full: html, text
055cc335c485aa94f48b440b0105eae2e1ecacb4 Thu Sep 10 09:41:34 2026 -0700
- lib/fa.c: grow the FASTA read buffer to hold the line being added, refs #38320
faMixedSpeedReadNext decided to grow its buffer when bufIx + lineSize no
longer fit, but then asked expandFaFastBuf to reach only lineSize.
expandFaFastBuf stops doubling as soon as it meets that size, so it could
hand back a buffer smaller than bufIx + lineSize. The copy loop wrote the
whole line regardless.
Ask for bufIx + lineSize + 1, which covers the line and the terminating NUL
written after the loop. The other two calls each add one byte to a buffer
that is exactly full, so doubling always covers them and they are unchanged.
Doubling usually leaves enough room, which is why this took so long to show
up. It needs a record whose lines vary a lot in length. faToTwoBit aborted
on such a file with a corrupted heap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
28538f8d9721d6b6503d16b4c6e7db1350cb7c53 Thu Sep 10 09:41:44 2026 -0700
- lib/tests: add faSpeedReadTest, covering the FASTA read buffer growth, refs #38320
Eleven cases, each a set of line lengths. Each one writes a FASTA file,
reads it back with faMixedSpeedReadNext, and checks the name, the size,
every base and the terminating NUL.
The test frees the buffer before each case, so the growth starts from the
same place every time. That is what makes a shape reproduce a given sequence
of buffer sizes, and so what makes these shapes mean anything. Freeing also
means a write past the end of the buffer has to survive a free before the
next case can print.
The shapes are the one from the ticket, a variant whose overrun was small
enough to go unnoticed, the edges of the initial 65536 buffer, the first
doubling, a short line before a long one, a record spanning several
doublings, an evenly wrapped control, a single short line, and a two record
file so the second record is checked for anything the first left behind.
Against the code before the previous commit the test reports a short read
and differing bases on the first case, and exits non zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/tests/expected/faSpeedReadTest - lines changed 24, context: html, text, full: html, text
- src/lib/tests/faSpeedReadTest.c - lines changed 177, context: html, text, full: html, text
ba5aa085f499f17ea3b88621e40a34c9c1ccb3d1 Thu Sep 10 11:03:15 2026 -0700
- hui.c: turn the track color picker on by default, refs #20460
The showColorPicker gate has been in the tree since v496 and defaulted to
FALSE, so the color picker was only visible on hgwdev and genome-test. The
default is now TRUE. The flag stays in place, so a mirror or hgwbeta can
still set showColorPicker=off without a code change.
Also record the new default in the hg.conf catalog.
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 3, context: html, text, full: html, text
c69d3e1d9dedbcf44849aa6b25cf1a785e153bf1 Thu Sep 10 11:13:54 2026 -0700
- hgTracks: turn on chromAlias names by default, refs #29201
The alias icon has been on for the RR since v498 through showAliases=on in
hg.conf on hgwdev, hgwbeta and the RR. The code default was still FALSE, so
mirrors and the GBiB never saw it. Flip the default so they do. A mirror can
still set showAliases=off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgTracks/hgTracks.c - lines changed 1, context: html, text, full: html, text
b809513a3acdf9d24d2f3b70cec639b7a33acb20 Thu Sep 10 11:14:30 2026 -0700
- hgConfCatalog: drop the gcOnTheFlyCoExist row, its code is gone, refs #38208
Hiram removed the C code that read gcOnTheFlyCoExist, but the catalog row
survived it and still cited hg/hgTracks/hgTracks.c. Since gate_lifecycle()
walks the catalog rather than the tree, the flag aged past the QA grace
window and the v503 sunset report flagged it as "stalled in QA -- either
turn these on or delete the feature", for a feature that had already been
deleted. --reconcile did see the row had no reader left, but files that
under question 1, which prints only with --verbose.
Removing the row drops it from the sunset report and from the reconcile
drift list, so it never reaches hgConfGateBacklog.txt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 5, context: html, text, full: html, text
e357453b3740779250e8cf80c2f5104eb77c824b Thu Sep 10 11:23:20 2026 -0700
- hgConfCatalog: report the hg.conf lines a flipped default made pointless, refs #37925
Flipping a gate's default to TRUE does not turn a feature on anywhere: it was
already on wherever somebody had appended the flag by hand while the feature
sat in QA. What the flip does is make those lines pointless, and while one is
there the flag reads on locally whatever the tree says, which is how a wrong
default survives on the machine most likely to catch it.
--redundant reads the hg.conf files readable from hgwdev, joins them against
the catalog and names the lines to delete. It keeps three populations apart:
a line that turns a shipped gate on is litter, a line that turns one off is a
live decision and has to be left alone, and a value that is not one of the six
words hgConfig.c accepts aborts every CGI that reads it. Silent and exit 0
when there is nothing to delete, the same shape as --reconcile, so
nightlyRegister.sh runs it as a third pass and the reminder rides the mail that
already goes out.
Only gates are considered. A knob is a switch a machine is entitled to set
forever, so reporting one would be the crying wolf the gate/knob split exists
to prevent.
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 220, context: html, text, full: html, text
- src/hg/utils/hgConfCatalog/nightlyRegister.sh - lines changed 34, context: html, text, full: html, text
bc4639b87acf68232656130b2972293975e11933 Thu Sep 10 12:43:45 2026 -0700
- genark: tolerate a missing or stale genarkOrg table, refs #38327
genarkGetOrgHash() aborted when the central database had no genarkOrg
table. A mirror's hgcentral has never had one: buildHgCentralSql.csh did
not list the table, so hgcentral.sql on hgdownload carries neither its rows
nor its schema. Add an sqlTableExists check, and add genarkOrg to the list
of tables that hgcentral.sql replaces entirely.
genarkMakeDbDb() defaulted genome to "Other" for an accession with no
genarkOrg row, but left organism NULL. hgConvert prints organism, so the
Convert page read "Genome: (null)". Default both. The copy of genarkOrg
on the RR is 20,754 rows behind hgwdev, so this is visible on
genome.ucsc.edu today for 23 of the 855 GenArk assemblies that appear in
liftOverChain.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/utils/qa/weeklybld/buildHgCentralSql.csh - lines changed 1, context: html, text, full: html, text
f219460db8db952415b5201ba01df99ce4999004 Thu Sep 10 13:46:42 2026 -0700
- genark: pass the liftOver accession list as an slName list, refs #38328
genarkLiftOverDbs() took a pre-quoted SQL fragment that its callers
assembled. It now takes a struct slName list and builds the query
itself with sqlDyStringCreate, so no caller writes SQL text.
Accessions that do not start with GC are skipped, since nothing else
can match the table. hdb.c and hgConvert.c updated for the new
signature.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgConvert/hgConvert.c - lines changed 5, context: html, text, full: html, text
a0adfc55d38c2fbdf34f3f2667052dbdb64db576 Thu Sep 10 14:39:03 2026 -0700
- docent: montage label gutter now fits the widest label
`montage:` reserved a fixed gutter of 1.5 em for the panel label. That fits the
one-letter auto labels (A, B, C) and nothing else. A word label -- "virtChrom",
"quickLift" -- overflowed the gutter and painted over the left edge of its own
panel. The panels were all placed at the same x, but the figure read as if they
were misaligned, because the text ran into the first column of pixels of one
panel and not the other.
The gutter is now measured. Each label is laid out at width 0, its scrollWidth
is read, and the gutter is the widest of those plus a 0.4 em pad, with the old
1.5 em kept as the floor so short labels still line up the way they did.
white-space:nowrap stops a two-word label from wrapping into a taller row and
shifting its panel down.
refs #37892
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/docent/docent.js - lines changed 11, context: html, text, full: html, text
bd4501775536afbc74d45e70960872fd6e61d09c Fri Sep 11 08:50:38 2026 -0700
- hgTracks: a quickLifted container no longer hides the tracks inside it
trackDbString writes "superTrack on show" for a container it is lifting, then
dumpTdbAndChildren walks the settings hash and writes the source assembly's own
superTrack setting after it. That setting reads "on hide" whenever the user has
not opened the container, and the later line is the one the hub reader keeps.
The container came across hidden and so did every track in it, so a lift of a
track that lives in a container produced a page with nothing on it but the
chain track.
Drop the stale setting before the walk, the way walkTree already drops the copy
the children inherit.
Lifting JARVIS from hg19 to hg38 used to draw only the quickLift chain track.
It now draws all ten tracks in the Constraint scores container. A top level
bigWig, a composite of bigWigs and a bigBed all lift as they did before, and
the hub file now carries one superTrack line rather than two that disagree.
The crash this ticket was opened for is a separate thing and was already fixed
by 07ba37612b1 and 5dd36916814. Rechecked on genome-test over eight windows
down to 200 bases: no crash, and the lifted bigWig draws.
refs #37969
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
45ebd62062de3b3a34cf306c89701089664bc36b Fri Sep 11 08:53:21 2026 -0700
- hgConfCatalog: catalog showAliases as defaulting on, refs #37925
c69d3e1d9d flipped the showAliases code default to TRUE in hgTracks.c but
left the catalog entry at default="FALSE". Two reports read that field and
both were wrong because of it.
--redundant skips any gate whose catalog default is not "TRUE", so it walked
past the showAliases=on line at /usr/local/apache/cgi-bin/hg.conf:541 -- the
exact kind of line it was added to find. With this it reports 3 redundant
lines instead of 2.
--sunset is worse than quiet. gate_lifecycle() sets shipped from the same
field, and the firstTrue age comes from git history, so it sees the flip. A
flip date next to a FALSE default is read there as a flip that was reverted,
which is the opposite of what happened.
Same pairing ba5aa085f49 did for showColorPicker ten minutes after the flip.
Found in the 2026-09-11 daily code review.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 3, context: html, text, full: html, text
c2a6ef817930149ad6f55818fa0196d99c47e02b Fri Sep 11 10:40:11 2026 -0700
- cheapcgi: skip a CGI pair with no =value instead of aborting, refs #38335
Both query string parsers looked for the '=' across the whole rest of the
string rather than inside the current pair. A pair with no '=' in it
therefore ran into the pair after it and took its value. "g-catV2&db=hg38"
was stored as one variable named "g-catV2&db", so db was lost with no
warning, and that corrupt name was copied on into the cart. The same pair
at the end of the string had no '=' left to find and aborted the whole
request, which is what the "Mangled CGI input string g-catV2" entries in the
hgw1 logs were.
Both parsers now find the end of the pair first, keeping the existing
separator precedence ('&', then ';' for DAS), and skip a pair with no '='.
A mixed "a=1;b=2&c=3" still parses the way it did.
Adds lib/tests/cgiParseTest, which runs 18 query strings through both
parsers. It covers the empty pair of #38185 as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/lib/tests/cgiParseTest.c - lines changed 95, context: html, text, full: html, text
- src/lib/tests/expected/cgiParseTest - lines changed 72, context: html, text, full: html, text
3eedbb7636648978b0371f2c54378e75fd1ac78c Fri Sep 11 12:20:14 2026 -0700
- Skip a CGI or cookie pair with no =value in three more parsers, refs #38340
The loop that #38335 fixed in the query string parsers is copied in three more
places, and each one still looks for the '=' across the whole rest of the string
instead of inside the pair it is reading. A pair with no value therefore runs
into the pair after it and takes its value, and the same pair at the end of the
string has no '=' left to find and aborts.
lib/cheapcgi.c parseCookies one bad cookie aborts every CGI for that
browser, on every request, until the
reader clears the cookie by hand
hg/hgSession/backup.c a session backup silently leaves out a
custom track
hg/utils/refreshNamedSessionCustomTracks
one bad session aborts the child, the
parent exits non-zero, and every session
after it goes unscanned, so the trash
cleaner removes their custom track files
All three are behind the hg.conf flag skipMalformedCgiPairs, off by default, and
registered as a release gate in hgConfCatalog. The kent libraries do not read
hg.conf, so hgConfig.c hands the setting to cheapcgi the way cfgSetLogCgiVars
already hands it cgiSetMaxLogLen. The query string parsers do not read the
flag; they were fixed unconditionally under #38335.
refreshNamedSessionCustomTracks rebuilds the session contents as it walks, so it
copies a malformed pair through untouched rather than stepping over it. A
session carrying one comes back byte for byte the same.
Adds lib/tests/cgiCookieTest and hg/hgSession/tests/backupParseTest. Both read
every case with the flag off and on, so they pin the old behavior as well as the
new one. The nightly tool has no seam for a unit test; its loop sits inside a
function that runs its own query.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgSession/backup.c - lines changed 28, context: html, text, full: html, text
- src/hg/hgSession/tests/backupParseTest.c - lines changed 77, context: html, text, full: html, text
- src/hg/hgSession/tests/expected/backupParseTest - lines changed 40, context: html, text, full: html, text
- src/hg/hgSession/tests/makefile - lines changed 31, context: html, text, full: html, text
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 9, context: html, text, full: html, text
- src/hg/utils/refreshNamedSessionCustomTracks/refreshNamedSessionCustomTracks.c - lines changed 38, context: html, text, full: html, text
- src/lib/tests/cgiCookieTest.c - lines changed 72, context: html, text, full: html, text
- src/lib/tests/expected/cgiCookieTest - lines changed 44, context: html, text, full: html, text
05faf6b30a29e2260029a5d385a992c3481bcae1 Fri Sep 11 12:54:27 2026 -0700
- hgSession: run the tests directory, refs #38340
hg/makefile's %.testAll rule runs "make test" in the app directory once that app
has a tests/makefile, and swallows the result with "|| true". hgSession had no
test target, so the backupParseTest added with the #38340 fix errored out and
was silently skipped. It passed only when run by hand.
Same three lines hubApi uses.
Note for anyone adding a tests directory under hg/: seventeen other apps there
have one and no test target, so whatever is in them is not running either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgSession/makefile - lines changed 2, context: html, text, full: html, text
964cacf58dec20af72e4c5ed64a33eaf8596c96f Fri Sep 11 15:22:00 2026 -0700
- trackDb docs: add the missing bigNet_intro blurb, refs #20824
The bigNet section in trackDbHub.v3.html declares a DIV with ID
bigNet_intro, but the library had no blurb with that class. The page
logged "Missing document blurb for ID: bigNet_intro" and the bigNet
section rendered with no introduction.
Added the blurb next to bigChain_intro, worded like the other format
intros and pointing at the bigNet help page.
Every other *_intro ID in trackDbHub.v3.html and trackDbDoc.html already
has a blurb; bigNet was the only one missing.
- src/hg/htdocs/goldenPath/help/trackDb/trackDbLibrary.shtml - lines changed 8, context: html, text, full: html, text
874a1eb980511c7e2273d8a6e4d29f00b81eae64 Fri Sep 11 20:22:55 2026 -0700
- quickLift help: list the formats that lift instead of the ones that do not, refs #35536
The "Unsupported Track Formats" section was a denylist. It went stale as
soon as a format was added, and it was already wrong: it listed PSL and
bigChain as unsupported when both lift.
Replaced it with a "Supported Track Formats" allowlist, which matches how
the code decides. A short paragraph after the list names the common
formats that do not lift and repeats the message the Convert page shows.
The WIG to bigWig conversion note moved out of the list into its own
paragraph.
Kept an unsupportedTypes anchor on the section so old links still land.
- src/hg/htdocs/goldenPath/help/quickLift.html - lines changed 39, context: html, text, full: html, text
97557b2dab79bd19b0345f1ea49a233c191300ae Sat Sep 12 06:59:07 2026 -0700
- hgConfCatalog: cite skipMalformedCgiPairs where the tree actually reads it, refs #37925
The row cited lib/cheapcgi.c, which is where the flag takes effect but not
where it is read: the kent libraries cannot read hg.conf at all. The read is
hg/lib/hgConfig.c:227, which hands the setting to cheapcgi through
cgiSkipMalformedPairs. The nightly reconcile had been reporting the row as
citing a file the read has left.
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 1, context: html, text, full: html, text
1270c4fd6763378548918470aa2a4b799c7c930d Sat Sep 12 06:59:17 2026 -0700
- urlCommandCatalog: catalog hgc's aliTrack parameter, refs #37923
aliTrackParam() puts &aliTrack=<track> on the alignment links and hgc reads it
back with cartUsualString to find the track it was called on. aliTable cannot
serve that purpose: it is the table name from the assembly the alignments came
from, and that name usually exists on the assembly being viewed as well, so it
cannot tell a quickLifted alignment track from a native one. Added to hgc
under #38249; it is in hgc's excludeVars, so it does not reach a saved session.
- src/hg/utils/urlCommandCatalog/urlCommandCatalog.py - lines changed 10, context: html, text, full: html, text
91259a696e598fb4b6807cda9b40eb72971e9209 Sat Sep 12 06:59:18 2026 -0700
- cartTrackVarCatalog: catalog the faceted composite's groupBy, refs #37838
<track>.groupBy says which of a faceted composite's two dimensions is kept
together in the image: "sample" puts one sample's data types side by side,
"dataType" puts the same data type for every sample side by side. Written by
cartDump from facetedComposite.js and read back in hgTrackUi, both of which
store it only if it is one of those two words, so nothing unvalidated reaches
the <script> block. trackDb supplies the starting choice with defaultGroupBy
and the cart value wins over it. Added under #36210.
- src/hg/utils/cartTrackVarCatalog/cartTrackVarCatalog.py - lines changed 14, context: html, text, full: html, text
b066328906fe2c8fbbfada16f18e8cc2fb9dc1cf Sat Sep 12 09:58:46 2026 -0700
- sessionCartAudit: match a catalog row that names a whole cart variable, refs #37979
peel() only ever considered suffixes that begin after a separator, so a catalog
row anchored on a fixed prefix could never match the variable it describes: by
the time the walk reached a separator, the hgta_ that anchors
hgta_fs.check.<db>.<table>.<field> was gone. A plain literal row was hit just
as hard, since dbRIP.genoRegion could only be tested as its own tail,
"genoRegion". 4,299 hgta_ names were reported as covered by nothing but a
catch-all because of it.
The #37838 catalog already says which kind of row it is and the audit was
throwing that away. A row whose separator is "." or "_" names a suffix that
follows a track name; anything else names the whole cart variable, with the
separator in front - "" for hgTables and the old per-dataset variables,
cgs_<track>_ for chromGraph. trackVarNames() now returns the separator with
the name, the whole-variable rows are compiled apart, and peel() tries the
whole name before walking suffixes, so the longer and correct match wins.
Two shorthands the catalog already uses are read rather than expanded by hand:
a comma list is several variables sharing one description, and a trailing * is
a family. A row that is prose rather than one pattern can be matched by
nothing, so --check names it instead of letting it count for nothing; there is
one today. The bare wildcard lists are sorted, because they were built by
walking a set and two runs of a published report diffed for no reason.
Over 6,631 saved sessions this moves 2,016 names: 1,990 out of the catch-all
bucket, 25 out of the bare-track-name bucket (filter text boxes whose stored
value is empty, which the visibility heuristic had been reading as track
names), and tfbsConsSitesCutoff out of unknown. Nothing leaves the catalogued
buckets. 1,208 hgta_fs.check names now match the row that describes them
rather than <wigTrack>.<wigVar> or <filterName>Type.
- src/hg/utils/sessionCartAudit/sessionCartAudit.py - lines changed 113, context: html, text, full: html, text
a3df9f62a5995302b5a07ce5e3dd0eadda3a7768 Sat Sep 12 09:59:01 2026 -0700
- cartTrackVarCatalog: describe hgTables' own cart variables, refs #37979
With peel() fixed, 2,331 hgta_ names in the saved sessions were left honestly
uncatalogued rather than absorbed by a catch-all. They fall into four groups,
each read at its call site before a row was written for it.
The two linked-table checkboxes, hgta_fs.linked.<db>.<table> and
hgta_fil.linked.<db>.<table>, which offer a joinable table's fields on the
Select Fields and filter pages. extraTableList finds the checked tables by
scanning the cart for the prefix, so the set is whatever the cart holds.
The filter ops. The catalog covered .pat alone; hgTables.h defines six, and
all six are in live sessions. pat, dd and cmp belong to one field, while
rawLogic, rawQuery and maxOutput apply to the whole table and still carry a
field slot in the name, filled with an empty string or a bare _.
Fifty-one session-scoped variables: intersection, correlation, subtrack merge,
identifiers, user regions, output naming, MAF output, and which table the
Select Fields, filter and histogram pages are about. The header itself
documents the convention that shapes half of them - the pages with a Cancel
button hold their state twice, hgta_<var> in force and hgta_next<Var> proposed,
copied one way on open and the other on Submit - so that is in the group's
description rather than in every note.
Renaming the .pat row to .<op> also removed an accidental cover: a row
registers its trailing component, and that "pat" had been standing in for
gvfTrack.c's %s_pat, which is an item label rather than a cart variable. Its
seven siblings were already in the baseline, so pat joins them there.
hgta_ names matched only by a catch-all go from 4,299 to 3. The three left are
hgta_identifierFile and hgta_userRegionsFile, described in the #37623
file-variable registry that this audit does not read, and hgta_userRegionsTable,
which nothing in the tree reads at all.
- src/hg/utils/cartTrackVarCatalog/cartTrackVarCatalog.py - lines changed 117, context: html, text, full: html, text
- src/hg/utils/cartTrackVarCatalog/cartVarsNotCataloged.txt - lines changed 1, context: html, text, full: html, text
fe79fa58f3040e69e5ac3fa38037674b007d2e61 Sat Sep 12 13:47:50 2026 -0700
- docent: ten regression tests for multi-region view, refs #38252
One script per Closed multi-region ticket, asserting the behavior the ticket
says is correct, on genome-test. Before these the only script here that
entered multi-region at all was rm35580, which uses singleAltHaplo to reach a
different bug.
Between them they cover the four modes, the dialog, the custom-region BED
reader, hideEmptySubtracks across windows, and highlights in both directions
across the mode change:
rm22144 the alt-haplotype input is in the dialog, and hgSuggest
type=altOrPatch resolves the ticket's own mhc, apd and NT_187643
rm23922 the Multi-region and Reverse buttons carry class='pressed' while
their mode is on, and lose it on exit
rm26772 a zero-length BED line names itself instead of aborting with
"Window out of range"
rm27855 hg19 GTEx Gene in singleAltHaplo on chr6_cox_hap2 renders and
clicks through instead of freezing
rm29452 the dialog's exit radio is enabled and checked from a normal view,
and selectable from inside exon view (#34776's half, the missing
hgTracks.virtModeType in the dialog's JSON)
rm29787 custom regions in UCSC chrom names work on hs1
rm30833 a highlight survives turning multi-region on
rm34250 a highlight made in multi-region survives exiting, back in chr1
coordinates
rm35472 hideEmptySubtracks over two windows keeps both subtracks that have
items in one of them, and still hides the one with items in neither
rm37175 exon view keeps the last searched transcript
All ten are assertion-only: every fix shipped long ago. make test is 47 of 47
green, 7m34s.
README.txt gains the two things they cost a red run each. Never assert on a
title attribute: hgTracks' tooltip code moves it into data-tooltip once the
page's JavaScript has run. And multi-region is fully reachable from a goto:
URL -- virtModeType, multiRegionsBedInput, singleAltHaploId, virtWinFull,
<composite>.hideEmptySubtracks -- but the dialog is not, because what it
decides is in JavaScript.
Three candidates were rejected and should not be picked again: #32544 was
closed by deferring to #37256 and is not fixed, #27891 needs a track with a
trackDb multiRegionsBedUrl and /gbdb/hg38/covidMuts/covidMuts.regions.bed does
not exist, and #24055's gesture is an ajax visibility change that no Docent
verb makes.
- src/hg/utils/docent/tests/regress/README.txt - lines changed 32, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm22144.docent.yaml - lines changed 56, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm23922.docent.yaml - lines changed 62, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm26772.docent.yaml - lines changed 48, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm27855.docent.yaml - lines changed 47, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm29452.docent.yaml - lines changed 83, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm29787.docent.yaml - lines changed 46, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm30833.docent.yaml - lines changed 52, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm34250.docent.yaml - lines changed 60, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm35472.docent.yaml - lines changed 57, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37175.docent.yaml - lines changed 55, context: html, text, full: html, text
7aba31f14aed7f2620e4746b54f9569a5c93e1ad Sat Sep 12 15:07:34 2026 -0700
- docent: ten regression tests for quickLift, on hgTracks and on hgc, refs #38252
Fourteen scripts here already lift something -- they are the ones that call
`convert: {quicklift: true}` -- so these take the parts of the lift that had no
test. Five read the lifted image and five read a details page:
rm38032 the target keeps the source's track order. First use of `ordered:`,
which was added to expect: for this bug
rm38042 a ClinVar CNV running past the chains quickLift loads is clipped
rather than dropped, so the spanned-item merge still has it
rm37646 a lolly composite subtrack lifts, and its map boxes still carry its
own track name -- the string the stale pop pointer clobbered
rm36048 the spanned-item merge still works on a lifted DECIPHER track
rm37815 "Hide all default tracks on the target" hides all six of hs1's own
tracks and keeps the lifted one
rm36059 a lifted GENCODE Versions item gives the real details page, in
destination coordinates, with no "Can't start query"
rm36370 a lifted knownGene click renders GeneReviews and Methods, the two
sections the ticket says were missing
rm36125 a lifted RefSeq item's page, and its Predicted Protein link
returning SHH's peptide instead of a blank page
rm36942 the Alignment Differences description, reached from a difference
item: the four colors and the figure
rm38146 the same page with a GenArk assembly as the SOURCE, down to the base
alignment that reads query bases out of a two bit file
All ten are assertion-only: every fix shipped long ago. make test is 57 of 57
green in 10m18s, up from 7m34s -- each script costs a convert, about 17 seconds,
because no URL builds a quickLift hub.
README.txt gains what the batch cost. A lifted row and map box carry a per-run
hub_<n>_ prefix, so rows: matches by suffix and a has: selector must use a
substring. Never assert a count an otto reload can move: rm38042 and rm36048
both read the merged-item box and leave its count (45 for ClinVar today) to a
comment. And a details page prints the track's own labels whether or not it
worked, so each hgc assertion names something only the fixed page has.
Three candidates were rejected: #38033's "(N items could not be lifted)" label
is only in the drawn image and the page JSON, where no expect: check reaches it;
#37970 needs a broadPeak track and hg38 has none; #37974's center-label drag is
pixels.
- src/hg/utils/docent/tests/regress/README.txt - lines changed 34, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36048.docent.yaml - lines changed 42, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36059.docent.yaml - lines changed 53, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36125.docent.yaml - lines changed 72, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36370.docent.yaml - lines changed 41, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36942.docent.yaml - lines changed 46, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37646.docent.yaml - lines changed 49, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37815.docent.yaml - lines changed 43, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm38032.docent.yaml - lines changed 53, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm38042.docent.yaml - lines changed 53, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm38146.docent.yaml - lines changed 50, context: html, text, full: html, text
31dd19d6395d10ee6632bb2afcf1947f33eff3dd Sat Sep 12 16:03:40 2026 -0700
- docent: ten regression tests for hgTrackUi, refs #38252
Four scripts here already touched hgTrackUi in passing (rm37389, rm37489,
rm38126, rm38272). These are about the page itself: the superTrack
configuration page, composite and subtrack configuration, filters, the color
override, the parent link, and two bad-input paths.
rm20460 the color override is offered on a genePred track and not on a
chain track, which is the type restriction 6d78a8e2d72 added
rm32263 a composite child's page names its container, and the link reaches
the container's own configuration page
rm34651 the density-graph options div is densGraphOptions<track> and the
old shared id is gone
rm35906 Clear filters survives a submit: the cart comes back on All, which
is what the button only appeared to do before
rm36484 the filterComposite select has no stray <br> inside it
rm36668 both the rearrangement and density-graph checkboxes on, submitted
together, does not crash hgTracks
rm36917 the superTrack page's Hide all / Show all / Apply to all controls,
and Show all leaving no child hidden
rm37130 under noParentConfig both filters are on the jaspar child page and
neither is on the parent
rm37282 Hide all greys the superTrack's own dropdown, and a child's Hide
does not force the container back to show
rm37743 a dup_1_refGene request with nothing in the cart errAborts with a
message naming it, instead of taking a SIGSEGV
All ten are assertion-only. make test is 67 of 67 green: the suite went from
57 scripts to 67 for 14 extra seconds, 10m32s in total, because hgTrackUi draws
no image and most of these never leave it.
README.txt gains the four things the batch settled. No track image means rows:
is unavailable and a positive text: is mandatory, since a crash hands the
browser an empty document where every noText: passes. Most hgTrackUi bugs ARE
the markup, so naming the id or class the commit changed is the right check
here, unlike on hgTracks. A cart round trip is the only way to tell a control
that works from one that looks right. And Docent cannot pick an option from a
select, so a visibility goes in through the URL and a button is clicked wherever
one exists.
Two candidates were rejected: #38192's missing-track message is live but the
ticket is still Reviewing, and #37840's "add db= to links" is not true of every
link on the page today, so a blanket assertion would fail for a reason that is
not a bug.
- src/hg/utils/docent/tests/regress/README.txt - lines changed 33, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm20460.docent.yaml - lines changed 43, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm32263.docent.yaml - lines changed 36, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm34651.docent.yaml - lines changed 39, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm35906.docent.yaml - lines changed 44, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36484.docent.yaml - lines changed 32, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36668.docent.yaml - lines changed 58, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm36917.docent.yaml - lines changed 48, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37130.docent.yaml - lines changed 45, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37282.docent.yaml - lines changed 48, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/rm37743.docent.yaml - lines changed 36, context: html, text, full: html, text
4a5041811d79f95f11000fbc3fc310f9a4ae417a Sun Sep 13 15:36:26 2026 -0700
- docent: a test run clears up after itself, refs #38252
A run left a log per script and a stills/ directory per script, and nothing
ever removed them, so `git status` in the test directories reported 89 files
that were not work. The obvious answer is a .gitignore, and it is the wrong
one: the files stay on disk, and git is taught to look away from the directory
new tests are written in.
`make test` now removes a passing script's log, stills and sessions. Nothing
reads any of it once the run is over -- nightly.sh reads this target's OUTPUT,
and the failure branch prints a failing log into that output while the file is
still there. A failing script keeps its log, and so does an xfail that passed,
which is the flip `make proof` is about and the one morning someone will want to
read the whole run. `parity` does the same with its three logs and the mp4 its
slow run records.
WARNING lines are echoed before the log goes. make test sends each script's
output to its log and prints only "ok", so a docent warning on a PASSING script
reached a file nobody opens -- and "tooltip never showed its own text" means the
step measured nothing. None of the 67 scripts warns today; the point is that
one that starts to will say so.
Five orphan logs are removed by hand in passing: rm36212.xfail.log,
rm36540.log, rm37388ui.xfail.log, rm37389.xfail.log and rm38272.xfail.log, left
behind when those scripts were renamed. make test deliberately does not sweep
logs it did not just write, since a rename in progress should not lose its
evidence; make clean is still there for a full wipe.
Measured: 67 of 67 green in 10m39s and the directory came back with nothing
untracked in it, the fifteen language tests next door pass and clear their
sessions/ too, and a script made to fail on purpose kept its log and its stills.
- src/hg/utils/docent/tests/docentTest.mk - lines changed 32, context: html, text, full: html, text
- src/hg/utils/docent/tests/regress/makefile - lines changed 5, context: html, text, full: html, text
386bd6292e071d16bc62d6135cfe503e21c3a462 Mon Sep 14 10:14:43 2026 -0700
- v504 preview2 (automated)
- src/utils/qa/weeklybld/buildEnv.csh - lines changed 2, context: html, text, full: html, text
switch to files view, user index