Commits for braney
switch to files view, user index
v502_base to v503_preview (2026-08-10 to 2026-08-17) v503
Show details
bd85159b36db7ff8c02b32eb18d96ed6066feb8f Wed Aug 5 13:59:46 2026 -0700
- hgTracks: measure text with the same font engine that will draw it, refs #38027
Pack mode works out how many rows a track needs by measuring its item labels,
and mgFontStringWidth answers from whichever text engine is loaded at the time.
FreeType was only loaded when an image was created, so the measuring could
happen on the bitmap engine while the drawing happened on FreeType -- and which
one you got depended on whether the ideogram image had been built first.
Load the engine in initTl(), alongside the rest of the font setup, before
anything measures a string. maybeNewFonts() and the new initFontEngine() share
chosenFreeTypeFont(), and mgLoadFontEngine() loads an engine without attaching
it to an image. ftInitialize() now keeps the face it already has when asked for
the same font file, so the glyph cache hanging off that face survives the extra
call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- src/hg/hgTracks/config.c - lines changed 37, context: html, text, full: html, text
- src/hg/hgTracks/hgTracks.h - lines changed 3, context: html, text, full: html, text
- src/hg/hgTracks/simpleTracks.c - lines changed 4, context: html, text, full: html, text
75b9f337975beb5a9eaec5ff00e48cb878591fd2 Fri Aug 7 12:07:35 2026 -0700
- bigLolly: reserve top margin from the largest lolly actually drawn, refs #38031
A track that sets lollySizeField scales each radius by the track height, so a
lolly can end up bigger than lollyMaxSize. The margins were reserved from
lollyMaxSize alone, so once the drawn radius passed it the top row of circles
ran off the top of the row and came out flat. Take the larger of the two, which
leaves every current picture unchanged since today's radii are all below
lollyMaxSize. The same margin positions the y-axis gridlines and the upper and
lower value labels, so all three move together and the axis still lines up with
the circles.
- src/hg/hgTracks/lollyTrack.c - lines changed 20, context: html, text, full: html, text
cdb3401bc60e54dae514885304914b5cd70c45ae Fri Aug 7 12:07:40 2026 -0700
- quickLift: keep the source track order on the target, refs #38032
The hub writer replaced each track's priority with a counter handed out in walk
order. The hub file is appended to across requests, and new tracks started
after the highest priority already in the file, so a track added in a later
request always sorted below one added earlier no matter where it sat on the
source. Carry the source priority across instead. The group setting already
survives into the hub, so the target orders within a group on the same key the
source does. The append-after-existing bookkeeping is no longer needed.
b53f0365865b5449ef209ea86c5220dd07dc8b28 Fri Aug 7 12:07:45 2026 -0700
- quickLift: don't report items that failed to lift as filtered out, refs #38033
Items that could not be mapped through the chain were counted by the same
variable the track filters use, so a lifted view claimed a number of items had
been filtered out when no filter had removed anything. Count the two cases
apart: failing a filter is a filter rejection, while passing the filters and
then getting nothing back from the lift is a lift failure, and say so in the
label.
- src/hg/hgTracks/bigBedTrack.c - lines changed 12, context: html, text, full: html, text
- src/hg/lib/trackDbCustom.c - lines changed 8, context: html, text, full: html, text
7f65d3da8f2138d847f15a8398f8dcee1a079027 Fri Aug 7 12:07:59 2026 -0700
- quickLift: clip an oversized item to the chains we loaded instead of dropping it, refs #38042
quickLift loads chains for the window plus padding, capped at 1 Mb. An item that
reaches further than that has ends where no chain reaches, remapRangeList can
place neither of them, and the whole item is dropped even though the part on
screen maps fine. ClinVar copy number variants run to 159 Mb, so 44 of the 46 in
one window disappeared.
Pull the ends in to what the chains cover before mapping. The browser never
draws the ends of an item that spans the window, so their exact position does not
matter, and the existing spanned-item merge still reports it. A clipped end is
snapped to a base inside an aligned block, since remapRangeList will only place a
coordinate that lands on real alignment.
The clip is in the quickLift code, not the shared liftOver remap path. Only
hgTracks asks for it, so the details page keeps the item's true extent. Gated by
quickLiftClipToChains, on by default.
- src/hg/hgTracks/bigBedTrack.c - lines changed 1, context: html, text, full: html, text
- src/hg/lib/quickLift.c - lines changed 104, context: html, text, full: html, text
c2f76c5303384cddb6b48852c54c30437ee029f2 Sat Aug 8 08:49:33 2026 -0700
- quickLift: order the target by rank in the source list, refs #38032
cdb3401bc60 carried each track's source priority into its hub stanza, on the
assumption that the source group survives the lift and orders the target the
same way it orders the source. It does not. trackHubAddGroupName rewrites the
group of every hub track to the hub's own name, so all lifted tracks land in one
QuickLift group and priorities from different source groups end up compared
against each other. MANE is priority 100 in the genes group; wgEncodeRegDnaseWig
is 1.8 in regulation. The group puts MANE on top on hg38; the raw priority put
it at the bottom on the target.
Write the track's rank in the list the caller has already sorted on group
priority and then track priority. The rank counts every track walked past, not
only the ones written out, so it depends only on how the source is laid out and
does not shift with the append-per-request behaviour this ticket is about. A
superTrack parent is not in that list and has no rank of its own, so it takes
the rank of the first child that pulled it in, less a half.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
efc3f584c95755330510d53c767f103d222e20d1 Sat Aug 8 13:21:26 2026 -0700
- cartTrackVarCatalog: fix the autoScale values, check value lists, refs #37838
The autoScale entry had the wrong vocabulary. The cart holds one of three
long literals from wiggleScaleOptionsParent (hui.c:1973). The catalog listed
on/off/group, which are the trackDb spellings; wiggleCart.c:454-459 translates
those into the literals and never stores them. It also listed "cumulative",
which is the C enum identifier wiggleScaleCumulative (hui.h:576), not a value
in either vocabulary.
This is worth more than a documentation fix. wiggleScaleStringToEnum()
errAborts on an unknown string, so autoScale=on in a cart crashes hgTracks
instead of being ignored. Any writer generated from this file would have
produced exactly that.
There were two copies of the bug. The lolly type carried its own autoScale
entry with on/off, but lollyTrack.c:485 calls the same
wigFetchAutoScaleWithCart(), so it shares the vocabulary and the errAbort.
Those two are the only readers of that function, so autoScale is now complete.
smoothingWindow listed "..." as if the ellipsis were a value. Enumerated.
The other eight wig enums were checked against their C arrays and were all
working: stringArrayIx() compares with differentWord(), which ignores case, so
the on/ON and Bar/bar mismatches did no harm. Their values now match the
arrays exactly, and each entry names the array it was checked against.
Two new fields on an entry, both needed by the JSON cart schema:
aliases other spellings a writer may use, mapped to the cart value, so
JSON can accept autoScale "on" and store the literal
valuesSrc the C array the values were verified against
--check grows check_values(), which fails on a placeholder in a values list,
an alias on an entry with no values, an alias pointing at a non-value, and an
alias that is also a value. It also reports the backlog: 39 enums whose value
lists have not been checked against the array that gates them. Each one can
crash a CGI the same way autoScale could.
- src/hg/utils/cartTrackVarCatalog/cartTrackVarCatalog.py - lines changed 130, context: html, text, full: html, text
241af03b8bb62d23ef45e248347edf47c45f4b08 Mon Aug 10 11:03:15 2026 -0700
- Merge branch 'worktree-rm-38031-38033' into master
quickLift item ordering, oversized-item clipping, filtered-item reporting,
and bigLolly top margin. refs #38031 refs #38032 refs #38033 refs #38042
39cc15a3cd25637b61e80c589a6027952339c6a3 Mon Aug 10 11:38:20 2026 -0700
- Merge branch 'worktree-cartJson-37838' into master
cartTrackVarCatalog: correct the autoScale vocabulary, add aliases and
valuesSrc, and check value lists under --check. refs #37838
cfd5fcabf643de689fd383958047718f67fe6575 Mon Aug 10 11:52:18 2026 -0700
- hgTracks: don't split a squishyPack track when the squished part is off screen, refs #37974
expandSquishyPackTracks decided to clone the track from slCount(squishTrack->items)
alone, without asking whether any of those items are in the window. Items outside
the window are never laid out into a row, so a squink built only from those has no
rows and no height, yet it still claims a center label slice at y == pixHeight and
sliceIsConsistent warns "slice(center) has an invalid X or Y offset". Only split
when at least one item above the point really overlaps the window, the same test
packCountRowsOverflow uses, with hgFind matches skipped since those always stay in
pack.
That makes the later "if the squish track has no items, continue" check unreachable,
so it goes. Removing it fixes a second bug: that path skipped both slReverse()
calls, so a squishyPack track whose items all stayed in pack kept its item list
reversed.
The warning goes to stderr and verbose is hIsPrivateHost(), so only hgwdev ever
showed it. The reversed item list was visible everywhere.
- src/hg/hgTracks/hgTracks.c - lines changed 25, context: html, text, full: html, text
a3cffed60a23b982c6e8b349975ab925768a13dc Mon Aug 10 14:47:59 2026 -0700
- Merge branch 'rm-38027' into master
hgTracks: load the font engine before anything measures a label, so the
ideogram no longer changes how tracks pack. refs #38027
26c8e4e2290ebd613b3aad5d5d2b6a12bdfff251 Mon Aug 10 17:16:54 2026 -0700
- hgTracks: apply the color a drag picks in the highlight picker, refs #37987
The Drag-and-select dialog's buttons read the hex text box beside the swatch, not
the picker itself. Clicking a palette square makes spectrum treat the color as
chosen, which updates that box. Dragging in the gradient square only repaints the
picker: spectrum does not consider the color chosen until "choose" is clicked, so
the box kept its old value and "Save Color" saved the old color.
A move handler now writes the dragged color into the box as the drag happens, and
a hide handler resyncs the box so a cancelled drag does not leave the dragged
color behind.
Also open the picker above the swatch when opening downwards would cover the
dialog's row of buttons, which otherwise cannot be clicked at all.
Reported on the mailing list in #37983.
9a1633df43738cc9bb0b64ad932514ffaae85895 Mon Aug 10 17:17:12 2026 -0700
- hgTracks: start the drag-select color picker on the saved color, refs #37990
Checking "don't show this again" makes the dialog's close handler remove it from
the DOM rather than hide it, so the next drag builds a new one. The rebuilt
picker took the default color and the color the user had saved was gone.
The dialog saves its color with dragSelect.saveHlColor, which writes
hgTracks.prevHlColor. makeHighlightPicker reads a prevHlColor of its own, which
nothing in this path ever sets, so it fell through to the default. Pass the
saved color in as the picker's starting color.
Leave the fourth argument undefined so the label keeps its default: the label
test reads label.length, so passing null there would throw.
Without the checkbox the dialog is only hidden and keeps its value, which is why
this was hard to reproduce from the report.
19ffb1f276b652de2cd1f593c19bd25630ed6b59 Mon Aug 10 17:17:28 2026 -0700
- hgTracks: let "Save Color" honor the "don't show this again" checkbox, refs #38071
Every other button in the Drag-and-select dialog clears
hgTracks.enableHighlightingDialog when the box is checked, and stores it. Save
Color did neither, so the box was ignored: the close handler removed the dialog
as though it had been turned off, and then it came back on the next drag.
saveHlColor only writes prevHlColor, so send the setting alongside it the way
"Zoom In" already does.
3258e4421c0a95ef8ca84445b077c77faaf04970 Mon Aug 10 18:14:37 2026 -0700
- clinvar: give clinvarSubLolly a maxHeightPixels ceiling, refs #38072
A bigLolly row is a fixed pixel height, so neither pix nor textSize reaches it.
A figure rendered at three times the normal width left the ClinVar submitted
interpretations row at 128 pixels while the tracks around it grew with the font,
so it came out a sliver with unreadable y-axis labels. Docent asks for a taller
row for a print render (see 140e29ae689, refs #37892), but each track's own
maxHeightPixels clamps that request, and this track had no setting, so the
ceiling fell back to the 128 default and the request was clamped straight back.
Raise the ceiling to 512. The default stays at 128, so an ordinary view is
unchanged and only a render that asks for more can get it. The two files are
kept in step because the alpha stanza carries its own copy.
- src/hg/makeDb/trackDb/human/clinvar.alpha.ra - lines changed 3, context: html, text, full: html, text
- src/hg/makeDb/trackDb/human/clinvar.ra - lines changed 5, context: html, text, full: html, text
12a0dc62bdfc01eb3f94cc675657ab21b893c67a Tue Aug 11 09:22:06 2026 -0700
- lib: reject the reserved 0x01 escape marker in sqlSafef, htmlSafef and cgiDecode, refs #38051
sqlSafef does not escape values as it formats them. It formats first, then
escapes in a second pass. To mark which spans need escaping it wraps each
quoted %s in an in-band marker byte, escPunc = 0x01. sqlEscapeAllStrings then
pairs up the 0x01 bytes, escapes the text between a pair, and copies text
outside a pair raw.
The marker shares the byte stream with the data. A value carrying its own pair
of 0x01 bytes forges an extra pair and shifts the pairing, so part of the value
is copied raw instead of escaped. For a value of "x\x01' OR '1'='1\x01x" the
query came out as name='x' OR '1'='1x' with live quotes. cgiDecode turns %01
into a literal 0x01 and filtered nothing, so this was reachable from a plain
GET parameter with no login. htmshell.c uses the same trick for htmlSafef, where
the payoff is XSS instead of SQL.
Two guards:
vaSqlSafefNoAbort now rejects a quoted %s value that already contains escPunc.
vaHtmlSafefNoAbort counts the markers after formatting and requires exactly two
per escaped string. Both use errAbort rather than sqlCheckError or the noAbort
return. sqlCheckError honors noSqlInj.level and can be downgraded to warn and
then return, which would leave the injection live, and vaHtmlDyStringPrintf
reads a negative return as "buffer too small" and would retry forever. A raw
0x01 is never legitimate in either place, so there is no false-positive cost.
cgiDecode and cgiDecodeFull now drop 0x01, both percent-encoded and raw, so the
marker never enters a cart or CGI string. Only 0x01. Tab, newline and CR are
left alone because custom-track textarea uploads need them. Multipart uploads do
not pass through cgiDecode, so the sqlSafef and htmlSafef guards are what cover
that path.
Pre-escaped %-s arguments are not checked individually. They come either from
sqlSafef output, which now aborts before it can produce a marker, or from
sqlCheckIdentifier and sqlCkIl, which whitelist from a disallow-all table that
never permits 0x01.
Removing the in-band marker altogether is the durable fix and is tracked
separately, since it rewrites a hot path and needs its own performance testing.
af0aeebb6fbccc3364ce3c85e7052baa2b124f10 Tue Aug 11 09:22:17 2026 -0700
- hg/lib/tests: add a sqlCheck case for the 0x01 escape marker bypass, refs #38051
New XM type wraps the given value in the 0x01 marker that sqlSafef uses
internally to delimit the spans it must escape, then calls sqlSafef. Two markers
forge an extra delimiter pair and used to leave the text between them unescaped,
so this must now abort.
The payload is built inside sqlCheck.c rather than passed in from the makefile,
so the test does not need a control byte in a shell argument.
This case passes whether or not HGDB_CONF is set, because the guard it exercises
uses a hard errAbort rather than sqlCheckError. Note that sqlCheck is not in the
default test target, and that case 3 does need HGDB_CONF set in order to abort.
- src/hg/lib/tests/expected/sqlCheck/sqlCheck.10 - lines changed 1, context: html, text, full: html, text
- src/hg/lib/tests/sqlCheck.c - lines changed 12, context: html, text, full: html, text
- src/hg/lib/tests/sqlCheck.mk - lines changed 2, context: html, text, full: html, text
a8f76e3fe849a2d1cde06b7f19bd9f4d8fbb81fb Tue Aug 11 09:50:19 2026 -0700
- redmineCli: watch the code reviewer on build-patch tickets, refs #37281
build-patch added the QA Team and the build meister as watchers, but not the
person named in Suggested Code Reviewer. That reviewer has to set Code Review
Status to Approved before QA can test, so they are the first person the ticket
waits on, and they were the one person not told it existed.
The reviewer name is now part of the default watcher list. Dedup is by user ID,
so nothing changes when the reviewer is also the build meister, and
--no-default-watchers still suppresses all three.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
44abc107a014bf17a4c5b0d5ede16f8d10ce180d Tue Aug 11 09:58:31 2026 -0700
- hgc: bound the copy in parseSs so an oversized ss or item cannot overflow, refs #38054
parseSs copied its argument into a fixed 1024-byte static buffer with strcpy and
no length check. The argument is the cart variable ss, or the item name, and a
visitor controls both. A legitimate value is a short triple of trash file paths,
so any oversized value is a bug or an attack.
Measured against an unpatched build, driving the item through QUERY_STRING:
1500 and 8000 bytes both wrote past the end of the buffer and the CGI carried on
and exited 0, so the corruption was silent. 100000 bytes gave SIGSEGV. Remote and
unauthenticated in every case.
safecpy does the same copy with a length check and aborts on overflow, which is
consistent with the three errAborts parseSs already has for the other malformed
cases. It also does not echo the value into the error message.
The boundary now behaves as expected. An item of 1023 bytes still parses, 1024
aborts, and a valid three-word value parses exactly as before. All six call sites
pass user data through this one copy, so they are all covered.
41e62d7a41147961651f6d317e3b90b7680d806c Tue Aug 11 10:22:02 2026 -0700
- hgMenubar: only allow incFile to name a file in the include directory, refs #38052
hgMenubar opens the file named by its incFile request parameter and prints it line
by line. The path was built as DOCUMENT_ROOT + incFile with no traversal stripping,
so a request could read any file the server can read, with no login.
Confirmed against an unpatched build. incFile=/../../../../etc/passwd returned the
password file. Pointing it at hg.conf.private returned 216 lines, 19 of which match
a password, secret or key setting, and that file is world readable so the apache
user can read it. This is credential disclosure, not just file disclosure.
incFile now has to start with /inc/ and contain no "..", otherwise it is ignored and
the normal menu bar is served. I kept the parameter rather than removing it because
printMenuBar already treats a non-default value specially, skipping printIncludes,
so an alternate include looks like a designed feature that a mirror could be using.
Nothing in our own tree passes incFile.
A rejected value is logged to stderr. Before this change a traversal attempt failed
with a 500, which was visible in the logs; without the log line the attempt would
now blend in as an ordinary page.
Falling back beats aborting here because this CGI is included into every static page
through SSI, so an abort on a bad parameter would break the page.
- src/hg/hgMenubar/hgMenubar.c - lines changed 13, context: html, text, full: html, text
2e48cc57a9fd1f065c439b14df8ddf805901e391 Tue Aug 11 11:14:20 2026 -0700
- udc: reject a ".." path component when turning a remote URL into a cache path, refs #38056
udc builds its on-disk cache path out of the remote URL. qEscaped deliberately leaves
'.' and '/' unescaped so cached names stay readable, and longDirHash only rewrites
components that are too long for the filesystem, so a ".." in the URL survives all
the way into the cache path. makeDirsOnPath then creates those directories and we
write bitmap and sparseData files into them, outside the cache root, as the web server
user. A custom track or hub bigDataUrl gets here with no login.
Demonstrated against an unpatched build with a local server standing in for a hostile
hub host, since the escape needs a server that answers a request whose path contains
"../". Opening http://host/a/../../../../../ESCAPED/pwn.bb created ESCAPED/pwn.bb five
levels above the cache root and wrote the server's content into sparseData inside it.
That is why the log review found no successful attempt: every one seen in the wild
pointed at a legitimate third-party host, and those normalize the path and refuse. An
attacker pointing at their own host would have succeeded.
The check goes in udcPathAndFileNames rather than in udcParseUrl, which is where the
draft on the ticket suggested it. That function is the one place a URL becomes a
filesystem path, and it covers both callers that build cache paths. udcParseUrl is
also used by udcIsLocal, which is only a predicate and is called from linefile.c,
hdb.c, liftOver.c and elsewhere to decide how to open something; aborting in there
would turn a question into a failure.
Only the ".." component is rejected, so names that merely begin with dots, such as
"..foo" or "a..b", keep working. Cache paths for ordinary URLs are byte for byte what
they were, which matters because a change there would invalidate every cached file on
every node. Verified by diffing the derived paths for a set of URLs across the change:
the only lines that differ are the malicious ones.
Escaping '.' in qEscaped would also have closed this, but it would rewrite every
existing cache path and force a full re-download for no gain over rejecting "..".
a47e3c796d32feca4b3747b6eeb37afb25d31ae1 Thu Aug 13 08:07:47 2026 -0700
- hgConfCatalog: classify the two new hgBlat form flags and quickLiftClipToChains as gates, refs #37925
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 61, context: html, text, full: html, text
8f329215737bf098902eb0d589be5c1b29008aa4 Thu Aug 13 08:18:44 2026 -0700
- hgLogin: treat the login-flow cart variables as server-owned, refs #38037
- src/hg/hgLogin/hgLogin.c - lines changed 24, context: html, text, full: html, text
ece2ff8889e304cf8e873063b331ba710bbc21fc Thu Aug 13 08:18:44 2026 -0700
- urlCommandCatalog: describe hgLogin's login-flow cart variables, refs #37923
- src/hg/utils/urlCommandCatalog/urlCommandCatalog.py - lines changed 20, context: html, text, full: html, text
- src/hg/utils/urlCommandCatalog/urlNamesNotCataloged.txt - lines changed 1, context: html, text, full: html, text
e7b98d8286de2133b30617a7eb9c78cd355fd21e Thu Aug 13 14:32:17 2026 -0700
- geneReviews otto: survive a knownGene rebuild, and keep the bigBed in step with the tables, refs #38098
hg19 and hg18 take their gene coordinates from knownGene. The hg19 knownGene
rebuild on 2026-07-15 moved the ends of most genes by a few bases, and the
validation step compared chrom, chromStart, chromEnd and name exactly. That made
1325 of 1939 rows look new, far past the 10 percent ceiling, so the job has
failed every week since July 21 and the tables have been stuck at their June 30
version.
validateGeneReviews.sh now compares the overall span of each gene on each
chromosome and counts a gene as unchanged when the old and new spans overlap.
geneReviewsDetail still compares whole rows exactly. A zero-overlap result now
fails with a message rather than an awk divide-by-zero.
checkGeneReviews.sh validates all three assemblies before deciding, so one bad
assembly no longer hides the state of the others. hg18 had not been checked
since July.
buildGeneReviews.sh pointed /gbdb at the new bigBed before validation ran, so a
failed run left the browser image and the geneReviews tables six weeks apart.
The relink moves to checkGeneReviews.sh, after validation passes and the tables
are installed.
- src/hg/utils/otto/geneReviews/buildGeneReviews.sh - lines changed 6, context: html, text, full: html, text
- src/hg/utils/otto/geneReviews/checkGeneReviews.sh - lines changed 26, context: html, text, full: html, text
- src/hg/utils/otto/geneReviews/validateGeneReviews.sh - lines changed 36, context: html, text, full: html, text
74f5343d5f22428477778d20a5dc6e9e39c213fd Thu Aug 13 14:36:56 2026 -0700
- geneReviews otto: pin the locale so a hand-run matches the cron run, refs #38098
The NCBI GeneReviews files are Latin-1. GRtitle_shortname_NBKid.txt holds two
titles with high bytes, "Cantu syndrome" and "Stromme Syndrome". Under a UTF-8
locale GNU grep decides the file is binary, drops those two lines and writes a
note to stderr that the wrapper's mail never shows. cron runs with no locale set
and is safe, but a hand-run from a login shell quietly loses two disease titles
and three geneReviewsDetail rows.
Set LC_ALL=C in all three scripts. This also keeps sort and join in agreement in
validateGeneReviews.sh whichever way the job is started.
- src/hg/utils/otto/geneReviews/buildGeneReviews.sh - lines changed 7, context: html, text, full: html, text
- src/hg/utils/otto/geneReviews/checkGeneReviews.sh - lines changed 6, context: html, text, full: html, text
- src/hg/utils/otto/geneReviews/validateGeneReviews.sh - lines changed 4, context: html, text, full: html, text
6cdc3398485dd66c0a16e0b0ccaa840911f87308 Thu Aug 13 14:47:48 2026 -0700
- geneReviews otto: go back to the exact coordinate check, refs #38098
Reverts the span-overlap comparison from e7b98d8286d. A wholesale shift in the
source coordinates, such as the one the July 15 hg19 knownGene rebuild caused,
should stop the job and have someone look at the new data. Loosening the test
would let the next one through unseen. The right response to a change like that
is to review it and install it by hand.
For geneReviews a row is again unchanged only when chrom, chromStart, chromEnd
and name all match. A comment on the test says why it is strict, so the next
person does not read the strictness as an oversight.
Kept from the reverted commit: LC_ALL=C, and a guard so zero rows in common
fails with a message rather than an awk divide-by-zero.
- src/hg/utils/otto/geneReviews/validateGeneReviews.sh - lines changed 30, context: html, text, full: html, text
707cff6538b51939df30f84525ff9295b1b17e10 Thu Aug 13 15:21:23 2026 -0700
- geneReviews otto: say FAILED in the subject when the run fails, refs #38098
A failed run mailed otto-group with the same "GENEREVIEW Build" subject as a
good one, and opened with the same routine hgLoadBed and bedToBigBed output. The
only sign of trouble was two lines at the bottom. Four failures went out that way
between July 21 and August 11 and read like four normal weekly updates.
Run the job to a log first so the exit status can choose the subject. A failure
now arrives as "GENEREVIEW Build FAILED" with the exit code appended to the body,
and the wrapper exits with the job's status rather than mail's.
Two smaller things fall out of the same change. A failure that produced no output
used to hit mail -E and be discarded, so it was silent; that case is now always
mailed. And the run output is kept in lastRun.log for anyone looking into a
failure after the fact.
A week where NCBI posts nothing still produces no output and no mail.
- src/hg/utils/otto/geneReviews/geneReviewsWrapper.sh - lines changed 21, context: html, text, full: html, text
be4f99627b03673652489ddaf9254782ab5c8b3e Fri Aug 14 09:07:33 2026 -0700
- hgc: read chain query sequence from the two bit file when the query assembly has no database, refs #37974
htcChainAli chose the query sequence with the otherDb branch ahead of the
otherTwoBitUrl branch, and otherDb is the second word of the track's type
line, so it is never NULL. The two bit branch has been unreachable since
a1cfd4eb6e2.
That only shows up now because a quickLift chain names its source assembly in
the type line. When the source is a GenArk accession there is no database to
connect to, so clicking an alignment link on the Alignment Differences page
aborted with "Couldn't connect to database 'GCA_002844635.1'". The track
already carries the right file in otherTwoBitUrl; nothing read it.
Move the two bit branch first and gate it on the database not existing, so a
native chain track still takes the database branch.
caab4045fa20627f1602955c53cacc05ea07de24 Fri Aug 14 11:01:17 2026 -0700
- stop directory makefiles from overriding the tree compiler flags, refs #38094
hg/hgTracks/makefile has carried "COPT = -ggdb" since 2019-02-01. Every
directory makefile includes inc/common.mk on its first line, so that
assignment came after the tree default and replaced it. The result is that
hgTracks itself has shipped with no optimization for seven years, while every
library it links against was built with -O3. Building it with the tree
default is 17.5% faster over a set of eight sessions, and 1.95x faster on a
wide clinical view, with byte identical PNG output.
Twenty three makefiles had a COPT or CFLAGS override. All are removed here.
hg/visiGene/vgPrepImage keeps its defines, but appends them with += instead
of assigning over the tree flags, because the ERMapper JPEG2000 headers need
them.
Compiling hg/hgTracks with optimization exposed seven errors that only appear
under -O3. Four source files are fixed: hgTracks.c used an uninitialized
labelfont, expRatioTracks.c and simpleTracks.c had five strncpy calls that
could leave a string unterminated, and netTrack.c had two int buffers too
small to hold the value written into them. The last two look like real
latent bugs, not warning noise.
checkCompileFlags.sh is added to stop this from recurring. The topChecks
target runs it, so every build path reaches it. It fails the build on any
COPT assignment, any CFLAGS assignment that is not +=, and any CFLAGS +=
carrying an -O flag. The check cannot live in inc/common.mk, because at the
point common.mk is parsed the overriding line has not been read yet. A
command line override such as "make COPT='-O0 -g'" still works, since the
check reads only file contents.
The 2026 conversion of the tree to -O3 did not find this. That work was
driven by the warnings -O3 produces, and a directory that overrides COPT
never receives the flag, so it never warns.
Verified: the tree builds at -O3 with no new errors and no new warnings, and
80 rendered images compare pixel identical against the old build.
- src/checkCompileFlags.sh - lines changed 82, context: html, text, full: html, text
- src/hg/altSplice/altSplice/makefile - lines changed 7, context: html, text, full: html, text
- src/hg/encode/docId/docIdSubmitDir/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/encode/docId/docIdTidy/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/encode/encodeRenameObj/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/encode/metaCheck/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/encode/regionOrtho/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/hgCollection/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/hgTracks/expRatioTracks.c - lines changed 3, context: html, text, full: html, text
- src/hg/hgTracks/hgTracks.c - lines changed 1, context: html, text, full: html, text
- src/hg/hgTracks/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/hgTracks/netTrack.c - lines changed 2, context: html, text, full: html, text
- src/hg/hgTracks/simpleTracks.c - lines changed 2, context: html, text, full: html, text
- src/hg/makeDb/simpleGap/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/mouseStuff/exonMap/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/oneShot/kgName/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/oneShot/protDat/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/pslCluster/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/pslDist/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/pslOneTwo/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/pslToFa/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/ratStuff/mafBedSubset/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/ratStuff/mafBreak/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/ratStuff/mafToProtein/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/testRewrite/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/utils/chainToBigChain/makefile - lines changed 1, context: html, text, full: html, text
- src/hg/visiGene/vgPrepImage/makefile - lines changed 2, context: html, text, full: html, text
- src/utils/bbTest/makefile - lines changed 2, context: html, text, full: html, text
443dc863b4872aa00e21a21d969f1bdf5c0f2b8d Fri Aug 14 13:06:03 2026 -0700
- pngwrite: set a fixed PNG row filter instead of searching, refs #38107
lib/pngwrite.c configures the image and never calls png_set_filter, so libpng
falls back to trying all five row filters on every row and keeping whichever
compresses best. That search is png_write_find_filter, which perf measures at
15.9% of a whole hgTracks run, and PNG encoding overall at 33.9%, more than
three times what drawing costs.
Fixing the filter to UP makes a page 19.2% faster over eight Recommended Track
Set views, geometric mean ratio 0.808. A row filter is a lossless per-row
transform, so the image is unchanged: 80 of 80 pixel comparisons identical with
a difference of zero. Output grows 1.7% over the workload, and three of the
eight images came out smaller.
UP was chosen by measuring all the filters on real browser images rather than by
reasoning about them. No filter at all is the obvious guess and is worse on
both counts: slower than UP and 12% larger.
The zlib compression level is the other half of the PNG cost and is deliberately
not touched here. It is also lossless but it trades processor time for bytes on
the wire, so it needs a decision rather than a patch. See #37398.
6de494f9973ea14ac9610977c1f58ac3aba5c792 Fri Aug 14 13:06:16 2026 -0700
- replaceFieldInPattern: skip fields the pattern never mentions, refs #38094
This runs once per item of every bigBed track that has a mouseOver or URL
pattern, and it looped over every field in the file. For each field it made two
allocations and then called dyStringSub unconditionally. dyStringSub copies the
whole string and allocates a new one even when it finds nothing to replace, so a
file with twenty fields paid twenty full rewrites per item to substitute the one
or two fields the pattern actually names. The braced form was already guarded
with stringIn; the bare form was not.
Guard the bare substitution the same way, stop early once no dollar sign is left
in the string, and keep the two specs on the stack for normal length field names.
Behavior is unchanged. dyStringSub with an absent pattern returns a byte
identical copy, so the guard only skips work that had no effect.
Measured: about 8% less processor time per page. Wall clock does not move,
because tracks load in parallel and this work sits in loader threads that were
not on the critical path. It is worth having for server capacity rather than
for page latency, and it is recorded that way so nobody expects the page to feel
faster.
d2520682cc7cac8838925cf1c9f7d9159a82df92 Sun Aug 16 10:44:02 2026 -0700
- pngwrite: replace leftover experiment marker with a real comment, refs #38107
The png_set_filter call in 443dc863b48 kept a scratch comment that said
"EXPERIMENT ONLY, refs #38094 - do not commit". The code is correct and
intended, and the justification is #38107, not #38094. Comment only, no
code change.
9ad04e0a0b06ec3c4f09ef1b6c3ce6be79b61c68 Sun Aug 16 11:56:56 2026 -0700
- cart: validate file names read back out of the cart
Several cart variables hold the name of a file the server created for a user.
Route them through one shared check, isServerUserFilePath(), which accepts the
trash directory, the session-data directories and myVariantsDataDir, and apply
it both where values enter the cart and where the file names are used.
A few of these variables may instead hold a remote URL. Those get their own
list and isServerUserFileOrUrl(), because the code that reads them chooses
between a fetch and a local open by looking for a protocol.
Consolidates two hand-rolled copies of the same test in blatShare.c and
customFactory.c, and drops the weaker private copy in sessionData.c.
Adds hg/utils/cartFileVarCatalog, a registry that scans the tree for a cart
value reaching a file call and reconciles what it finds against the lists in
cart.c, so a new one of these cannot be added without somebody noticing. Its
--reconcile is quiet enough for the nightly cron the other catalogs use, and it
is what turned up seven of the names now on those lists.
refs #37623
- src/hg/hgCollection/hgCollection.c - lines changed 2, context: html, text, full: html, text
- src/hg/hgGene/rnaStructure.c - lines changed 8, context: html, text, full: html, text
- src/hg/hgIntegrator/hgIntegrator.c - lines changed 4, context: html, text, full: html, text
- src/hg/hgTables/genomeSpace.c - lines changed 4, context: html, text, full: html, text
- src/hg/hgTables/identifiers.c - lines changed 4, context: html, text, full: html, text
- src/hg/hgTables/userRegions.c - lines changed 5, context: html, text, full: html, text
- src/hg/hgTracks/config.c - lines changed 4, context: html, text, full: html, text
- src/hg/hgTracks/hgTracks.c - lines changed 7, context: html, text, full: html, text
- src/hg/lib/customFactory.c - lines changed 5, context: html, text, full: html, text
- src/hg/lib/customTrack.c - lines changed 8, context: html, text, full: html, text
- src/hg/lib/sessionData.c - lines changed 6, context: html, text, full: html, text
- src/hg/near/hgNear/customColumn.c - lines changed 2, context: html, text, full: html, text
- src/hg/utils/cartFileVarCatalog/cartFileVarCatalog.py - lines changed 540, context: html, text, full: html, text
- src/hg/utils/cartFileVarCatalog/harvestCartFileVars.py - lines changed 513, context: html, text, full: html, text
- src/hg/visiGene/hgVisiGene/hgVisiGene.c - lines changed 2, context: html, text, full: html, text
ea99dac2c5ba8bc9de2e08c220b3929886cffd1c Sun Aug 16 12:27:17 2026 -0700
- hgConfCatalog: cite the new location of the sessionDataDirOld read
The read moved from customFactory.c to trashDir.c when the two hand-rolled
copies of that path test were folded into one. --fix-citations only repairs a
line number inside the same file, so a read that changes file has to be pointed
at its new home by hand.
refs #37925
- src/hg/utils/hgConfCatalog/hgConfCatalog.py - lines changed 1, context: html, text, full: html, text
switch to files view, user index