File Changes for braney

v503_preview2 to v503_base (2026-08-24 to 2026-08-31) v503

• src/hg/hgConvert/hgConvert.c
  • lines changed 4, context: html, text, full: html, text
    d0942d5563ca2fdd0170c4730bb6bf1ecc8258ea Thu Aug 27 09:16:16 2026 -0700
    quickLift: take out the unused visibility string, refs #38032 trackHubBuild and walkTree never wrote visDy. hgConvert allocated it, passed it down, then reassigned it inside the chain loop, leaking a dyString per chain.
• src/hg/inc/trackHub.h
  • lines changed 1, context: html, text, full: html, text
    d0942d5563ca2fdd0170c4730bb6bf1ecc8258ea Thu Aug 27 09:16:16 2026 -0700
    quickLift: take out the unused visibility string, refs #38032 trackHubBuild and walkTree never wrote visDy. hgConvert allocated it, passed it down, then reassigned it inside the chain loop, leaking a dyString per chain.
• src/hg/lib/trackHub.c
  • lines changed 3, context: html, text, full: html, text
    d0942d5563ca2fdd0170c4730bb6bf1ecc8258ea Thu Aug 27 09:16:16 2026 -0700
    quickLift: take out the unused visibility string, refs #38032 trackHubBuild and walkTree never wrote visDy. hgConvert allocated it, passed it down, then reassigned it inside the chain loop, leaking a dyString per chain.
• src/hg/makeDb/trackDb/buildTrix
  • lines changed 39, context: html, text, full: html, text
    0b36c1276f7fd41e8ba9ad53bbdb20c1d438913d Tue Aug 25 12:08:14 2026 -0700
    trackDb: build the trix once per database, and run the beta make in parallel, refs #35489 The trackDb make for a final build takes about 18 minutes. It is not one slow step. It is 348 per-database chains of short processes and network round trips, run strictly one after another. Two changes. First, stop building the same trix twice. The beta recipe called buildTrix once for hgwbeta and again for hgwdev, with identical inputs and only the destination differing. buildTrix now takes -alsoTo=machine:path, so the index is built once and sent to both places. It also assembles the files in a temporary directory under their final names, so each destination takes one rsync instead of one per file. That takes the trix step from ten ssh connections per database down to two, and it no longer leaves .offsets files behind in the trackDb directory. Second, run the make in parallel. The header comment warning against this was out of date. hgTrackDb and hgFindSpec put their temporary files in TMPDIR under process-unique names, and every other file and table a recipe writes is already named after its database, so the per-database chains do not collide. Two makes at once are still unsafe, because they share table names, and the comment now says that instead. makeStrictBeta.csh passes -O -j 8. Set TRACKDB_MAKE_JOBS to change the job count. Going much above 8 wants ssh connection sharing first, or hgwbeta's sshd starts refusing connections. Measured over all 348 databases, with the output compared line by line against a serial run each time: 321 seconds today, 194 with the tdbQuery fix alone, 81 with -j 8 alone, and 39 with both. The -j 8 figure lands on the hg38 chain, which is why the tdbQuery fix matters more than its share of the total suggests.
  • lines changed 5, context: html, text, full: html, text
    adca1531bedb1f883c7f46f37bfcf690858050c2 Wed Aug 26 08:15:31 2026 -0700
    trackDb: rsync the trix files by name, not the directory they sit in, refs #35489 The beta make died on the first database with: rsync: [generator] failed to set times on "/data/trix/.": Operation not permitted (1) rsync error: some files/attrs were not transferred (code 23) My change yesterday replaced five per-file rsyncs with one rsync of the temporary directory. A trailing slash on the source means "the contents of this directory", but rsync still puts the directory itself in the file list. So -a gets applied to the destination directory too, and rsync tries to give /data/trix the temporary directory's timestamp. The build user does not own /data/trix, the utime call fails, and rsync exits 23. strace shows the extra call the file list form never makes: dir source: utimensat(AT_FDCWD, ".", [... 2019-09-09 ...]) = 0 file list: no call on "." The -alsoTo destination on hgwdev would have failed the same way, since the build user does not own /hive/data/inside/trix either. Name the five files as arguments, the way the code did before. That is still one rsync and one ssh connection per destination, so the speedup stays.
• src/hg/makeDb/trackDb/makefile
  • lines changed 12, context: html, text, full: html, text
    0b36c1276f7fd41e8ba9ad53bbdb20c1d438913d Tue Aug 25 12:08:14 2026 -0700
    trackDb: build the trix once per database, and run the beta make in parallel, refs #35489 The trackDb make for a final build takes about 18 minutes. It is not one slow step. It is 348 per-database chains of short processes and network round trips, run strictly one after another. Two changes. First, stop building the same trix twice. The beta recipe called buildTrix once for hgwbeta and again for hgwdev, with identical inputs and only the destination differing. buildTrix now takes -alsoTo=machine:path, so the index is built once and sent to both places. It also assembles the files in a temporary directory under their final names, so each destination takes one rsync instead of one per file. That takes the trix step from ten ssh connections per database down to two, and it no longer leaves .offsets files behind in the trackDb directory. Second, run the make in parallel. The header comment warning against this was out of date. hgTrackDb and hgFindSpec put their temporary files in TMPDIR under process-unique names, and every other file and table a recipe writes is already named after its database, so the per-database chains do not collide. Two makes at once are still unsafe, because they share table names, and the comment now says that instead. makeStrictBeta.csh passes -O -j 8. Set TRACKDB_MAKE_JOBS to change the job count. Going much above 8 wants ssh connection sharing first, or hgwbeta's sshd starts refusing connections. Measured over all 348 databases, with the output compared line by line against a serial run each time: 321 seconds today, 194 with the tdbQuery fix alone, 81 with -j 8 alone, and 39 with both. The -j 8 figure lands on the hg38 chain, which is why the tdbQuery fix matters more than its share of the total suggests.
• src/hg/utils/cartTrackVarCatalog/cartTrackVarCatalog.py
  • lines changed 10, context: html, text, full: html, text
    19be3fe94ad785b9bd787fa4e425e7e3c4f2a1d5 Fri Aug 28 14:02:40 2026 -0700
    cartTrackVarCatalog: add facetSortOrder, refs #37838 <track>.facetSortOrder, written only by facetedComposite.js and read back at hgTrackUi.c:3314 to override trackDb's defaultSortField. Same 'field=+ field2=-' syntax as the composite sortOrder it was copied from.
• src/hg/utils/docent/README.md
  • lines changed 6, context: html, text, full: html, text
    3d8157de7ba1d2e846856deda743e0aae39b0dda Tue Aug 25 16:31:36 2026 -0700
    docent: use the shared Playwright install instead of a private ~/pwrec tree, refs #38188 PW_ENV in docent.mk and in the tests makefile named $(HOME)/pwrec, a tree that only existed in one home directory, so nobody else could run docent or its tests without setting the variable by hand. Both now default to /hive/groups/browser/uiTest/pw, one pinned copy of playwright, chromium, ffmpeg and js-yaml shared by every browser-driving test in the tree. A new PW_DIR variable still points either makefile at a private install. Nothing in docent.js changes except two comment lines. All nine docent tests and all three derivation baselines pass against the shared pin.
  • lines changed 11, context: html, text, full: html, text
    074123ba2fb70164a7aa364f7c8baf21d70f55a5 Thu Aug 27 09:04:05 2026 -0700
    docent: let expect: check the order rows were drawn in, refs #37892 rows: has been a set test, and exact: true only added "and nothing else", so no assertion in the language could fail because the rows came back in the wrong order. ordered: true adds that check: the named rows have to appear top to bottom in the order given. A row that was not drawn at all is reported once, by rows:, and skipped here rather than failing twice. A failure names the first pair that is inverted. The case that prompted it is #38032, a quickLift target returning its tracks in the order they were lifted rather than the order they have on the source. A tour of that bug passed every check on the broken build. Two tests: ordered asserts the hg38 order, and ordered.xfail names the same two rows backwards and has to fail, since a flag that cannot fail is only a second copy of the set test beside it.
• src/hg/utils/docent/docent.js
  • lines changed 4, context: html, text, full: html, text
    3d8157de7ba1d2e846856deda743e0aae39b0dda Tue Aug 25 16:31:36 2026 -0700
    docent: use the shared Playwright install instead of a private ~/pwrec tree, refs #38188 PW_ENV in docent.mk and in the tests makefile named $(HOME)/pwrec, a tree that only existed in one home directory, so nobody else could run docent or its tests without setting the variable by hand. Both now default to /hive/groups/browser/uiTest/pw, one pinned copy of playwright, chromium, ffmpeg and js-yaml shared by every browser-driving test in the tree. A new PW_DIR variable still points either makefile at a private install. Nothing in docent.js changes except two comment lines. All nine docent tests and all three derivation baselines pass against the shared pin.
  • lines changed 17, context: html, text, full: html, text
    074123ba2fb70164a7aa364f7c8baf21d70f55a5 Thu Aug 27 09:04:05 2026 -0700
    docent: let expect: check the order rows were drawn in, refs #37892 rows: has been a set test, and exact: true only added "and nothing else", so no assertion in the language could fail because the rows came back in the wrong order. ordered: true adds that check: the named rows have to appear top to bottom in the order given. A row that was not drawn at all is reported once, by rows:, and skipped here rather than failing twice. A failure names the first pair that is inverted. The case that prompted it is #38032, a quickLift target returning its tracks in the order they were lifted rather than the order they have on the source. A tour of that bug passed every check on the broken build. Two tests: ordered asserts the hg38 order, and ordered.xfail names the same two rows backwards and has to fail, since a flag that cannot fail is only a second copy of the set test beside it.
• src/hg/utils/docent/docent.mk
  • lines changed 5, context: html, text, full: html, text
    3d8157de7ba1d2e846856deda743e0aae39b0dda Tue Aug 25 16:31:36 2026 -0700
    docent: use the shared Playwright install instead of a private ~/pwrec tree, refs #38188 PW_ENV in docent.mk and in the tests makefile named $(HOME)/pwrec, a tree that only existed in one home directory, so nobody else could run docent or its tests without setting the variable by hand. Both now default to /hive/groups/browser/uiTest/pw, one pinned copy of playwright, chromium, ffmpeg and js-yaml shared by every browser-driving test in the tree. A new PW_DIR variable still points either makefile at a private install. Nothing in docent.js changes except two comment lines. All nine docent tests and all three derivation baselines pass against the shared pin.
• src/hg/utils/docent/tests/README.txt
  • lines changed 1, context: html, text, full: html, text
    3d8157de7ba1d2e846856deda743e0aae39b0dda Tue Aug 25 16:31:36 2026 -0700
    docent: use the shared Playwright install instead of a private ~/pwrec tree, refs #38188 PW_ENV in docent.mk and in the tests makefile named $(HOME)/pwrec, a tree that only existed in one home directory, so nobody else could run docent or its tests without setting the variable by hand. Both now default to /hive/groups/browser/uiTest/pw, one pinned copy of playwright, chromium, ffmpeg and js-yaml shared by every browser-driving test in the tree. A new PW_DIR variable still points either makefile at a private install. Nothing in docent.js changes except two comment lines. All nine docent tests and all three derivation baselines pass against the shared pin.
  • lines changed 4, context: html, text, full: html, text
    074123ba2fb70164a7aa364f7c8baf21d70f55a5 Thu Aug 27 09:04:05 2026 -0700
    docent: let expect: check the order rows were drawn in, refs #37892 rows: has been a set test, and exact: true only added "and nothing else", so no assertion in the language could fail because the rows came back in the wrong order. ordered: true adds that check: the named rows have to appear top to bottom in the order given. A row that was not drawn at all is reported once, by rows:, and skipped here rather than failing twice. A failure names the first pair that is inverted. The case that prompted it is #38032, a quickLift target returning its tracks in the order they were lifted rather than the order they have on the source. A tour of that bug passed every check on the broken build. Two tests: ordered asserts the hg38 order, and ordered.xfail names the same two rows backwards and has to fail, since a flag that cannot fail is only a second copy of the set test beside it.
• src/hg/utils/docent/tests/makefile
  • lines changed 2, context: html, text, full: html, text
    3d8157de7ba1d2e846856deda743e0aae39b0dda Tue Aug 25 16:31:36 2026 -0700
    docent: use the shared Playwright install instead of a private ~/pwrec tree, refs #38188 PW_ENV in docent.mk and in the tests makefile named $(HOME)/pwrec, a tree that only existed in one home directory, so nobody else could run docent or its tests without setting the variable by hand. Both now default to /hive/groups/browser/uiTest/pw, one pinned copy of playwright, chromium, ffmpeg and js-yaml shared by every browser-driving test in the tree. A new PW_DIR variable still points either makefile at a private install. Nothing in docent.js changes except two comment lines. All nine docent tests and all three derivation baselines pass against the shared pin.
• src/hg/utils/docent/tests/ordered.docent.yaml
  • lines changed 26, context: html, text, full: html, text
    074123ba2fb70164a7aa364f7c8baf21d70f55a5 Thu Aug 27 09:04:05 2026 -0700
    docent: let expect: check the order rows were drawn in, refs #37892 rows: has been a set test, and exact: true only added "and nothing else", so no assertion in the language could fail because the rows came back in the wrong order. ordered: true adds that check: the named rows have to appear top to bottom in the order given. A row that was not drawn at all is reported once, by rows:, and skipped here rather than failing twice. A failure names the first pair that is inverted. The case that prompted it is #38032, a quickLift target returning its tracks in the order they were lifted rather than the order they have on the source. A tour of that bug passed every check on the broken build. Two tests: ordered asserts the hg38 order, and ordered.xfail names the same two rows backwards and has to fail, since a flag that cannot fail is only a second copy of the set test beside it.
• src/hg/utils/docent/tests/ordered.xfail.docent.yaml
  • lines changed 19, context: html, text, full: html, text
    074123ba2fb70164a7aa364f7c8baf21d70f55a5 Thu Aug 27 09:04:05 2026 -0700
    docent: let expect: check the order rows were drawn in, refs #37892 rows: has been a set test, and exact: true only added "and nothing else", so no assertion in the language could fail because the rows came back in the wrong order. ordered: true adds that check: the named rows have to appear top to bottom in the order given. A row that was not drawn at all is reported once, by rows:, and skipped here rather than failing twice. A failure names the first pair that is inverted. The case that prompted it is #38032, a quickLift target returning its tracks in the order they were lifted rather than the order they have on the source. A tour of that bug passed every check on the broken build. Two tests: ordered asserts the hg38 order, and ordered.xfail names the same two rows backwards and has to fail, since a flag that cannot fail is only a second copy of the set test beside it.
• src/hg/utils/hgConfCatalog/hgConfAges.json
  • lines changed 125, context: html, text, full: html, text
    5dc55ce0216403150c4b82f912db3339202b300f Fri Aug 28 14:02:25 2026 -0700
    hgConfCatalog: cite the file, not the line, refs #37925 A stored line number is derived data. Thirteen of the last sixteen commits from nightlyRegister.sh changed nothing but line numbers, because one edit near the top of hgTracks.c moves fifteen at once. The line is now recomputed for display from the harvester, so that job commits only when a setting is genuinely new. Also repoints login.approvedReturn and sessionDataDir at the files that read them today, classifies cspResponseHeader as a release gate, and rebuilds the age cache at v502.
• src/hg/utils/hgConfCatalog/hgConfCatalog.py
  • lines changed 488, context: html, text, full: html, text
    5dc55ce0216403150c4b82f912db3339202b300f Fri Aug 28 14:02:25 2026 -0700
    hgConfCatalog: cite the file, not the line, refs #37925 A stored line number is derived data. Thirteen of the last sixteen commits from nightlyRegister.sh changed nothing but line numbers, because one edit near the top of hgTracks.c moves fifteen at once. The line is now recomputed for display from the harvester, so that job commits only when a setting is genuinely new. Also repoints login.approvedReturn and sessionDataDir at the files that read them today, classifies cspResponseHeader as a release gate, and rebuilds the age cache at v502.
• src/hg/utils/hgConfCatalog/nightlyRegister.sh
  • lines changed 29, context: html, text, full: html, text
    5dc55ce0216403150c4b82f912db3339202b300f Fri Aug 28 14:02:25 2026 -0700
    hgConfCatalog: cite the file, not the line, refs #37925 A stored line number is derived data. Thirteen of the last sixteen commits from nightlyRegister.sh changed nothing but line numbers, because one edit near the top of hgTracks.c moves fifteen at once. The line is now recomputed for display from the harvester, so that job commits only when a setting is genuinely new. Also repoints login.approvedReturn and sessionDataDir at the files that read them today, classifies cspResponseHeader as a release gate, and rebuilds the age cache at v502.
• src/hg/utils/tdbQuery/tdbQuery.c
  • lines changed 157, context: html, text, full: html, text
    8d56c47e3c7d8cfedd92002ef9ce0ad3977e1f74 Tue Aug 25 12:08:01 2026 -0700
    tdbQuery: index record positions instead of rescanning the whole list, refs #35489 The parent/child position check called closestTdbAboveLevel once per child record, and that function walked the entire record list, called countAncestors on every candidate, and compared long file path names with strcmp. On the big assemblies this dominated the run. A profile of hg38 put 71.6% of samples on one line of that loop and another 16.6% in strcmp. Build the answer once instead. For each file, sort the record positions by line, then precompute for every position the closest record above it at or below each depth. A query is then a binary search plus one array lookup. The result is the same record the old scan returned. The test is still a strict "position before the child", and when two positions share a line the earliest in master list order still wins, which is what the backwards group walk in tdbFileIndexNew preserves. hg38 goes from 13.3 to 0.41 seconds with -release=beta, and from 49.1 to 0.64 seconds with no release. Summed over the 348 databases in the trackDb makefile, 45.7 to 14.0 seconds. The makefile test target, which queries all databases at once, goes from 84.7 to 10.5 seconds for alpha.
• src/hg/utils/urlCommandCatalog/urlCommandCatalog.py
  • lines changed 54, context: html, text, full: html, text
    ae642c6c0fb0fcc12c48e0e29168c08feedead8b Fri Aug 28 14:02:33 2026 -0700
    urlCommandCatalog: catalog hgSession's three save-form inputs, refs #37923 All three stay in the cart after the Save form: hgS_newSessionName is in 6400 of 6607 saved sessions, hgS_newSessionShare in 6398, hgS_newSessionDescription in 165. Each is removed in one of the JSON endpoints and none by the form path that posts them, so the first two were in the baseline on a reading the data does not support. New partialDrop= records which remove exists and which path it misses, since the persistence audit keys on the name alone and would otherwise call a real leak stale on every run.
• src/hg/utils/urlCommandCatalog/urlNamesNotCataloged.txt
  • lines changed 2, context: html, text, full: html, text
    ae642c6c0fb0fcc12c48e0e29168c08feedead8b Fri Aug 28 14:02:33 2026 -0700
    urlCommandCatalog: catalog hgSession's three save-form inputs, refs #37923 All three stay in the cart after the Save form: hgS_newSessionName is in 6400 of 6607 saved sessions, hgS_newSessionShare in 6398, hgS_newSessionDescription in 165. Each is removed in one of the JSON endpoints and none by the form path that posts them, so the first two were in the baseline on a reading the data does not support. New partialDrop= records which remove exists and which path it misses, since the persistence audit keys on the name alone and would otherwise call a real leak stale on every run.
• src/product/scripts/trackDbCacheCleaner.sh
  • lines changed 184, context: html, text, full: html, text
    02f92c8818bdf0f54381cd1eb962f8f5c16dea7d Tue Aug 25 16:34:19 2026 -0700
    trackDbCacheCleaner: expire trackDb cache entries that nothing reads, refs #37551 The trackDb cache holds one directory per database or track hub. The browser expires a cache file only when a request visits its directory and finds the file older than the trackDb table or the hub, and nothing ever removes a directory. A hub URL that is requested once and never again keeps its cache files forever. A tmpfs cache directory hides this until the next reboot. A disk backed cache directory never hides it, and /data/trackDbCache on hgwdev had grown to 16G in 49,749 directories since it was created on May 13. trackDbCacheCleaner.sh removes a cache directory when nothing in it has been read for expireDays days, 30 by default. Last use is the newer of the access time and the modify time, so on a file system mounted noatime it falls back to the write time, which only ever keeps entries longer. A directory still in use is left alone, because the browser opens every file in a directory it visits and that keeps the access times current. The first run on hgwdev took 30 seconds and dropped /data/trackDbCache from 16G to 8.2G, removing 30,117 directories and 60,504 files. hg38, hg19 and mm39 were untouched, and an expired assembly rebuilt its entry on the next request.
  • lines changed 22, context: html, text, full: html, text
    4088b6e57be69de4f37e00beefb13969a34eb09e Tue Aug 25 16:40:27 2026 -0700
    trackDbCacheCleaner: do not abandon the run when one removal fails, refs #37551 A cache directory can be owned by any user who ran a command line utility, not just by apache, so a removal can fail on permissions. With set -beEu that aborted the whole run part way through, leaving a half cleaned cache and printing no summary at all. Count the failures instead, warn about the first ten, and finish the rest of the cleaning. A run with any failure reports FAILED and exits 1, so a cron notices, and the next run retries what was left behind.
• src/utils/qa/weeklybld/makeStrictBeta.csh
  • lines changed 8, context: html, text, full: html, text
    0b36c1276f7fd41e8ba9ad53bbdb20c1d438913d Tue Aug 25 12:08:14 2026 -0700
    trackDb: build the trix once per database, and run the beta make in parallel, refs #35489 The trackDb make for a final build takes about 18 minutes. It is not one slow step. It is 348 per-database chains of short processes and network round trips, run strictly one after another. Two changes. First, stop building the same trix twice. The beta recipe called buildTrix once for hgwbeta and again for hgwdev, with identical inputs and only the destination differing. buildTrix now takes -alsoTo=machine:path, so the index is built once and sent to both places. It also assembles the files in a temporary directory under their final names, so each destination takes one rsync instead of one per file. That takes the trix step from ten ssh connections per database down to two, and it no longer leaves .offsets files behind in the trackDb directory. Second, run the make in parallel. The header comment warning against this was out of date. hgTrackDb and hgFindSpec put their temporary files in TMPDIR under process-unique names, and every other file and table a recipe writes is already named after its database, so the per-database chains do not collide. Two makes at once are still unsafe, because they share table names, and the comment now says that instead. makeStrictBeta.csh passes -O -j 8. Set TRACKDB_MAKE_JOBS to change the job count. Going much above 8 wants ssh connection sharing first, or hgwbeta's sshd starts refusing connections. Measured over all 348 databases, with the output compared line by line against a serial run each time: 321 seconds today, 194 with the tdbQuery fix alone, 81 with -j 8 alone, and 39 with both. The -j 8 figure lands on the hg38 chain, which is why the tdbQuery fix matters more than its share of the total suggests.
• src/utils/ts/README
  • lines changed 15, context: html, text, full: html, text
    4f9164cb6142e7468325253a67c6ac8a6a3d1f91 Tue Aug 25 09:54:31 2026 -0700
    ts: give each ticket sandbox its own udc cache, refs #37867 Every parked instance and the live sandbox shared trash/udcCache, so a cache entry written by one instance was read by all of them. A sparse or bad entry from one build reappeared under another, which breaks the freeze. Each ticket now gets trash/udcCache/ts/NNNNN. It stays in the shared trash, so the trash cleaner ages it out like any other udc cache, and it stays data, so sync does not clear it. remove deletes it, since it sits inside the shared trash and not in the sandbox directory. conf now sets it too, so an instance frozen before this existed does not need a re-freeze.
• src/utils/ts/ts
  • lines changed 28, context: html, text, full: html, text
    4f9164cb6142e7468325253a67c6ac8a6a3d1f91 Tue Aug 25 09:54:31 2026 -0700
    ts: give each ticket sandbox its own udc cache, refs #37867 Every parked instance and the live sandbox shared trash/udcCache, so a cache entry written by one instance was read by all of them. A sparse or bad entry from one build reappeared under another, which breaks the freeze. Each ticket now gets trash/udcCache/ts/NNNNN. It stays in the shared trash, so the trash cleaner ages it out like any other udc cache, and it stays data, so sync does not clear it. remove deletes it, since it sits inside the shared trash and not in the sandbox directory. conf now sets it too, so an instance frozen before this existed does not need a re-freeze.