4697bbddd881c72cccb85b9ff0aacd769396b9d0 braney Thu Sep 10 07:34:35 2026 -0700 docent: record what evidence each regression test has, and count it A regression test written after the fix asserts the right answer, but nobody has watched it fail for the reason it exists, and a loose assertion in that state is indistinguishable from no test at all. Four of the 37 scripts here have actually been watched to flip. That was recorded only as prose in each script's header, so answering "how many of these are real regression tests" meant a grep and a read, and the number could not be quoted. Every script now carries a top-level `proof:` key, one quoted line per piece of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only the keys it names off the parsed document, so this costs a run nothing. tests/proof.js reads them and tallies, wired up as `make proof` in the shared docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line left unquoted -- that last one because nearly every note names a ticket and a bare # in an unquoted YAML scalar silently truncates the sentence at the ticket number, which is how the first pass of this change lost half its text. The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip, caught-regression. Today that reads 31 / 2 / 0 / 3 / 1. nightly.sh now records the flips it finds. An xfail that PASSES is the best evidence this suite produces -- the same server, the same fixtures, the same script, one real build apart -- and until now it arrived as a red mail and was thrown away with the log 60 days later. It is appended to /hive/users/braney/docentNightly/flips.log, one line per script ever, outside the checkout because --update resets the tree. The mail says what to do with it. The three flips that already happened (rm38272 2026-09-06, rm36212 2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded there by hand. Full suite run after the change: 37 scripts, all ok. refs #38252 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> diff --git src/hg/utils/docent/tests/regress/rm36836.docent.yaml src/hg/utils/docent/tests/regress/rm36836.docent.yaml index 411a565e27f..4e4961695cc 100644 --- src/hg/utils/docent/tests/regress/rm36836.docent.yaml +++ src/hg/utils/docent/tests/regress/rm36836.docent.yaml @@ -1,40 +1,43 @@ # #36836 -- a GenArk assembly and an assembly hub both crashed when opened by URL, and on # hgwdev only. # # Reported with four URLs: two on hgwdev that crashed and the same two on hgwbeta and hgw0 # that were fine. That shape is the whole point of the ticket. The code that broke it was # only ever on hgwdev, so a release-branch build could not show it and the RR was never # affected. genome-test IS hgwdev's build, which is why this test belongs here and would # have been worthless anywhere else. # # It was not fixed so much as withdrawn: Galt reverted the two commits from #36835 # (0542bdef848 and 289c4a361a8) at Brian's request, over a speed concern that had not been # measured. So the behavior asserted here is the behavior from before #36835, and what the # script guards against is a second attempt at that work landing the same way. # # No `db:` at the top on purpose -- neither step uses one. Each URL names its own assembly, # which is exactly the gesture the ticket reported, so both are driven with goto: rather # than with the hub: verb. # # The daph hub is a public UCSC example hub on GitHub, not a colleague's public_html, so it # is about as durable as an outside fixture gets. preflight reads the hubUrl out of the # goto below, so if GitHub ever moves it the nightly says a fixture is missing rather than # reporting a browser regression. # # Both assertions name a real data row, not just the ruler. A crash left no track image at # all, so the ruler alone would already fail on it -- but a page that draws the ruler and # nothing else is also what an assembly whose tracks did not resolve looks like, and that # is the half of this bug that was about hub loading rather than about the CGI staying up. # gc5Base is in every GenArk hub, and myTrack is the daph hub's only track. Row names are # matched by suffix, so the per-run hub_<n>_ prefix on both does not have to be known. +proof: + - "assertion-only 2026-09-06 -- written from the ticket after the fix had shipped" + target: genome-test reset: true fast: true steps: # The GenArk assembly, by accession, with no hub URL of its own. - goto: "/cgi-bin/hgTracks?db=GCF_000003625.3&pix=1100" - expect: {rows: [ruler, gc5Base], noText: "Warning/Error"} # The assembly hub, by genome= plus hubUrl=, at the position the ticket gave. - goto: "/cgi-bin/hgTracks?genome=daph&hubUrl=https://raw.githubusercontent.com/ucsc-browser/assemblyHubEx/master/Daphnia/hubExamples/hubAssembly/daph/hub.txt&position=scaffold_1%3A35591-35626&pix=1100" - expect: {rows: [ruler, myTrack], noText: "Warning/Error"}