4697bbddd881c72cccb85b9ff0aacd769396b9d0 braney Thu Sep 10 07:34:35 2026 -0700 docent: record what evidence each regression test has, and count it A regression test written after the fix asserts the right answer, but nobody has watched it fail for the reason it exists, and a loose assertion in that state is indistinguishable from no test at all. Four of the 37 scripts here have actually been watched to flip. That was recorded only as prose in each script's header, so answering "how many of these are real regression tests" meant a grep and a read, and the number could not be quoted. Every script now carries a top-level `proof:` key, one quoted line per piece of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only the keys it names off the parsed document, so this costs a run nothing. tests/proof.js reads them and tallies, wired up as `make proof` in the shared docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line left unquoted -- that last one because nearly every note names a ticket and a bare # in an unquoted YAML scalar silently truncates the sentence at the ticket number, which is how the first pass of this change lost half its text. The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip, caught-regression. Today that reads 31 / 2 / 0 / 3 / 1. nightly.sh now records the flips it finds. An xfail that PASSES is the best evidence this suite produces -- the same server, the same fixtures, the same script, one real build apart -- and until now it arrived as a red mail and was thrown away with the log 60 days later. It is appended to /hive/users/braney/docentNightly/flips.log, one line per script ever, outside the checkout because --update resets the tree. The mail says what to do with it. The three flips that already happened (rm38272 2026-09-06, rm36212 2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded there by hand. Full suite run after the change: 37 scripts, all ok. refs #38252 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> diff --git src/hg/utils/docent/tests/regress/README.txt src/hg/utils/docent/tests/regress/README.txt index 60e45b9c2e5..e8649af87a1 100644 --- src/hg/utils/docent/tests/regress/README.txt +++ src/hg/utils/docent/tests/regress/README.txt @@ -1,73 +1,105 @@ Docent regression tests ----------------------- One script per fixed bug, named for its ticket. Run by hand: make test # every *.docent.yaml here make test T=rm36382 # just one + make proof # what evidence each script has, and the tally The candidate list this directory is being built from, with the recipe and assertion worked out for each ticket, is at /hive/groups/browser/redmineNotes/37892/claude/2026-09-04_1100_regression_candidates.md What these are, and what they are not ------------------------------------- -Each script asserts the behavior the ticket says is correct, on genome-test. None of them -was run against a build that still had the bug, so none has been seen to fail for the -reason it exists. That is a deliberate choice about cost, and it puts the whole weight on -how tight the assertion is: +Each script asserts the behavior the ticket says is correct, on genome-test. Most were +written after the fix had already shipped, so most have never been seen to fail for the +reason they exist. `make proof` says exactly how many have and which ones, reading a +`proof:` key that every script carries; as of 2026-09-10 it is 4 of 37. That is a +deliberate choice about cost, and for the other 33 it puts the whole weight on how tight +the assertion is: * name the error string the ticket quoted in `noText:`, not a generic "Error" * prefer `rows: [...] exact: true` and `noRows:` over a bare `rows:` * a test that only checks a row is PRESENT usually passes on the buggy build too, because the bug was an extra row, a wrong label, or a bad tooltip Three things will rot these tests --------------------------------- Most recipes start from the saved session named in the ticket, because that is the cheapest way to reach the exact state. A session that is deleted does not fail loudly: hgTracks serves a page saying it could not find it, and every `noText:` check on that page passes. So a session-based test also asserts something that is only true when the session really loaded. Six recipes need a test hub on a colleague's public_html. Same problem, same remedy. Fixtures we own live in ~/public_html/docentFixtures/, and `make preflight` checks that every hub a script here names still answers. Copy a reporter's hub in there rather than loading theirs, so nothing outside this repository can change what a test measures. A fixture hub must never name a track anything the assembly might also call it. A track name resolves to `img_data_<name>` first and only then to a hub row's `hub_<n>_<name>`, so an exact native id wins: the hub row is on the page, and every `track:`, `mouseover:` and `rows:` in the script reads the NATIVE row instead. Nothing warns. rm35920's fixture called its track `ultras`, hg38 has its own `ultras`, and that script asserted a tooltip off the native data for as long as it existed -- it looked green and tested nothing. Prefix a fixture's track names with the ticket number. -rm36212 is the one to read before writing another --------------------------------------------------- +Proof: which scripts have been watched to fail for their own reason +------------------------------------------------------------------- -It is the only script here that has been watched to fail on a build with the bug AND to -pass on a build with the fix, which is the evidence every other script in this directory -would like to have and does not. The recipe: build the fix into a ticket sandbox, point a -copy of the script at that port with `target: http://127.0.0.1:PORT/cgi-bin`, and record -in the comment which checks flipped. It costs one build and it settles what a tight -assertion can only argue. +Every script carries a top-level `proof:` key, one quoted line per piece of evidence, +each `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only the keys it names, +so the key costs a run nothing. `make proof` tallies it and fails on a line that is +malformed or names a level outside the vocabulary, which is what keeps it countable. + +The levels, weakest first: + + assertion-only asserts the fixed behavior; never seen to fail for its own reason + xfail seen failing right now for its own reason; the fix has not shipped + sandbox-ab seen failing on a build with the bug and passing on a build with + the fix, both built by hand + server-flip seen failing then passing on a real server as a real build arrived + caught-regression went red for a regression that was then filed and fixed + +Two ways to earn the middle levels. sandbox-ab is the one you can choose to do: build the +fix into a ticket sandbox, point a copy of the script at that port with +`target: http://127.0.0.1:PORT/cgi-bin`, and record which checks flipped. It costs one +build and it settles what a tight assertion can only argue. + +server-flip is the one this directory gets for free, and it is better evidence, because +nothing about the server changed except the build. Commit a script for an unshipped fix +as an .xfail. `make test` fails when an xfail PASSES, so the morning the fix reaches +genome-test the nightly goes red and says so. nightly.sh appends that to + + /hive/users/braney/docentNightly/flips.log + +one line per script ever, outside the checkout because --update resets the tree. Then +drop the .xfail from the name and add the server-flip line to the script's proof: key. +rm38272, rm36212 and rm38310 all arrived that way. + +rm36212 is still the one to read before writing another +-------------------------------------------------------- + +It is the worked example of both routes: sandbox-ab on 2026-09-09 against parked #36212 +on port 48099, then server-flip the same morning when cbb406cd96e reached genome-test. It is also the first script to assert a COLOR, using `expect: {color: ...}`, because it is the first bug here that leaves the page identical -- same rows, same height, same item names, same tooltips. When rows:, height:, text: and has: are all blind to a bug, the pixels are what is left. See tests/colorchecks.docent.yaml for the check itself. rm38310 is the second color check, and the second script watched both ways --------------------------------------------------------------------------- Same recipe as rm36212, and worth reading for the reason it needs pixels, which is different. Its bug does not draw the wrong color; it replaces the row with the bigWarn bar, 240,240,180 (undefinedYellowColor, hg/hgTracks/simpleTracks.c), and paints an error message INSIDE the png. So the row is still drawn, still the same name, and every text check on the page passes -- the message is in the image, where noText: cannot reach it. That is also why the ticket was filed saying there was no warning at all.