4697bbddd881c72cccb85b9ff0aacd769396b9d0 braney Thu Sep 10 07:34:35 2026 -0700 docent: record what evidence each regression test has, and count it A regression test written after the fix asserts the right answer, but nobody has watched it fail for the reason it exists, and a loose assertion in that state is indistinguishable from no test at all. Four of the 37 scripts here have actually been watched to flip. That was recorded only as prose in each script's header, so answering "how many of these are real regression tests" meant a grep and a read, and the number could not be quoted. Every script now carries a top-level `proof:` key, one quoted line per piece of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only the keys it names off the parsed document, so this costs a run nothing. tests/proof.js reads them and tallies, wired up as `make proof` in the shared docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line left unquoted -- that last one because nearly every note names a ticket and a bare # in an unquoted YAML scalar silently truncates the sentence at the ticket number, which is how the first pass of this change lost half its text. The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip, caught-regression. Today that reads 31 / 2 / 0 / 3 / 1. nightly.sh now records the flips it finds. An xfail that PASSES is the best evidence this suite produces -- the same server, the same fixtures, the same script, one real build apart -- and until now it arrived as a red mail and was thrown away with the log 60 days later. It is appended to /hive/users/braney/docentNightly/flips.log, one line per script ever, outside the checkout because --update resets the tree. The mail says what to do with it. The three flips that already happened (rm38272 2026-09-06, rm36212 2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded there by hand. Full suite run after the change: 37 scripts, all ok. refs #38252 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> diff --git src/hg/utils/docent/tests/regress/rm36212.docent.yaml src/hg/utils/docent/tests/regress/rm36212.docent.yaml index aa8413f23e9..228ba50298a 100644 --- src/hg/utils/docent/tests/regress/rm36212.docent.yaml +++ src/hg/utils/docent/tests/regress/rm36212.docent.yaml @@ -30,30 +30,34 @@ # itemRgbAndColor's items come back 0,0,255 at 98% of the row while its center label stays # 0,255,0. So the assertion below is known to be about the bug and not about something # near it. # # The fixture is ~/public_html/docentFixtures/itemRgbHub/, four tracks over four identical # copies of one bed9 file. Every item in the file carries a pure blue itemRgb column # (0,0,255) and the tracks that set `color` set a pure green (0,255,0), so which setting # won is readable straight off the pixels. It declares `visibility pack` itself, so there # is no `hide: all` here -- a hub track's cart name carries a per-run hub_<n>_ prefix that # `track:` cannot write. # # This is the first script in this suite to assert a COLOR, because it is the first bug # here that leaves the page identical: same rows, same height, same item names, same # tooltips, same image height. `expect: {color: ...}` reads the pixels hgTracks drew in # the row. +proof: + - "sandbox-ab 2026-09-09 -- built the bedItemRgb() reorder into parked #36212 on port 48099 (~/kentItemRgb, branch itemRgb36212); this script fails on the control and passes there" + - "server-flip 2026-09-09 -- cbb406cd96e reached genome-test and the nightly's xfail passed" + target: genome-test db: hg38 position: chr1:1000000-1010000 reset: true fast: true steps: - hub: {url: "https://hgwdev.gi.ucsc.edu/~braney/docentFixtures/itemRgbHub/hub.txt", db: hg38} - go: chr1:1000000-1010000 # All four rows have to be drawn before any color means anything. A row that failed to # load draws a pale yellow message instead of items, and the color checks below would # then be reporting on that message. - expect: rows: [itemRgbNeither, itemRgbOn, colorOnly, itemRgbAndColor] noText: "Unable to open"