4697bbddd881c72cccb85b9ff0aacd769396b9d0 braney Thu Sep 10 07:34:35 2026 -0700 docent: record what evidence each regression test has, and count it A regression test written after the fix asserts the right answer, but nobody has watched it fail for the reason it exists, and a loose assertion in that state is indistinguishable from no test at all. Four of the 37 scripts here have actually been watched to flip. That was recorded only as prose in each script's header, so answering "how many of these are real regression tests" meant a grep and a read, and the number could not be quoted. Every script now carries a top-level `proof:` key, one quoted line per piece of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only the keys it names off the parsed document, so this costs a run nothing. tests/proof.js reads them and tallies, wired up as `make proof` in the shared docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line left unquoted -- that last one because nearly every note names a ticket and a bare # in an unquoted YAML scalar silently truncates the sentence at the ticket number, which is how the first pass of this change lost half its text. The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip, caught-regression. Today that reads 31 / 2 / 0 / 3 / 1. nightly.sh now records the flips it finds. An xfail that PASSES is the best evidence this suite produces -- the same server, the same fixtures, the same script, one real build apart -- and until now it arrived as a red mail and was thrown away with the log 60 days later. It is appended to /hive/users/braney/docentNightly/flips.log, one line per script ever, outside the checkout because --update resets the tree. The mail says what to do with it. The three flips that already happened (rm38272 2026-09-06, rm36212 2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded there by hand. Full suite run after the change: 37 scripts, all ok. refs #38252 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> diff --git src/hg/utils/docent/tests/regress/makefile src/hg/utils/docent/tests/regress/makefile index 5d6f4edf8b8..f6d6dfaf395 100644 --- src/hg/utils/docent/tests/regress/makefile +++ src/hg/utils/docent/tests/regress/makefile @@ -1,29 +1,34 @@ # Docent REGRESSION tests: one script per bug we have already fixed, asserting the # behavior the ticket says is correct. Separate from ../ on purpose, and run nightly # against genome-test. # # Measured 2026-09-04: the eleven scripts next door take 56s all together (3-8s each, all # in FAST mode). A `convert:` test measured 16s, because a quickLift hop is hgConvert plus # a hub build plus the click through to the browser. Fourteen of the forty here open with # `convert:`, so the estimate for a full run is about seven minutes, not the half hour a # guess would suggest. Re-measure rather than trusting this comment. # # make test # every *.docent.yaml here # make test T=rm36382 # just one +# make proof # what evidence each script has that it would catch its bug # make derive # the derivation alone, for the scripts with a baseline # # Naming: rm<ticket>.docent.yaml, so the ticket is the filename. Every script says in its # first lines which ticket it is, what the symptom was, and what the assertion would # catch -- a regression test whose comment does not name the bug is unmaintainable, since # the next reader cannot tell a deliberate assertion from an accident. # -# These assert the FIXED behavior and were never watched to fail on a buggy build, so the -# assertions are deliberately tight: `exact:` and `noRows:` rather than a bare `rows:`, -# and `noText:` on the exact error string the ticket quoted. A loose assertion here is -# indistinguishable from no test at all. +# Most of these assert the FIXED behavior and were never watched to fail on a buggy build, +# so the assertions are deliberately tight: `exact:` and `noRows:` rather than a bare +# `rows:`, and `noText:` on the exact error string the ticket quoted. A loose assertion +# here is indistinguishable from no test at all. +# +# Which ones HAVE been watched to fail for their own reason is recorded in each script's +# `proof:` key and counted by `make proof`. See README.txt for the levels and for the two +# ways to earn one. DOCENT = ../../docent.js PREFLIGHT = ../preflight.js include ../docentTest.mk