4697bbddd881c72cccb85b9ff0aacd769396b9d0
braney
  Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it

A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.

Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.

tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.

The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.

nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.

Full suite run after the change: 37 scripts, all ok. refs #38252

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/docent/tests/regress/rm37389.docent.yaml src/hg/utils/docent/tests/regress/rm37389.docent.yaml
index dd56c6bf31c..00c2ce748b1 100644
--- src/hg/utils/docent/tests/regress/rm37389.docent.yaml
+++ src/hg/utils/docent/tests/regress/rm37389.docent.yaml
@@ -1,50 +1,53 @@
 # #37389 -- the track description was missing from a quickLifted hgc page when the source
 # assembly is a GenArk hub.
 #
 # It was broken twice and by different causes, which is why the comment is long.
 #
 # The v498 fix (5aed2d465f1) filled in a TODO branch in getTrackHtml() (hg/lib/hui.c) that
 # had done nothing whenever quickLiftDb named a hub assembly. That fix works.
 #
 # Then 1258d7f65e7 ("trackHub: keep a description a track inherited from its parent", refs
 # #38126) broke it again on master, and only on master: measured 2026-09-05, v502 and v503
 # both drew the description and the tip did not. It made trackHubAddOneDescription return
 # early rather than assign a failed fetch, so tdb->html keeps a non-NULL value; hgc's
 # getHtmlFromSelfOrParent asked `tdb->html == NULL` before falling back to the source hub,
 # that test stopped holding, and getTrackHtml was never called. Filed as #38275 and fixed
 # in 4be6477ebcb by asking isEmpty() instead.
 #
 # So this script has now watched the behaviour break and come back, which is more than the
 # rest of the suite can say. It was an .xfail between the two.
 #
 # The session is braney/crash1, the same one rm37388 uses, because it pins a state that is
 # awkward to build in steps: hg38 with a quickLift hub active whose source is the GenArk
 # GCA_018466835.2, and an xenoRefGene track from that hub. rm38272 uses it too, so preflight
 # covers it for all three.
 #
 # A separate from-scratch reproduction, which needs no session at all and can be pointed at
 # any server, is kept at
 # /hive/groups/browser/redmineNotes/38252/claude/repro_37388_hgTrackUi.js -- it was written
 # for #38272 but walks the same ground. There is also a Docent tour of this bug at
 # ~/docentTours/bugs/rm37389.docent.yaml, recorded while it was broken.
+proof:
+  - "caught-regression 2026-09-05 -- 1258d7f65e7 broke the description again on the tip only, v502 and v503 still drew it; filed as #38275, fixed in 4be6477ebcb and green again"
+
 target: genome-test
 db: hg38
 reset: true
 fast: true
 steps:
   - loadSession: {user: braney, name: crash1}
   - expect: {noText: "Could not find session"}
 
   - click: {track: xenoRefGene, item: "NM_126741.2"}
 
   # The page itself is fine -- that is #37388, which is fixed and has its own script. What
   # is missing is only the description block.
   - expect: {noText: "Couldn't set connection database", text: "Item: NM_126741.2"}
 
   # The assertion this script exists for. "RefSeq mRNAs mapped to this assembly" is the
   # longLabel and is on the page whether or not the description rendered, so neither half of
   # this is that: "Description" is the heading printTrackHtml writes above the block, and the
   # second is a phrase out of the hub's own description file, which is the only thing that
   # can only be there if the file was really fetched from the GenArk hub.
   - expect: {text: "Description"}
   - expect: {text: "translated blat alignments"}