4697bbddd881c72cccb85b9ff0aacd769396b9d0
braney
  Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it

A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.

Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.

tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.

The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.

nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.

Full suite run after the change: 37 scripts, all ok. refs #38252

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/docent/tests/regress/rm36805.docent.yaml src/hg/utils/docent/tests/regress/rm36805.docent.yaml
index f61db2b4bd0..07d2211188b 100644
--- src/hg/utils/docent/tests/regress/rm36805.docent.yaml
+++ src/hg/utils/docent/tests/regress/rm36805.docent.yaml
@@ -1,48 +1,51 @@
 # #36805 -- clicking a TOGA item a SECOND time hung the display.
 #
 # Reported by Hiram on mm10: turn on only "TOGA vs hg38", click an item to get the details
 # pop-up, dismiss it, click any item again, and the display hangs. Gerardo's recipe on the
 # ticket is exact and is what this follows: load the session, click
 # ENST00000518529.SFTA3.1, click the x on the pop-up, click the same item again.
 #
 # This is the first script here that does anything TWICE. Everything else in the suite is a
 # straight line from a fresh cart: reset, a few steps, assert. A bug that only exists on the
 # repeat is invisible to that shape, and there are at least two of them in the pool (#36805
 # and #37014). One click of this item passes on the broken build as happily as on the fixed
 # one, so the second click is not a flourish, it IS the test.
 #
 # raw: true on the click is the other half. Docent's ordinary item click follows the item's
 # own map-box href straight to hgc, which is the right thing when the assertion is about the
 # details PAGE. Here the bug is in the DIALOG: hgTracks answers a real click with an ajax
 # pop-up (popUpHgcOrHgGene.hgc in hg/js/hgTracks.js), and following the href never opens one,
 # so it could never see this. raw: presses the mouse where a user presses it and lets the
 # page do what it does.
 #
 # :visible matters on both checks. jQuery UI HIDES #hgcDialog on close, it does not remove
 # it, so a bare has: "#hgcDialog" would match the closed dialog and the second assertion
 # would pass without the pop-up ever coming back.
 #
 # No build left to watch this fail on: fixed for v492 in December 2025, so hgwbeta and the
 # RR have had it for months.
+proof:
+  - "assertion-only 2026-09-06 -- written from the ticket after the fix had shipped"
+
 target: genome-test
 db: mm10
 reset: true
 fast: true
 steps:
   - loadSession: {user: Gerardo, name: RM_36805_TOGA_hangs}
 
   # A session that has gone away is not an error -- hgTracks serves a 200 with no image and
   # every noText on it passes -- so the row check is what says the session really loaded.
   - expect: {noText: "Could not find session", rows: [HLTOGAannotvHg38v1]}
 
   - click: {track: HLTOGAannotvHg38v1, item: "ENST00000518529.SFTA3.1", raw: true}
   - expect: {has: "#hgcDialog:visible"}
 
   # The x button, which is the gesture the ticket names. Scoped to the visible dialog: the
   # page carries more than one jQuery UI dialog in its DOM.
   - click: '.ui-dialog:visible .ui-dialog-titlebar-close'
   - expect: {noHas: "#hgcDialog:visible"}
 
   # The whole point. Same item, same gesture, second time.
   - click: {track: HLTOGAannotvHg38v1, item: "ENST00000518529.SFTA3.1", raw: true}
   - expect: {has: "#hgcDialog:visible"}