4697bbddd881c72cccb85b9ff0aacd769396b9d0
braney
  Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it

A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.

Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.

tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.

The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.

nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.

Full suite run after the change: 37 scripts, all ok. refs #38252

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/docent/tests/regress/rm35333.docent.yaml src/hg/utils/docent/tests/regress/rm35333.docent.yaml
index ab5873fc8f7..a9a2861ebdb 100644
--- src/hg/utils/docent/tests/regress/rm35333.docent.yaml
+++ src/hg/utils/docent/tests/regress/rm35333.docent.yaml
@@ -1,33 +1,36 @@
 # #35333 -- a bigBed whose field count disagrees with the trackDb type line caused a
 # SEGFAULT, which reached the reader as "hgTracks object missing" and, in the ticket's
 # words, was bewildering to a collaborator.
 #
 # The track declares `type bigBed 12 +` over a bed9+ file. hubCheck catches the mismatch,
 # which is good, but a malformed hub should not crash the browser.
 #
 # Our own copy of the reporter's hub, at ~/public_html/docentFixtures/hgtracks_obj_bug/,
 # with bigDataUrl pointing at the local file. Note the clone's `groups` and `html` lines
 # still point at the reporter's public_html, which hubClone left absolute; those are
 # cosmetic for this test but are a dependency worth removing if it ever fails oddly.
 #
 # The position is the first contig in 1q21_contigs.bb (chr1:142,654,935-142,673,231). The
 # ticket's own steps say to go to chr1 after hiding everything, and the crash needs an
 # item in view.
+proof:
+  - "assertion-only 2026-09-04 -- written from the ticket after the fix had shipped"
+
 target: genome-test
 db: hs1
 position: chr1:142654000-142674000
 reset: true
 fast: true
 steps:
   # No hide: all and no track: step here, on purpose. The hub declares `visibility pack`
   # for this track, so attaching it is enough. A first version hid everything and then
   # named the track, and the track did not come back: for an ATTACHED hub the cart
   # variable is the hub-prefixed name (hub_194767_H9_shasta_1q21), and track: sends the
   # plain name, which turns on nothing. rows: matches by suffix on the way out, so the
   # assertion below does not need the prefix -- but the cart write would have.
   - hub: {url: "https://hgwdev.gi.ucsc.edu/~braney/docentFixtures/hgtracks_obj_bug/hub.txt", db: hs1}
   - go: chr1:142654000-142674000
 
   # The row check is the real assertion: the SEGFAULT produced no image at all, so a
   # noText check on its own would have passed on the page the bug produced.
   - expect: {rows: [H9_shasta_1q21], noText: "hgTracks object missing"}