4697bbddd881c72cccb85b9ff0aacd769396b9d0
braney
  Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it

A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.

Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.

tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.

The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.

nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.

Full suite run after the change: 37 scripts, all ok. refs #38252

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/docent/tests/regress/rm36029.docent.yaml src/hg/utils/docent/tests/regress/rm36029.docent.yaml
index 5168f46a3e8..526e0e96c0e 100644
--- src/hg/utils/docent/tests/regress/rm36029.docent.yaml
+++ src/hg/utils/docent/tests/regress/rm36029.docent.yaml
@@ -1,41 +1,44 @@
 # #36029 -- with several phyloP tracks on and MAF hidden, turning a MAF to full did not
 # display it. The controls showed it as full; the row was not drawn. A cart reset cleared
 # it, and it came back later.
 #
 # The ticket's session is on the RR (/s/Markd/maf-display-bug-36029) and so cannot be
 # loaded from genome-test, and the state it holds is three phyloP tracks, which is a step
 # to build.
 #
 # Track names on hg38: the conservation containers are cons100way (UCSC 100 Vertebrates),
 # cons241way (Zoonomia 241 Placent) and cons470way (Hiller Lab 470 Mammals). Each has a
 # phyloP view and an alignment view. The 241-way MAF the reporter could not display is
 # cactus241wayBM, under cons241wayViewalign.
 #
 # Asking for a phyloP view turns the whole composite on, alignment view included: a first
 # version of this test asserted the MAF was absent after naming only the phyloP views and
 # found multiz100way, cactus241wayBM and multiz470way all drawn. So the alignment views
 # have to be hidden by name to reach the state the report started from. That is the same
 # shape as the views test in the suite next door, where a hideKids has to be aimed at the
 # view that holds the subtrack rather than at the composite.
+proof:
+  - "assertion-only 2026-09-04 -- written from the ticket after the fix had shipped"
+
 target: genome-test
 db: hg38
 position: chr7:155799529-155812871
 reset: true
 fast: true
 steps:
   - go: chr7:155799529-155812871
   - hide: all
 
   # Three phyloP tracks on, all three alignment views off: the state the report began in.
   - track:
       cons100wayViewphyloP: full
       cons241wayViewphyloP: full
       cons470wayViewphyloP: full
       cons100wayViewalign: hide
       cons241wayViewalign: hide
       cons470wayViewalign: hide
   - expect: {noRows: [cactus241wayBM, multiz100way, multiz470way], noText: "Warning/Error"}
 
   # Now ask for the 241-way MAF. This is the step that did nothing.
   - track: {cactus241wayBM: full}
   - expect: {rows: [cactus241wayBM], noText: "Warning/Error"}