4697bbddd881c72cccb85b9ff0aacd769396b9d0
braney
  Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it

A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.

Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.

tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.

The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.

nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.

Full suite run after the change: 37 scripts, all ok. refs #38252

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/docent/tests/regress/rm35865.docent.yaml src/hg/utils/docent/tests/regress/rm35865.docent.yaml
index 8fffb60b5b9..3fb2b240651 100644
--- src/hg/utils/docent/tests/regress/rm35865.docent.yaml
+++ src/hg/utils/docent/tests/regress/rm35865.docent.yaml
@@ -6,30 +6,33 @@
 #
 # So the assertion cannot be the heading. A page with the heading and no data is exactly
 # what was reported, and "Microarray Expression Data" is on it either way. The two data set
 # names under the heading are the check: GNF Expression Atlas 2 and the Gladstone set are
 # what the section is supposed to list, and neither can appear unless hgGene found data to
 # put there.
 #
 # This is also the first script here that opens hgGene at all. The suite reaches hgTracks,
 # hgTrackUi, hgc, hgSearch, hgCustom, hgConvert and hgPhyloPlace; hgGene had no coverage.
 #
 # The gene is reached by its coordinates and a click, not by pasting the hgGene URL from
 # the ticket. That URL names a transcript (ENST00000269305.9), and a transcript accession is
 # data: the next knownGene update can retire it and the script would then fail for a reason
 # that is not a bug. Clicking by POSITION rather than by item name keeps the same accession
 # out of the click. TP53's window on hg38 is a genome coordinate, which does not move.
+proof:
+  - "assertion-only 2026-09-06 -- written from the ticket after the fix had shipped"
+
 target: genome-test
 db: hg38
 reset: true
 fast: true
 position: chr17:7668421-7687490
 steps:
   - go: chr17:7668421-7687490
   - hide: all
   - track: {knownGene: pack}
   - expect: {rows: [knownGene], noText: "Warning/Error"}
 
   # By TITLE, which for knownGene is the gene symbol. Not by item:, which would be the
   # transcript accession this script is trying to avoid, and not by position: a positional
   # click looks for a box carrying an hgc href, and knownGene's boxes link to hgGene, so
   # areaXY finds nothing at all there.