4697bbddd881c72cccb85b9ff0aacd769396b9d0
braney
  Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it

A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.

Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.

tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.

The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.

nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.

Full suite run after the change: 37 scripts, all ok. refs #38252

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/docent/tests/regress/rm37489.docent.yaml src/hg/utils/docent/tests/regress/rm37489.docent.yaml
index 2b1adfae954..c311f98b223 100644
--- src/hg/utils/docent/tests/regress/rm37489.docent.yaml
+++ src/hg/utils/docent/tests/regress/rm37489.docent.yaml
@@ -1,50 +1,53 @@
 # #37489 -- a quickLifted track showed the TEMPLATE of its dataVersion setting instead of
 # the version.
 #
 # ClinVar's trackDb carries `dataVersion /gbdb/$D/bbi/clinvar/version.txt`, and the browser
 # reads that file and prints what is in it. On a lifted track $D was never substituted, so
 # the settings page showed the reader:
 #
 #     Version: /gbdb/$D/bbi/clinvar/version.txt
 #
 # a path with an unexpanded variable in it, where a release date belongs.
 #
 # This is the first script here that asserts a VALUE rather than the shape of a page.
 # Everything else says a row was drawn, or that a page does not carry an error. None of
 # them would notice a track that draws perfectly and tells the reader something false, and
 # that is most of what a data bug looks like.
 #
 # The positive check is the PREFIX only. "ClinVar Release:" is written by the pipeline and
 # does not move; the date after it changes every month, and asserting it would make this
 # script fail on the next ClinVar update, which is not a browser regression. noText: "$D"
 # is the symptom itself, and it cannot stand alone: a page that failed to render carries no
 # "$D" either.
 #
 # The lift is done here rather than loaded from Gerardo's RM_37489 session. That session
 # still exists on genome-test but its quickLift is gone -- it comes back as plain hg38 with
 # clinvarMain, clinvarCnv and clinvarSubLolly and no hub_ prefix on any of them. A lifted
 # session dies with the trash it points at, so a script that needs a lift has to make one.
+proof:
+  - "assertion-only 2026-09-06 -- written from the ticket after the fix had shipped"
+
 target: genome-test
 db: hg38
 position: chr13:32396673-32400065
 reset: true
 fast: true
 steps:
   - go: chr13:32396673-32400065
   - hide: all
   - track: {clinvarMain: pack}
   - expect: {rows: [clinvarMain]}
 
   - convert: {to: hs1, quicklift: true}
   - open: lift
   - expect: {rows: [clinvarMain], noText: "Warning/Error"}
 
   # The lifted COMPOSITE's settings page, which is where dataVersion is printed. Measured
   # on genome-test: a lifted view carries no hgTrackUi link for the subtrack at all. Every
   # settings link under ClinVar points at the container, g=hub_<n>_clinvar, and there are
   # four of them on the page, so the first one is named explicitly rather than left to a
   # selector that matches several.
   - click: 'a[href*="hgTrackUi"][href*="_clinvar"] >> nth=0'
   - expect:
       text: "ClinVar Release:"
       noText: "$D"