4697bbddd881c72cccb85b9ff0aacd769396b9d0
braney
  Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it

A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.

Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.

tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.

The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.

nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.

Full suite run after the change: 37 scripts, all ok. refs #38252

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/docent/tests/regress/rm36340.docent.yaml src/hg/utils/docent/tests/regress/rm36340.docent.yaml
index 671b131c061..6de79bbeb3d 100644
--- src/hg/utils/docent/tests/regress/rm36340.docent.yaml
+++ src/hg/utils/docent/tests/regress/rm36340.docent.yaml
@@ -1,50 +1,53 @@
 # #36340 -- a search-result click after a quickLift could not find the source coordinate.
 #
 # Gerardo's steps: load his session, QuickLift to human HG02257.pat 2021, search "brca2" on
 # the lifted view, and click the BRCA2 hit under "MANE Select Plus Clinical". The result was
 #
 #     Sorry, couldn't locate chr13:32315508-32400268 in HG02257.alt.pat.f1_v2 May 2021
 #     human (HG02257.pat 2021)
 #
 # -- the hit carried the SOURCE assembly's coordinate and the target could not resolve it.
 # Same error from the "NCBI Gene Orthologs" hit.
 #
 # The target is named by ACCESSION, not by label, and that is not cosmetic here. GenArk
 # carries two releases whose labels differ only in punctuation, and for this sample they
 # swap haplotype between them: GCA_018466835.1 is HG02257 PAT (2021) while
 # GCA_018466835.2 is HG02257 MAT (2024). A label fragment like "HG02257.pat" would be one
 # hub update away from matching the wrong assembly.
 #
 # The session is Gerardo's own, kept because what it holds -- a custom track plus a hub on
 # hg38 -- is what made the search return several kinds of hit.
+proof:
+  - "assertion-only 2026-09-05 -- written from the ticket after the fix had shipped"
+
 target: genome-test
 db: hg38
 reset: true
 fast: true
 steps:
   - loadSession: {user: Gerardo, name: quickLift_CT_hub}
   - expect: {noText: "Could not find session"}
 
   - convert: {to: GCA_018466835.1, quicklift: true}
   - open: lift
   - expect: {noText: "Sorry, couldn't locate"}
 
   - goShow: "brca2"
   - expect: {url: "/hgSearch", noText: "Sorry, couldn't locate"}
 
   # The first BRCA2 hit on the results page. Which of the several hits it is does not
   # matter much -- they all resolve to the same gene span -- but where it LANDS matters a
   # great deal, and that is what the next step reads.
   - click: 'a:has-text("BRCA2")'
 
   # This is the assertion the script exists for, and noText: is the weaker half of it.
   # The bug sent the click at the SOURCE coordinate, chr13:32,315,508-32,400,268, which the
   # target cannot resolve. Fixed, the click lands on the target's own contig:
   # JAGYVI010000002.1:12,922,586-13,007,328, measured 2026-09-05, and 84,742 bases wide
   # against BRCA2's 84,760 on hg38. Naming the source coordinate in noUrl: is what makes
   # this fail on the buggy build rather than merely pass on the fixed one.
   - expect:
       url: "position=JAGYVI010000002.1%3A12922586"
       noUrl: "chr13%3A32315508"
       noText: "Sorry, couldn't locate"
       rows: [mane, knownGene]