4697bbddd881c72cccb85b9ff0aacd769396b9d0
braney
  Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it

A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.

Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.

tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.

The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.

nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.

Full suite run after the change: 37 scripts, all ok. refs #38252

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/docent/tests/regress/rm36387.docent.yaml src/hg/utils/docent/tests/regress/rm36387.docent.yaml
index a7b1aad988e..a93c868ecc3 100644
--- src/hg/utils/docent/tests/regress/rm36387.docent.yaml
+++ src/hg/utils/docent/tests/regress/rm36387.docent.yaml
@@ -1,47 +1,50 @@
 # #36387 -- an HGVS term pasted into the position box crashed the browser at once.
 #
 # The term the Stanford reader sent, and the whole of the bug, is that it carries a
 # ZERO-WIDTH SPACE (U+200B) between "):" and "n.46G>A". Pasting from a clinical report is
 # how it got there. hgTracks' search handler passed the raw term to encodeURIComponent, so
 # the zero-width character reached hgSuggest, which put it in a LIKE and died:
 #
 #     mySQL error 1267: Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and
 #     (utf8mb4_general_ci,COERCIBLE) for operation 'like'
 #
 # From hgTracks that came back as an Error 500. Fixed in the JS by stripping zero-width
 # characters before the term is encoded (#36412 is the build patch that carried it).
 #
 # So the term below MUST keep its \u200b escape. Without it there is no bug and the test
 # passes on any build ever shipped.
 #
 # What the fixed behavior actually is, measured on genome-test 2026-09-05: the position box
 # submits to hgSearch, which reports no results for the term. That is right and is what
 # Christopher Lee described on the ticket -- an HGVS term is not something hgSearch
 # resolves, and the bug was never that it should. An earlier version of this test waited
 # for #imgTbl and timed out, because it assumed a fix meant landing on the variant.
 #
 # The assertion that carries the weight is noUrl. The stripped character is invisible in
 # the rendered page, so the only evidence that the fix is in place is that %E2%80%8B did
 # not survive into the search URL. text: is the companion half: on the buggy build hgTracks
 # returned 500 and the search page never rendered at all.
+proof:
+  - "assertion-only 2026-09-05 -- written from the ticket after the fix had shipped"
+
 target: genome-test
 db: hg38
 position: chr12:120291800-120291900
 reset: true
 fast: true
 steps:
   - go: chr12:120291800-120291900
   - hide: all
   - track: {mane: pack}
   - expect: {rows: [mane], noText: "Warning/Error"}
 
   # goShow: rather than go:, because the position box is where the term was typed and
   # where the crash happened. The zero-width space is written as an escape so it survives
   # a copy and is visible to the next reader.
   - goShow: "RNU4-2 (NR_003137.3):\u200bn.46G>A"
 
   - expect:
       url: "/hgSearch"
       noUrl: "%E2%80%8B"
       text: "Search across the Genome Browser site"
       noText: "Internal Server Error"