4697bbddd881c72cccb85b9ff0aacd769396b9d0 braney Thu Sep 10 07:34:35 2026 -0700 docent: record what evidence each regression test has, and count it A regression test written after the fix asserts the right answer, but nobody has watched it fail for the reason it exists, and a loose assertion in that state is indistinguishable from no test at all. Four of the 37 scripts here have actually been watched to flip. That was recorded only as prose in each script's header, so answering "how many of these are real regression tests" meant a grep and a read, and the number could not be quoted. Every script now carries a top-level `proof:` key, one quoted line per piece of evidence, ` -- `. docent.js reads only the keys it names off the parsed document, so this costs a run nothing. tests/proof.js reads them and tallies, wired up as `make proof` in the shared docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line left unquoted -- that last one because nearly every note names a ticket and a bare # in an unquoted YAML scalar silently truncates the sentence at the ticket number, which is how the first pass of this change lost half its text. The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip, caught-regression. Today that reads 31 / 2 / 0 / 3 / 1. nightly.sh now records the flips it finds. An xfail that PASSES is the best evidence this suite produces -- the same server, the same fixtures, the same script, one real build apart -- and until now it arrived as a red mail and was thrown away with the log 60 days later. It is appended to /hive/users/braney/docentNightly/flips.log, one line per script ever, outside the checkout because --update resets the tree. The mail says what to do with it. The three flips that already happened (rm38272 2026-09-06, rm36212 2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded there by hand. Full suite run after the change: 37 scripts, all ok. refs #38252 Co-Authored-By: Claude Opus 5 (1M context) diff --git src/hg/utils/docent/tests/regress/rm36540.xfail.docent.yaml src/hg/utils/docent/tests/regress/rm36540.xfail.docent.yaml index 94888a6ecf7..1a49ddfc2cd 100644 --- src/hg/utils/docent/tests/regress/rm36540.xfail.docent.yaml +++ src/hg/utils/docent/tests/regress/rm36540.xfail.docent.yaml @@ -1,32 +1,34 @@ # #36540 -- EXPECTED TO FAIL. The ticket is Closed; the symptom is not. # # Measured 2026-09-04 with the reporter's own hub URL, not just our copy: # # genome.ucsc.edu v502 chromInfo error present # hgwbeta.soe.ucsc.edu v503 chromInfo error present # genome-test v503 chromInfo error present # # The ticket's last note says "This is now on the RR", and the discussion on it is about # making the error message better rather than about making genome= resolve. Whatever # landed, a user who writes genome= instead of db= for a hub-backed assembly still gets # # Can't start query: select * from chromInfo where chrom = 'chr9' # mySQL error 1146: Table 'hs1.chromInfo' doesn't exist (db=hs1) # # So this cannot assert fixed behavior, because there is none to assert. It is kept as an # xfail instead, which is how this suite pins a trap: the run fails if it ever PASSES, so # the day someone fixes it, they are told there is a test waiting rather than left to # discover it. If #36540 is reopened and fixed, drop the .xfail from the name. -# +proof: + - "xfail 2026-09-04 -- red on genome.ucsc.edu v502, hgwbeta v503 and genome-test v503 with the reporter's own hub URL; genome= still does not resolve" + target: genome-test reset: true fast: true db: hs1 steps: - goto: "/cgi-bin/hgTracks?genome=hs1&hubUrl=https://hgwdev.gi.ucsc.edu/~braney/docentFixtures/NOTCH2NLHub/hub.txt&position=chr9%3A1-1000000&pix=1100" # "chromInfo" alone is the assertion, not the whole quoted string: the message names the # assembly, and the point is that no query against a chromInfo table should reach the # user whichever assembly it names. Paired with a row check because the early-error page # this bug produced draws no image, so noText on its own would have passed on it. - expect: {rows: [ruler], noText: "chromInfo"}