4697bbddd881c72cccb85b9ff0aacd769396b9d0 braney Thu Sep 10 07:34:35 2026 -0700 docent: record what evidence each regression test has, and count it A regression test written after the fix asserts the right answer, but nobody has watched it fail for the reason it exists, and a loose assertion in that state is indistinguishable from no test at all. Four of the 37 scripts here have actually been watched to flip. That was recorded only as prose in each script's header, so answering "how many of these are real regression tests" meant a grep and a read, and the number could not be quoted. Every script now carries a top-level `proof:` key, one quoted line per piece of evidence, ` -- `. docent.js reads only the keys it names off the parsed document, so this costs a run nothing. tests/proof.js reads them and tallies, wired up as `make proof` in the shared docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line left unquoted -- that last one because nearly every note names a ticket and a bare # in an unquoted YAML scalar silently truncates the sentence at the ticket number, which is how the first pass of this change lost half its text. The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip, caught-regression. Today that reads 31 / 2 / 0 / 3 / 1. nightly.sh now records the flips it finds. An xfail that PASSES is the best evidence this suite produces -- the same server, the same fixtures, the same script, one real build apart -- and until now it arrived as a red mail and was thrown away with the log 60 days later. It is appended to /hive/users/braney/docentNightly/flips.log, one line per script ever, outside the checkout because --update resets the tree. The mail says what to do with it. The three flips that already happened (rm38272 2026-09-06, rm36212 2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded there by hand. Full suite run after the change: 37 scripts, all ok. refs #38252 Co-Authored-By: Claude Opus 5 (1M context) diff --git src/hg/utils/docent/tests/regress/rm35580.docent.yaml src/hg/utils/docent/tests/regress/rm35580.docent.yaml index 293b3c98dc6..c92467a16f8 100644 --- src/hg/utils/docent/tests/regress/rm35580.docent.yaml +++ src/hg/utils/docent/tests/regress/rm35580.docent.yaml @@ -1,46 +1,49 @@ # #35580 -- "Show placed on its chromosome" errored instead of drawing the view. # # The reporter was at chr16_KI270853v1_alt:744117-744278, clicked the GRC Patches item, # and on the hgc page clicked "Show chr16_KI270853v1_alt placed on its chromosome". That # link puts hgTracks in singleAltHaplo mode, and it came back with # # unexpected current window 55471440, expected first window 55471712 # # The first attempt at a fix had to be reverted on the v487 branch (#36311, a build patch) # because it broke the UCSC Unusual Regions track in multi-region mode. The fix that # actually shipped is in v488, so this asserts the behavior of that one, not the reverted # one. # # The session is kept. Gerardo/RM_35326_bug is on genome-test, and what it contributes is # hard to build in steps: a position on an alt contig, with the GRC Patches track showing # and the four recount3 tracks on -- the ticket's own title asks whether recount3 is # involved, so the assertion below names two of them and would catch the view coming back # without them. # # The item on altSeqLiftOverPsl is called "chr16", which reads oddly next to the alt-contig # position but is what hgTracks puts in the map box: the item names the chromosome the alt # belongs to. +proof: + - "assertion-only 2026-09-05 -- written from the ticket after the fix had shipped" + target: genome-test db: hg38 reset: true fast: true steps: - loadSession: {user: Gerardo, name: RM_35326_bug} # A missing session is not an error -- hgTracks answers 200 with an early-error page and # every noText check on it passes -- so both halves are needed here. - expect: {noText: "Could not find session", rows: [altSeqLiftOverPsl]} - click: {track: altSeqLiftOverPsl, item: chr16} - expect: {text: "Show chr16_KI270853v1_alt placed on its chromosome"} - click: 'a:has-text("placed on its chromosome")' # The row list is what makes this tight. The bug produced an error page with no track # image at all, so noText: on its own would have passed on it; and asking for the # recount3 rows checks the tracks that #36311's revert was about are still drawn in # singleAltHaplo mode. - expect: rows: [altSeqLiftOverPsl, recount3_gtex, recount3_tcga] noText: "unexpected current window" url: "virtModeType=singleAltHaplo"