4697bbddd881c72cccb85b9ff0aacd769396b9d0 braney Thu Sep 10 07:34:35 2026 -0700 docent: record what evidence each regression test has, and count it A regression test written after the fix asserts the right answer, but nobody has watched it fail for the reason it exists, and a loose assertion in that state is indistinguishable from no test at all. Four of the 37 scripts here have actually been watched to flip. That was recorded only as prose in each script's header, so answering "how many of these are real regression tests" meant a grep and a read, and the number could not be quoted. Every script now carries a top-level `proof:` key, one quoted line per piece of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only the keys it names off the parsed document, so this costs a run nothing. tests/proof.js reads them and tallies, wired up as `make proof` in the shared docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line left unquoted -- that last one because nearly every note names a ticket and a bare # in an unquoted YAML scalar silently truncates the sentence at the ticket number, which is how the first pass of this change lost half its text. The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip, caught-regression. Today that reads 31 / 2 / 0 / 3 / 1. nightly.sh now records the flips it finds. An xfail that PASSES is the best evidence this suite produces -- the same server, the same fixtures, the same script, one real build apart -- and until now it arrived as a red mail and was thrown away with the log 60 days later. It is appended to /hive/users/braney/docentNightly/flips.log, one line per script ever, outside the checkout because --update resets the tree. The mail says what to do with it. The three flips that already happened (rm38272 2026-09-06, rm36212 2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded there by hand. Full suite run after the change: 37 scripts, all ok. refs #38252 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> diff --git src/hg/utils/docent/tests/regress/rm35920.docent.yaml src/hg/utils/docent/tests/regress/rm35920.docent.yaml index da6921fd2a4..08a43159b9f 100644 --- src/hg/utils/docent/tests/regress/rm35920.docent.yaml +++ src/hg/utils/docent/tests/regress/rm35920.docent.yaml @@ -1,75 +1,78 @@ # #35920 -- a hub declaring the wrong bigBed type crashed hgTracks or drew garbage item # names, and gave no warning until an item of the track came into view. What the reader # then saw was a "jsEmbedded" dialog titled with the server's hostname, which says nothing # about the real problem. Fixed by #19984 and closed by QA on 2025-12-02. # # The hub has one track for each half of the problem: rm35920Ultras says `type bigBed 12 +` # over a file with 4 fields, rm35920UltraZoos says `type bigBed 4 +` over a file with 3. # # The hub is our own copy of the reporter's, at # ~/public_html/docentFixtures/Auto-generated_hub/, with bigDataUrl pointing at the local # files so nothing outside this repository can change what the test loads. hubCheck # reports the two type mismatches and a missing description page, which is correct: those # are what the hub is for. It is the crash and the garbage labels that must be gone, not # the malformed hub. # # --- 2026-09-09, rewritten. The script this replaces was never testing the hub. --- # # The reporter called his two tracks `ultras` and `ultraZoos`, and so did our copy. hg38 has # its OWN `ultras`, under the unusualcons superTrack. A track name resolves to `img_data_ # <name>` first and only then to a hub row's `hub_<n>_<name>`, so the exact native id won # every time: `track: {ultras: pack}` turned on the native track (and its superTrack), and # `mouseover: {track: ultras}` read the native row. The native track's items are named uc.N # as well, so the old `expect: {tip: "uc.1"}` passed on the native data and nothing warned. # The nightly went red on 2026-09-08 only because the native row's tooltip stopped matching. # The fixture's tracks now carry an rm35920 prefix. A hub fixture must never name a track # anything a native assembly might also call it. # # With the collision gone, what the two malformed tracks actually do on genome-test, hgwbeta # and the RR alike (checked all three on 2026-09-09): # # rm35920Ultras bigBed 12 + over 4 fields -- row drawn, EMPTY. No crash, no dialog, # no garbage. The declared field count has a number, so #19984's # auto-detection does not apply; hgTracks cannot read the rows as # declared and drops them. # rm35920UltraZoos bigBed 4 + over 3 fields -- items drawn, with an EMPTY name. The hgc # link carries `i=` with nothing after it, which is what a correctly # declared `type bigBed 3` over the same file produces (measured against # a probe hub). That is the garbage string gone. # # So the tooltip assertion the old script carried cannot be restored: the hub track has no # named item to hover, on any server. What the ticket asked for is the crash and the garbage # names, and that is what is asserted below. # # The window holds items from both files: ultraZoos has three (chr1:10,533,682-10,533,851) # and ultras.bb has uc.1 (chr1:10,537,639-10,537,846), because the ticket is explicit that # nothing goes wrong until an item is in view. # # The hub declares `visibility pack` itself, so there is no `track:` step here: a hub track's # cart name carries a per-run hub_<n>_ prefix that `track:` cannot write, and the step that # used to be here only ever moved native tracks. +proof: + - "assertion-only 2026-09-04 -- written from the ticket after the fix had shipped" + target: genome-test db: hg38 position: chr1:10533000-10538500 reset: true fast: true steps: - hub: {url: "https://hgwdev.gi.ucsc.edu/~braney/docentFixtures/Auto-generated_hub/hub.txt", db: hg38} - go: chr1:10533000-10538500 # The crash half. Both malformed tracks have to reach the page as rows: the failure # produced a dialog and no image at all, and jsEmbedded is the string from that dialog. - expect: rows: [rm35920Ultras, rm35920UltraZoos] noText: "jsEmbedded" # The garbage-name half. The ticket says the bad string appeared when you clicked an item, # so click one. `at:` rather than `item:` because the fixed behavior leaves these boxes # nameless -- there is no name to click by, which is the point. The click has to land on a # real hgc detail page for this track (title "ultras", the hub track's longLabel) and not # on the crash dialog. - click: {track: rm35920UltraZoos, at: "chr1:10533692"} - expect: url: "hgc" text: "ultras" noText: "jsEmbedded"