4697bbddd881c72cccb85b9ff0aacd769396b9d0
braney
  Thu Sep 10 07:34:35 2026 -0700
docent: record what evidence each regression test has, and count it

A regression test written after the fix asserts the right answer, but nobody
has watched it fail for the reason it exists, and a loose assertion in that
state is indistinguishable from no test at all. Four of the 37 scripts here
have actually been watched to flip. That was recorded only as prose in each
script's header, so answering "how many of these are real regression tests"
meant a grep and a read, and the number could not be quoted.

Every script now carries a top-level `proof:` key, one quoted line per piece
of evidence, `<level> <YYYY-MM-DD> -- <what was seen>`. docent.js reads only
the keys it names off the parsed document, so this costs a run nothing.

tests/proof.js reads them and tallies, wired up as `make proof` in the shared
docentTest.mk. It exits 1 on a malformed line, an unknown level, or a line
left unquoted -- that last one because nearly every note names a ticket and a
bare # in an unquoted YAML scalar silently truncates the sentence at the
ticket number, which is how the first pass of this change lost half its text.

The levels, weakest first: assertion-only, xfail, sandbox-ab, server-flip,
caught-regression. Today that reads 31 / 2 / 0 / 3 / 1.

nightly.sh now records the flips it finds. An xfail that PASSES is the best
evidence this suite produces -- the same server, the same fixtures, the same
script, one real build apart -- and until now it arrived as a red mail and was
thrown away with the log 60 days later. It is appended to
/hive/users/braney/docentNightly/flips.log, one line per script ever, outside
the checkout because --update resets the tree. The mail says what to do with
it. The three flips that already happened (rm38272 2026-09-06, rm36212
2026-09-09, rm38310 2026-09-10) were recovered from the old logs and seeded
there by hand.

Full suite run after the change: 37 scripts, all ok. refs #38252

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/docent/tests/regress/nightly.sh src/hg/utils/docent/tests/regress/nightly.sh
index c710d6e1ad0..2d1de12ae40 100755
--- src/hg/utils/docent/tests/regress/nightly.sh
+++ src/hg/utils/docent/tests/regress/nightly.sh
@@ -8,54 +8,65 @@
 #
 # Runs the tests that are COMMITTED, listed from git rather than from the directory.
 # The directory also holds work in progress -- scripts written against a ticket whose
 # recipe is not right yet -- and those must not mail a failure every night.  A newly
 # committed test is picked up with no edit here.
 #
 # It runs out of whatever checkout this script itself lives in, so a copy in a second
 # checkout needs no argument.  The cron runs it out of a clone of its own,
 #
 #     /hive/users/braney/docentNightly/kent
 #
 # the way catalogNightly has one under /hive/users/braney, so that an ordinary day's
 # editing in ~/kent cannot change what the cron measures.  Nothing here is built, so
 # the clone needs no submodules and no make.
 #
+# When an xfail PASSES -- the fix reached the server -- that flip is appended to
+# flips.log beside the logs, one line per script ever.  It is the only place this job's
+# best evidence is kept: the log it came from is deleted after 60 days, and a mail is
+# not a record.  See `make proof` in this directory for what the flips are for.
+#
 # --update brings that clone to origin/master and then re-runs this script from the
 # result.  Two reasons for the re-exec rather than a separate driver script beside the
 # crontab: the whole job stays in the tree where it can be reviewed and committed, and
 # a change to this file takes effect the same night as a change to a test, instead of
 # a night later.
 #
 # --update will only ever touch a checkout that is a pristine mirror of origin/master.
 # A working tree with uncommitted edits, or one holding a commit that has not been
 # pushed, is left alone and reported, because a `reset --hard` there would throw away
 # work.  That is what makes it safe for the flag to exist in a script that also sits in
 # a working tree.
 
 set -u
 export PATH=/usr/bin:/bin:/usr/local/bin:$PATH
 
 HERE=$(cd "$(dirname "$0")" && pwd)
 PW=/hive/groups/browser/uiTest/pw
 PW_ENV="env PLAYWRIGHT_BROWSERS_PATH=$PW/browsers NODE_PATH=$PW/node_modules"
 TO=${DOCENT_NIGHTLY_TO:-braney@ucsc.edu}
 LOGDIR=${DOCENT_NIGHTLY_LOGS:-/hive/users/braney/docentNightly/logs}
+FLIPS=${DOCENT_NIGHTLY_FLIPS:-/hive/users/braney/docentNightly/flips.log}
 STAMP=$(date +%Y-%m-%d_%H%M)
+# Only a label for the flips.log line.  Every script here says `target: genome-test`
+# itself; this is not read from them, so override it if that ever stops being true.
+TARGET=${DOCENT_NIGHTLY_TARGET:-genome-test}
 
 mkdir -p "$LOGDIR"
 OUT="$LOGDIR/$STAMP.txt"
+VERDICTS="$LOGDIR/.$STAMP.verdicts"
+trap 'rm -f "$VERDICTS"' EXIT
 
 # A job that could not start has to arrive looking like the others, or a night when
 # nothing ran reads as a quiet night.  Same subject shape, same log file, exit 0.
 bail() {
 {
     echo "Docent regression run, $(date)"
     echo "checkout: $HERE"
     echo
     echo "$@"
     echo
     echo "Nothing was tested.  This is a problem with this job, not with the browser."
 } > "$OUT" 2>&1
 mail -s "docent regression: BROKEN (0 ok) $(date +%F)" "$TO" < "$OUT"
 exit 0
 }
@@ -107,44 +118,81 @@
     echo "--- preflight (the sessions and hubs these tests depend on) ---"
     # Reported separately from the tests on purpose.  A session that has been deleted
     # or a hub that has moved is not a browser regression, and the two must not arrive
     # as the same red.
     #
     # preflight is given the committed list rather than left to scan the directory, so a
     # dead fixture belonging to a work-in-progress script is not reported as a problem
     # with a run that never included it.
     if (cd "$HERE" && $PW_ENV node ../preflight.js . $TESTS 2>&1); then
       pf=ok
     else
       pf=MISSING
     fi
     echo
     echo "--- tests ---"
-    if (cd "$HERE" && make test T="$TESTS" 2>&1); then
+    # Written to a file of its own and then echoed, rather than straight into the
+    # block's redirect, because the flip scan below has to read the verdicts back and
+    # $OUT is the file this block is being written to.
+    if (cd "$HERE" && make test T="$TESTS" > "$VERDICTS" 2>&1); then
       tests=pass
     else
       tests=FAIL
     fi
+    cat "$VERDICTS"
 
     if [ "$tests" = FAIL ] && [ "$pf" = MISSING ]; then
       subject_state="FAIL (fixtures missing too)"
     elif [ "$tests" = FAIL ]; then
       subject_state="FAIL"
     elif [ "$pf" = MISSING ]; then
       subject_state="pass, but a fixture is missing"
     else
       subject_state="pass"
     fi
   fi
 
+  # An xfail that PASSED is the one piece of evidence this job produces that cannot be
+  # got any other way: the same server, the same fixtures, the same script, one real
+  # build apart.  That is what separates a test that has been watched to fail for its
+  # own reason from a test that only asserts the answer -- see `make proof`.  Until now
+  # it arrived as a red mail and was then thrown away, so record it before the log ages
+  # out, and say what to do with it.
+  #
+  # The record lives OUTSIDE the checkout on purpose.  --update does `reset --hard`, so
+  # anything written into the tree here is gone the next night.
+  #
+  # One line per script, ever.  Without the dedupe this would append every night from
+  # the flip until the promotion commit lands, which is exactly the stretch when nobody
+  # is looking.
+  flipped=$(awk '/^=== /{n=$2} /supposed to fail, and it passed/{print n}' "$VERDICTS" 2>/dev/null)
+  if [ -n "$flipped" ]; then
+    echo
+    echo "--- flips (an xfail passed, so the fix reached this server) ---"
+    for t in $flipped; do
+      if [ -f "$FLIPS" ] && awk -v t="$t" '$2 == t { found = 1 } END { exit !found }' "$FLIPS"; then
+        echo "  $t -- already recorded in $FLIPS"
+      else
+        printf '%s %-22s passed on %s  commit=%s  log=%s\n' \
+          "$(date +%F)" "$t" "$TARGET" \
+          "$(git -C "$HERE" rev-parse --short HEAD 2>/dev/null)" "$STAMP.txt" >> "$FLIPS"
+        echo "  $t -- recorded in $FLIPS"
+      fi
+    done
+    echo
+    echo "  To close one out: drop the .xfail from the script's name, and add a"
+    echo "  server-flip line to its proof: key naming the commit that shipped the fix."
+    echo "  Then \`make proof\` counts it.  Until that commit is pushed this stays red."
+  fi
+
   echo
   echo "full log kept at $OUT"
 } > "$OUT" 2>&1
 
 # subject_state is set inside the block above, which runs in this shell, so it survives.
 n=$(grep -c '^  ok$' "$OUT" 2>/dev/null || echo 0)
 mail -s "docent regression: ${subject_state:-unknown} ($n ok) $(date +%F)" "$TO" < "$OUT"
 
 # Keep two months of logs and no more.
 find "$LOGDIR" -name '*.txt' -mtime +60 -delete 2>/dev/null
 
 exit 0