756582322f8a53fef19e8c24ba902e353ba3623f
chmalee
  Thu Aug 27 14:38:34 2026 -0700
uiTest: shared browser UI test harness, plus an hgTracks example, refs #38188

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/uiTest/README.md src/hg/utils/uiTest/README.md
new file mode 100644
index 00000000000..df75b6fedb6
--- /dev/null
+++ src/hg/utils/uiTest/README.md
@@ -0,0 +1,189 @@
+# uiTest -- browser tests for the Genome Browser CGIs
+
+One harness, so browser testing stops being three private efforts that share no
+login, no target list and no way of being run.
+
+    cd src/hg/hgTracks/tests
+    make uiTest
+
+That works with no setup and no conf file: it runs ten checks against
+genome-test and tells you what it found. Everything below is for when you want
+more than that.
+
+## What this is
+
+`uiTest` runs the tests in one CGI's `tests/` directory. It dispatches by
+extension and merges both engines into one result:
+
+| File | Engine |
+|---|---|
+| `t*.js` | the JavaScript runner in `lib/run.js` |
+| `*.docent.yaml` | `docent.js`, run as a subprocess |
+
+Docent (`src/hg/utils/docent/`, RM #37892) is not modified by any of this. The
+integration is its exit code, which is why the two can share a directory without
+either one having to know about the other.
+
+Which to write:
+
+> **"Go somewhere, turn tracks on and off, assert what is drawn"** -- write a
+> `.docent.yaml`.
+>
+> **Needs a login, files, the database, JavaScript internals, an HTTP status, a
+> timing number, or a comparison between two page loads** -- write a `t*.js`.
+
+## Running
+
+    make uiTest                       # everything in this directory
+    make uiTest T=t01                 # one file
+    make uiTest G='hide all'          # one check
+    make uiTest TARGET=hgwdev-$USER   # against your sandbox
+    make headed                       # show the browser, slowly
+    make selfcheck                    # check the setup, run nothing
+    make lint                         # the waitForTimeout policy
+
+`uiTest --help` lists every flag. `make selfcheck` first when something looks
+wrong: it separates "the setup is broken" from "the feature is broken", which is
+the distinction the exit codes exist to preserve.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| 0 | everything that ran passed (skips allowed) |
+| 1 | a check failed -- a bug in the thing under test |
+| 2 | usage or configuration error (bad conf permissions, unknown target, a write aimed at production) |
+| 3 | infrastructure (no browser, server unreachable, login broken) |
+| 4 | everything was skipped and `--strict` was given |
+
+**Skipped never folds into passed.** The summary reads `10 passed, 0 failed, 2
+skipped`, and a suite that quietly skipped everything must not read as green. An
+`*.xfail.docent.yaml` that passes is a failure, the same rule docent applies.
+
+## The conf file
+
+`~/.hg.uiTest.conf`, mode 600. **Optional** -- with no conf file at all, uiTest
+runs against genome-test and skips anything that needs a login or the database.
+Copy `sample.hg.uiTest.conf` and edit it when you want your own sandbox or a
+login.
+
+Same format as `~/.hg.conf`: `name=value`, `#` comments, `include` and `delete`.
+It is a separate file on purpose: `~/.hg.conf` is read by every CGI and by
+`hgsql`, and test credentials do not belong in that blast radius. A file whose
+name starts with `.` and that allows group or other access is rejected before
+anything runs -- the same rule `checkConfigPerms()` in `hg/lib/hgConfig.c`
+applies.
+
+Precedence for every key: **CLI flag > environment > conf file > compiled
+default.** Environment names are `UITEST_TARGET`, `UITEST_ACCOUNT`,
+`UITEST_CONF`, `UITEST_ARTIFACTS`, `UITEST_HEADED` and `UITEST_PW_PREFIX`.
+
+### Targets
+
+`rr`, `genome-test`, `hgwdev`, `hgwbeta` and `hgwdev-<user>` are compiled in, so
+they work with no conf file. A `target.<name>` line adds more -- the docker QA
+instances, say. A full `http(s)` URL works as a target too. An unknown target is
+a configuration error rather than a silently-tried hostname.
+
+### Test accounts: one per person
+
+Make yourself an hgLogin account named `<user>Qa` and name it in your own conf.
+This is a safety property, not a preference. A run logs in as you, and a hub or
+a session belongs to whoever's account made it, so a run of yours cannot reach a
+hub or a session somebody else was in the middle of. A shared account would let
+one person's run reach another's work.
+
+A test that declares `needs.login` and finds no account is **skipped with a
+reason**, never failed. A developer with no QA password must not see a red run.
+
+## Playwright
+
+One shared, pinned install at `/hive/groups/browser/uiTest/pw`. Braney owns it;
+its `README.md` there carries the pin and the recipe for moving it. Docent points
+at the same directory, so there is one Playwright on this machine and not one per
+person.
+
+`lib/pw.js` finds it, in this order:
+
+1. `UITEST_PW_PREFIX`
+2. `pw.prefix` in your conf
+3. `/hive/groups/browser/uiTest/pw`
+4. `$HOME/.uiTestPw`
+5. a bare `require('playwright')`
+
+**The /tmp trap.** `NODE_PATH` loses to any `node_modules` directory in an
+ancestor of the working directory, and hgwdev has a stale
+`/tmp/node_modules/playwright` owned by someone else. Anything run with
+`NODE_PATH` set and a working directory under `/tmp` loads that instead and dies.
+uiTest loads Playwright by absolute path rather than through `NODE_PATH`, and an
+absolute-path require does not consult ancestor `node_modules` at all, so a run
+works from any working directory whatever `NODE_PATH` says. Docent does use a
+bare `require`, so when uiTest runs a docent script it sets `NODE_PATH` for that
+subprocess and gives it an explicit working directory that is not `/tmp`.
+
+`tests/t03-pw.js` pins this down: it moves its own working directory into a tree
+carrying exactly that decoy and checks which copy gets loaded.
+
+If the installed version does not match the pin, uiTest **warns and carries on**,
+and records both in `results.json`. Version skew should be visible, not a wall.
+
+## What a run leaves behind
+
+    <artifacts>/<target>/<YYYYMMDD-HHMMSS>-<cgi>/
+        results.json     machine readable, and the source of truth
+        run.log
+        shots/NN-<check>.png
+        trace/<test>.zip
+        <script>.docent.log
+
+`results.json` is schema-versioned and carries the target, the account, the git
+commit, the node and Playwright versions, the pin, totals by status, and for each
+check its name, status, milliseconds, the authored failure message, **the URL
+that produced it**, and the paths to its screenshot and trace.
+
+A trace is written only for a test file that failed, so tracing is free on a
+green run. When one does fail:
+
+    npx playwright show-trace <artifacts>/.../trace/t01-render.zip
+
+That replays the failed run with DOM snapshots, network and console. It is the
+most useful thing in this directory and nobody finds it on their own.
+
+There is no HTML report yet, on purpose. `results.json` is written from the
+start, so a report is a pure add-on later that reads that file and can never
+disagree with it -- and by then we will know from use what it should contain.
+
+## Where things live
+
+| | |
+|---|---|
+| `lib/env.js` | the conf file, targets, accounts, what this machine may do. No playwright, no CGI names, no network |
+| `lib/pw.js` | finding Playwright |
+| `lib/browser.js` | launching, contexts, tracing, screenshots. Could drive any website |
+| `lib/site.js` | hgLogin, the login cache, CGI URLs, the always-on error assertion, reading the page's own JavaScript |
+| `lib/wait.js` | the anti-flake helpers |
+| `lib/run.js` | checks, skips, xfail, results |
+| `lib/pages.js` | finds a CGI's page object |
+| `uiTest.mk` | the make include a CGI's `tests/makefile` picks up in two lines |
+| `tests/` | the harness's own tests. No browser, no network -- `make test` here really runs |
+
+A CGI's tests live in `src/hg/<cgi>/tests/`, never here. `src/hg/hgTracks/tests/`
+is the worked example; copy it.
+
+Those `tests/` directories are **not** swept by the tree-wide `make test`.
+`src/hg/makefile`'s `testAll` covers `APPS`, and the CGIs are in `BROWSER_BINS`,
+so nothing reaches them today. `uiTest.mk` supplies an inert `test:` target
+anyway, so the day a CGI joins `APPS`, `make test` stays green on a machine with
+no network instead of trying to open a browser.
+
+## Flake
+
+`page.waitForTimeout` is banned in `lib/` and in `pages/`, and in a test it must
+carry a `// flake:` comment naming what it waits for and why no condition exists.
+`make lint` enforces that. Use `lib/wait.js` instead: `until`, `eventually`,
+`noPending`, `navigated`, `stable`.
+
+**The framework does not retry a check and does not retry a suite.** A flaky
+check is a bug in the check or in the CGI, and an automatic retry hides both.
+With a person at the terminal the right answer to a flake is `make headed` and a
+look.