23790efe33021ccd1691bf5f4262c1396a1f6e9d braney Fri May 1 12:49:13 2026 -0700 quickLiftBench: phase_asserts mechanism + parallel-fetch regression case phase_asserts is a per-case schema in cases.yaml that declares regex matches against the per-iteration timing spans, with optional required/min_median_ms/max_median_ms bounds. When a case declares phase_asserts, the runner captures phase data automatically and runs the asserts after iterations complete; any failure prints to stderr and exits non-zero. The new regress_quickLift_parallel case uses this to assert that the "Waiting for parallel..." span fires for the lifted multi-track session Brianraney/benchQuickPara with a median between 500 and 15000 ms -- discriminating the working sandbox build from a pre-fix hgwdev where only non-quickLift tracks parallelize at ~50 ms. refs #37488, #37470 diff --git src/utils/qa/quickLiftBench/README.md src/utils/qa/quickLiftBench/README.md index a0230bc9a7b..53aedd3951c 100644 --- src/utils/qa/quickLiftBench/README.md +++ src/utils/qa/quickLiftBench/README.md @@ -121,30 +121,63 @@ Two TSVs are written to `results/<YYYYMMDD-HHMMSS>/`: - `results.tsv` — one row per (case, variant, iteration) with http_ms, load_ms_sum, draw_ms_sum, n_tracks, total_ms, status_code, error. - `summary.tsv` — two sections: 1. per (case, variant): n, n_ok, http/load_sum/draw_sum/total median and p90. 2. per (case, compare-pair): left vs right total medians and the `right/left` ratio for each metric. - `phases.tsv` (only with `--phases`) — long-form rows of every `<span class='timing'>label: NNN millis</span>` marker emitted by hgTracks (chromAliasSetup, trackDbLoad, parallel data fetch, image generation, cart write, etc.), one row per (case, variant, iteration, phase). A per-(case, variant, phase) median+p90 summary is appended. Useful for localizing where time is going when total medians differ. +## Regression assertions: `phase_asserts` + +A case can declare assertions against the per-iteration phase timings, so +the bench acts as a tripwire for regressions instead of just emitting +numbers. When any case declares `phase_asserts`, that case's phase data is +captured automatically (no `--phases` flag needed) and assertions run after +all iterations complete. A failure prints to stderr and the script exits +non-zero. + +```yaml +- id: regress_my_thing + server: hgwdev + variants: + base: User/sessionName + phase_asserts: + - variant: base + phase: 'Waiting for parallel \(\d+ threads for \d+ tracks\) remote data fetch' + required: true # span must appear in every iteration + max_median_ms: 15000 # optional median upper bound + min_median_ms: 1 # optional median lower bound +``` + +Semantics: + +- `phase` is a Python regex matched against each phase label (the part + before `:` in `<span class='timing'>label: NNN millis</span>`). +- `required: true` (default) — assert fails if the regex matches no phase + in any iteration of the variant. +- `max_median_ms` / `min_median_ms` — optional bounds on the median across + iterations. Per iteration, all matching phases' ms values are summed, + then the per-iteration sums are reduced via median. +- A FAIL prints `[FAIL] case/variant /pattern/ reason` and `sys.exit(1)`. + A short pairwise table is also printed to stderr at the end of a run. ## Dependencies ``` pip install requests pyyaml ``` ## Notes - The script does not parallelize requests against a single server. quickLift renders are single-threaded per request; parallel requests would measure contention rather than work. - If hgTracks returns the bot-block page or an `errAbort`, the row is written with `error` set and `*_ms` empty rather than aborting the run.