5d76e853fa84baa6c4a05880d3f3b985a5e51878 braney Sat Aug 8 13:22:15 2026 -0700 docent: add session, loadSession and expect verbs, refs #37892 session: writes the whole cart at that step to sessions/<base>/<name>.txt, in the format hgSession's save-to-a-local-file button produces. No login is involved. With sessionUrlBase: set, the run also prints the load URL, so a tour hands back the state it built instead of only a picture of it. A file rather than a live hgsid link, because the hgsid cart keeps changing as the tour runs. loadSession: is the other direction: a settings file by URL, a share link, a named session, or a local file written by an earlier session:. The load always goes to the server named in target:, since named sessions are per-server and every later step navigates to target: by absolute URL. expect: is the only verb that can fail a run. It checks the rows drawn, the image height, the tooltip now up, and the page text, then throws, so make stops at that step rather than writing a wrong figure over a right one. A session taken on a lifted view is short-lived: it names the quickLift hub by its trash path, and re-pointing it at a durable copy does not work today. See #38046. The README says so rather than working around it. diff --git src/hg/utils/docent/README.md src/hg/utils/docent/README.md index 2b1f3df22b2..b8a475ba67b 100644 --- src/hg/utils/docent/README.md +++ src/hg/utils/docent/README.md @@ -1,75 +1,80 @@ # Docent — a language for authoring guided tours of the Genome Browser A **docent** leads a tour and explains what you are looking at. A Docent script is that tour, written down: an ordered list of high-level verbs (`go`, `hide`, `track`, `mouseover`, `convert`, `drag`, `shot`) that drive a real browser against a real server. -`docent.js` renders one script into two things at once: +`docent.js` renders one script into three things at once: -- a **silent mp4** of the whole tour, and -- a named **PNG still** at every `shot:` marker. +- a **silent mp4** of the whole tour, +- a named **PNG still** at every `shot:` marker, and +- a loadable **session file** at every `session:` marker. So a published figure is literally a frame of the tour, and the two can never drift -apart. The surface syntax is YAML — so ordinary editors highlight it and no one has to +apart. A session file goes further: it hands back the state itself, so a reader can open +the view the figure was taken from instead of only looking at it. The surface syntax is YAML — so ordinary editors highlight it and no one has to learn a new parser — but the language is the verb vocabulary layered on top, not the serialization. Scripts are named `<base>.docent.yaml` (a bare `<base>.docent` works too). The verbs deliberately encode *browser mechanics* rather than selectors: `convert:` knows that the Hide-defaults checkbox reverts when the Assembly menu reloads, `track: {clinvar: pack}` asks trackDb which containers and checkboxes that implies, and `mouseover:` knows that a lifted track's DOM id gains a per-run `hub_<n>_` prefix. The author writes intent; the renderer deals with the UI. ## Run ``` PLAYWRIGHT_BROWSERS_PATH=~/pwrec/browsers NODE_PATH=~/pwrec/node_modules \ node docent.js AP1.docent.yaml ``` Needs `playwright`, `js-yaml`, and `ffmpeg`. At UCSC these live in a shared install at `~braney/pwrec` (`pwrec/browsers` for Chromium, `pwrec/node_modules` for the modules) — point the two variables above anywhere you have them. Outputs, relative to the script's own directory: - mp4 → `../<base>.mp4` (override with `mp4:` or a second argument) - stills → `stills/<base>/<name>.png` (override with `stills:`) +- sessions → `sessions/<base>/<name>.txt` (override the parent with `sessions:`) `docent.mk` in this directory has the make rules — include it from a project that keeps a set of scripts and it rebuilds only the ones whose source changed. See the usage comment at the top of that file. ## Top of file (all optional) ```yaml target: genome-test # or rr, hgwdev, hgwbeta, hgwdev-<user>, or a full https://.../cgi-bin db: hg38 # source assembly position: chr7:155.8M # starting position (tracked from then on; `go:` sets it too) reset: true # cartReset first (clean cart + fresh quickLift hub) size: [1000, 760] # viewport pix: 850 # browser image width pace: 1.2 # seconds to dwell after each step shotHold: 2.2 # extra seconds the video pauses at a shot trackAnim: false # skip the visible dropdown gesture on `track:` (nav only) fast: false # true = figures only: no dwells, no cursor animation, no mp4 scale: 1 # >1 = print-resolution stills (see Print resolution) pinMouseovers: false # record every mouseover for `pinShot:` (see below) mp4: ../myname.mp4 # override output paths if you want stills: stills/AP1 +sessions: sessions # PARENT of the session dirs; files land in <parent>/<base>/ +sessionUrlBase: https://hgwdev-you.gi.ucsc.edu/~you/docent/sessions # see Sessions ``` `target:` defaults to genome-test, so a script that forgets to say where it runs will not quietly hit someone's personal sandbox. ## Print resolution A screen still is about 120 dpi across a journal's 7-inch column — fine on a monitor, too coarse to print. `scale:` (or `DOCENT_SCALE=k`, or `make hires`) renders **the same tour with k times the pixels**, which is not the same thing as enlarging the stills afterwards: a still only ever has the pixels it was drawn with, so every layer is asked to draw more of them. ``` make hires # every scenario at 3x -> stills.hires/<base>/ @@ -141,35 +146,152 @@ | `track: {anyTrackName: dense}` | Any track by its **trackDb** name. A name trackDb doesn't have (hub, custom track, quickLift target) is sent literally as `name=mode`. | | `mouseover: {track: dbSnp155Common, item: rs28406051}` | Hover a **named** item to raise its tooltip. `item:` matches the item's map box (its `&i=<name>` HREF or its TITLE) and hovers that rectangle's center — so it lands on the **correct row** even when items are stacked, which a bare coordinate can't do. It then waits for **that item's own tooltip**, not merely for a tooltip: the browser shows one 500 ms after `mouseenter` and hides it 500 ms after `mouseleave` (`hg/js/utils.js`), so the cursor gliding in crosses other items and one of THEIR tooltips can still be on screen on arrival — which is how an Alignment Differences mismatch got pinned as its neighbour's "identical". The expected text comes from the item's own map box, rendered the way the tooltip renders it (the attribute holds markup and undecoded entities), and `DOCENT_ROWS=1` warns if it never appears. `title: "SHH"` matches the TITLE text only; `value: "..."` matches a JSON `mouseOver` span (wig/dense tracks). | | `mouseover: {track: mane, at: chr7:155805900}` | Hover by **position** when you don't need a specific item: genomic coord (`at:`), a fraction across the view (`frac: 0.5`), or a raw pixel (`x: 400`); the y is forced to the middle of that track's row (cannot disambiguate stacked items). Optional on any `mouseover`: `hold: 2.5` (seconds to dwell) and `shot: tip_mane` (capture the image **plus** the tooltip in one still). | | `mouseover: {track: dbSnp155Common, item: rs28406051, pin: true}` | `pin: true` **records** that tooltip (its text + position) so a later `pinShot:` can show several mouseovers open together in one figure. Nothing is added to the recorded page, so the **mp4 is unaffected** (it still shows only the transient native tooltip). Set `pinMouseovers: true` at the top of the file to record every mouseover by default (`pin: false` opts one out). Records accumulate within a view and are cleared on the next nav. | | `pinShot: all_tips` | Write `<name>.png` with **all recorded (pinned) tooltips open at once**, each with a **cursor drawn at the point it was raised from** — so the figure says which feature every tooltip belongs to instead of leaving the reader to infer it from the anchor. Rendered on a throwaway page that shares the session (same cart/view) — never on the recorded page — so it never appears in the mp4. Consumes the recorded set (clears it). Place it after the `mouseover` steps whose tooltips you want shown together, before any nav/zoom. Map form `pinShot: {name: all_tips, cursors: false}` drops the pointers. | | `convert: {to: GCA_018466845.2, quicklift: true, hideDefaults: true}` | View→Convert, then **type the target into the page's own "Search for target genome" bar** and click the suggestion (`search:` overrides what is typed, `pick:` disambiguates the menu); the Assembly dropdown is checked afterwards and only opened by hand if the search didn't land there. QuickLift on, **re-checks Hide-defaults** (it reverts when the Assembly menu reloads), Submit. `to:` accepts an accession or a label fragment (`2257.pat`, matched against the dropdown text). | | `convert: {to: ..., shot: convert_filled}` | `shot:` inside `convert:` captures the Convert page, which no other verb can reach. A bare name is the **filled-in page just before Submit**. The map form names up to three moments: `shot: {opened: a, filled: b, result: c}` — `opened` as the page comes up, `filled` ready to Submit, `result` the conversion-result page (whose coordinate link `open: lift` clicks). These are viewport stills, so they show the page from the top. | | `hub: https://example.org/hub.txt` | **Quick, silent** attach of a track hub by URL (`hgTracks?hubUrl=...`): connects the hub so its tracks are available at their hub-declared visibility. Follow with `track:` to turn specific ones on. Map form `hub: {url: ..., db: hg38, position: chr7:...}` overrides the db/position (default: current `db` + last position). | | `addHub: https://example.org/hub.txt` | **Demonstrates the attach through the UI** (for the figure/video): opens My Data → Track Hubs, clicks the **Connected Hubs** tab, types the URL into the box, and clicks **Add Hub** — cursor glides and the URL is typed on screen. Then, on the "Hub Connect Successful" page, it **clicks the `Open:` link for `db`** so the demo ends on the browser with the hub loaded. Map form `addHub: {url: ..., db: hg38, shot: loaded}` sets which assembly to open and captures the still on that tracks view. Use `hub:` instead when you just need the hub attached without showing the steps. | | `addCustomTrack: <text-or-url>` | **Demonstrates loading a custom track via the UI**: opens My Data → Custom Tracks, types the track data (or a data URL) into the paste box, clicks **Submit**, then clicks through to the browser (**Go to first annotation**). Bare string is the data or URL; map form `addCustomTrack: {data: "track ...\nchr7 ...", db: hg38, goto: first, shot: loaded}` (use `url:` for a data URL, `goto: current` to land on **Return to current position** instead). Data is inserted literally (tabs/newlines preserved). In YAML, a multi-line track uses a block scalar: `addCustomTrack: |` then the indented lines. | | `addPublicHub: GTEx` | **Demonstrates connecting a PUBLIC hub via the UI**: opens My Data → Track Hubs, the **Public Hubs** tab, types the search terms, clicks **Search Public Hubs**, then clicks **Connect** on the matching hub row, and finally **clicks the `Open:` link for `db`** to land on the browser. Bare string is the search term (also used to match the row). A search usually returns several hubs, so use the map form `addPublicHub: {search: "GTEx", match: "GTEx Analysis Hub", db: hg38, shot: loaded}` to pick the exact hub by a substring of its row text (`match:` defaults to `search:`) and the assembly to open. If no row matches it **won't connect** (warns and stops) rather than pick the wrong hub. | | `drag: chr7:155,805,900-155,806,950` | Emulates the **Shift+drag-select** gesture: the cursor sweeps across the selection (a visible selection box is drawn) and the browser's own drag-select dialog is raised, then a button is clicked. The argument is one genomic region, `chrom:start-end`; a bare range **zooms**. For any other action, or to pass other keys, put the region under `range:` and quote it — unquoted commas split a `{..}` flow map: `drag: {range: "chr7:155,805,900-155,806,950", shot: dragselect, then: highlight}`. Endpoints that are not genomic coordinates are given as a fraction (`fromFrac:`/`toFrac:`) or raw px (`fromX:`/`toX:`) instead. Optional `track:` picks the row the box is drawn over; default is the top of the image. `shot: dragselect` captures the open dialog. `then:` = `zoom` (default → **Zoom In**) \| `highlight` (→ Single Highlight) \| `cancel` (Escape, view unchanged). (A real button-held drag would just pan, so the dialog is driven directly.) | | `open: lift` | Click the returned coordinate link → the lifted view. | | `zoom: out` / `zoom: in` | One zoom step (2×). | | `montage: {name: figure1, shots: [source, lifted]}` | Compose stills already written this run into **one multi-panel PNG**, which is what a journal wants for a figure with parts (A), (B), and so on. Panels are stacked in order and lettered automatically; `labels: [Before, After]` overrides the letters, `labels: false` drops them, `direction: horizontal` puts them side by side, and `gap:` / `labelSize:` tune the spacing and lettering. Composed at deviceScaleFactor 1 with every panel at its **natural pixel size**, so the composite is pixel-for-pixel its inputs: a `make hires` montage is print resolution because the panels were, not because anything was upscaled. Panels narrower than the widest are left-aligned and padded, never stretched. A named shot that was never taken is warned about and skipped. Put it last, after the `shot:`/`pinShot:` steps it names. | +| `loadSession: https://example.org/settings.txt` | Start from a **saved state** instead of a clean cart, so one tour can begin where another ended and a bug report that arrives as a session link becomes a starting position. Four forms: a **settings file by URL** (as above), a **share link** (`loadSession: https://genome.ucsc.edu/s/Braney/hg38`), a **named session** (`loadSession: {user: Braney, name: hg38}`), or a **local file** written by an earlier `session:` (`loadSession: {file: saved}` → `sessions/<base>/saved.txt`, sent up through hgSession's own upload form, so the project's sessions need not be published at all). Quick and silent, like `hub:` — this is setup, not something the tour demonstrates; add `shot:` to capture where it lands. Whatever the form, the load is issued against `target:` — see **Sessions** for why a share link is not simply followed. | +| `expect: {rows: [ruler, mane]}` | **The one verb that can fail a run.** Everything else renders happily whatever it is handed, so a wrong figure is written over a right one and only an eye catches it. State the expectation instead and the run stops, non-zero, at the step that broke it. Checks, any combination: `rows:` (these were drawn — plain names, matched by suffix so a lifted `hub_<n>_mane` counts), `exact: true` (…and nothing else), `noRows:` (these were not), `height: 2000` (the still is no taller than that in pixels; `"<1200"`, `">=300"` for another comparison), `tip: "mismatch A->C"` (the tooltip now up says this), `text:` / `noText:` (the page does / does not contain this — `noText: "Too Long"` catches the Apache 414 that renders as a perfectly good page). A failure names every check that failed **and the rows actually drawn**. `warn: true` downgrades it to a warning for a check worth logging but not worth stopping a build over. | +| `session: source` | Write `sessions/<base>/<name>.txt`: the **whole cart at this step**, in the format hgSession's "save settings to a local file" produces, so anyone can load it and get this exact view. Every track's visibility, the attached hubs, the custom tracks, the window. Off the video and off the page — it is fetched over the tour's own cookies, so the tour is not disturbed and nothing appears in the mp4. With `sessionUrlBase:` set at the top of the file, the run also prints the ready-made load URL. See **Sessions**. | | `shot: source` | Write `<name>.png` **and** pause the video here. On a tracks page the still is the track image (`#imgTbl`), plus any open tooltip/dialog. On any other page (an hgc detail page, an external page a link led to) it is the **viewport only — the top of the page**, never the whole scrolling document. | Escape hatches for anything the verbs don't cover: `goto: <url>`, `click: <sel>`, `hover: <sel>`, `wait: <sel>`, `sleep: <ms>`. +## Sessions + +A `shot:` gives a picture of the view. A `session:` gives the view itself: + +```yaml +sessionUrlBase: https://hgwdev-you.gi.ucsc.edu/~you/docent/sessions +steps: + - hide: all + - track: {clinvar: pack} + - session: clinvar_view + - shot: clinvar_view +``` + +That writes `sessions/<base>/clinvar_view.txt` and prints the URL that loads it: + +``` +SESSION sessions/BP1/clinvar_view.txt (221 settings) + https://genome-test.gi.ucsc.edu/cgi-bin/hgTracks?hgS_doLoadUrl=submit&hgS_loadUrlName=https%3A%2F%2F... +``` + +The file is what hgSession's **save settings to a local file** button produces, and it is +made the same way (`doSaveLocal` in `hg/hgSession/hgSession.c`), so **no login is +involved** and the file holds the whole cart: every track's visibility, the attached hubs, +the custom tracks, the window. + +It is a **file rather than a live `hgsid` link** because the hgsid cart goes on changing as +the tour runs. A link handed out at step 5 would open on whatever step 20 left behind. The +file is a snapshot of the step it was taken at, which is what `shot:` already means — so a +figure and a working session can be produced at the same moment and cannot disagree. + +The fetch goes through the tour's own cookies but never through the page, so **the tour is +not disturbed and nothing appears in the mp4**. hgSession drops its own `hgS_*` variables +before it checks the cart out, so the next step starts from an unchanged state. + +Publishing is the project's business, the way stills already are. `sessionUrlBase:` is the +http URL of the **parent** of the `<base>/` directories; without it, only the path is +printed. hgTracks fetches the file itself, so it has to be reachable over http by the +server named in `target:` — a `~/public_html` path on hgwdev is enough, and a private one +is not. + +### Starting from one + +`loadSession:` is the other direction, and it is what makes a session useful for work rather +than only for sharing. A bug report usually arrives as a session link, and a tour that has to +rebuild that state by hand is guessing at it: + +```yaml +steps: + - loadSession: https://genome.ucsc.edu/s/Braney/hg38 + - expect: {noText: "Can't find session"} + - shot: as_reported +``` + +The load always goes to the server named in `target:`, even when the link names another +host. Every later step navigates to `target:` by absolute URL, so following the link would +leave the tour on the other server's cart and the next step would silently abandon it. +**Named sessions are per-server** — the RR reads `hgcentral`, hgwdev and genome-test read +`hgcentraltest` — so a link copied off genome.ucsc.edu only loads if that session also +exists on the machine being driven. Docent warns when the hosts differ. A settings *file* by +URL has no such problem, since hgTracks simply fetches it, and no warning is given. + +`loadSession: {file: <name>}` reads a file written by an earlier `session:` in the same +project and sends it up through hgSession's upload form, so a script can save a state, +wander off, and come back to it without publishing anything. + +### Two things to know + +- **A session taken on a lifted view is short-lived.** quickLift builds its hub under + `trash/`, and the file records it by that path (`assumesHub 192070=../trash/quickLift/...`). + It reloads correctly while the hub is there; it stops working when trash is cleaned. The + GenArk hub beside it is a stable `/gbdb` path and does not expire. So a lifted session is + good for showing a colleague what you are looking at this week, not for a paper. +- **`make hires` writes to `sessions.hires/`.** A print run's cart carries `pix=2550` and + `textSize=24`, which is not a state anyone wants handed to them, so it is kept out of the + way of the screen run's files rather than overwriting them. + +## Expectations + +Every other verb renders happily whatever it is handed. A superTrack that came up whole +makes an image 7,581 px tall, a subtrack that never hid stays in the figure, a pinned +tooltip grabs the neighbouring item, an Apache 414 arrives as a perfectly valid page saying +*Request-URI Too Long*. All of those have shipped at least once, all were caught by eye, and +all of them are mechanically checkable. `expect:` is the only verb that stops a run: + +```yaml + - track: {varsInPubs: hideKids, pubtator: pack} + - expect: {rows: [ruler, mane, pubtator], exact: true, height: 2000, noText: "Too Long"} + - mouseover: {track: quickLiftChain, item: "4.3.157828209.157828210", pin: true} + - expect: {tip: "mismatch A->C"} +``` + +A failure prints every check that failed **and the rows that were actually drawn**, then +exits non-zero at that step, so `make` fails instead of writing a wrong figure over a right +one and the stills that would have followed are never taken: + +``` +step 5 (expect) failed: rows not drawn: clinvarMain; image is 69px, wanted <50 + drawn: ruler, mane +``` + +Notes: + +- **Row names are matched by suffix**, the way `mouseover:` resolves a track, so `mane` + is satisfied by a lifted view's `hub_192070_mane` and a script does not have to know the + per-run hub number. +- **`height:` is in still pixels**, which is the CSS height times the device pixel ratio — + the number someone means by "7,581 px tall". A print run makes it k times bigger, so a + ceiling that passes at 1x will fail at 3x. Scale it with `scale:` or leave the check to + the screen run. +- **A bare number is a ceiling** (`height: 2000` means no taller than 2000), which is the + check anyone actually wants. `"<1200"`, `">=300"` and `"=850"` are there when it is not. +- **`warn: true`** logs the failure and carries on, for a check worth recording but not + worth stopping a build over. + ## Speed A full run is a real browser against a real server, and most of its wall clock is the pacing that makes the *video* watchable, not work. Measured on BP1 (15 steps, 4 stills): **64 s**, of which ~40 s is dwells, cursor animation and dropdown theatrics, ~23 s is page loads, and ~1 s is the mp4 transcode. So when you are iterating on the **figures**, skip the video: make FAST=1 BP1 # or DOCENT_FAST=1, or `fast: true` in the script FAST drops the dwells (`pace`, `shotHold`, mouseover holds), moves the cursor in one jump, skips the dropdown open/highlight, and records no video, so there is no mp4 to transcode either — **24 s** instead of 64 s for BP1. The stills carry the same content; a couple of things about them are incidental rather than identical, so build normally for the figures