b4e78d426dcdb47a20f1979025d5a0969ea79ac4 braney Thu Sep 17 17:07:49 2026 -0700 docent: a box: check for where an element sits, and lists for text:/noText:, refs #37892 `expect:` could say what was in a page and never where it was on the screen. has:/noHas: take a CSS selector, which describes the tree; color: reads pixels but only inside a track's row. #38251 moved the narrow-window menu icon out of the blue bar with every selector still matching and every word of the page still there, and nothing in the language could ask about it. box: takes `inside:` (every edge within another element's box, with `tolerance:` px of slack), `clear:` (no overlap with anything a selector matches, `gap:` for a minimum separation) and `height:`/`width:`. Every element `sel:` matches has to satisfy every clause, so {sel: "ul.nice-menu > li", inside: "#main-menu-whole"} reads as "every menu item is in the bar". Boxes are read in document coordinates; an element with no box at all is skipped rather than treated as a zero-sized box at the origin, which would sit "inside" anything. A failure prints the measurement: #topRightLinks is not inside #main-menu-whole: 114px above it -- it is at 666,0 34x32, #main-menu-whole at 0,114 1000x32 The image height: and box's own height:/width: now share one cmpSize(), so the two cannot drift into different comparison grammars. text: and noText: take a list, the way rows:, has: and noHas: always did. They have to: a list handed to a check that stringifies its argument fails OPEN -- ["a", "b"] becomes "a,b", which no page contains, so it passes on anything, and passes silently. Six scripts in one batch were written that way and all six looked green. pagechecks and its .xfail twin cover both, the xfail with one entry aimed wrongly and one aimed rightly in each list, so it fails only if every entry is really looked at on its own. diff --git src/hg/utils/docent/README.md src/hg/utils/docent/README.md index 2186b57f549..e68b1a27ba8 100644 --- src/hg/utils/docent/README.md +++ src/hg/utils/docent/README.md @@ -1,471 +1,505 @@ # Docent — a language for authoring guided tours of the Genome Browser A **docent** leads a tour and explains what you are looking at. A Docent script is that tour, written down: an ordered list of high-level verbs (`go`, `hide`, `track`, `mouseover`, `convert`, `drag`, `shot`) that drive a real browser against a real server. `docent.js` renders one script into three things at once: - a **silent mp4** of the whole tour, - a named **PNG still** at every `shot:` marker, and - a loadable **session file** at every `session:` marker. So a published figure is literally a frame of the tour, and the two can never drift apart. A session file goes further: it hands back the state itself, so a reader can open the view the figure was taken from instead of only looking at it. The surface syntax is YAML — so ordinary editors highlight it and no one has to learn a new parser — but the language is the verb vocabulary layered on top, not the serialization. Scripts are named `.docent.yaml` (a bare `.docent` works too). The verbs deliberately encode *browser mechanics* rather than selectors: `convert:` knows that the Hide-defaults checkbox reverts when the Assembly menu reloads, `track: {clinvar: pack}` asks trackDb which containers and checkboxes that implies, and `mouseover:` knows that a lifted track's DOM id gains a per-run `hub__` prefix. The author writes intent; the renderer deals with the UI. ## Run ``` PW=/hive/groups/browser/uiTest/pw PLAYWRIGHT_BROWSERS_PATH=$PW/browsers NODE_PATH=$PW/node_modules \ node docent.js AP1.docent.yaml ``` `docent.js` is not a single file: it requires `targetConf.js` from beside it, which answers where a run is pointed (`target:` and `DOCENT_TARGET`), which `hg.conf` that server reads, which hgcentral that names, and which account a `login:` step signs in with. `tests/preflight.js` requires the same module, which is the point of it being one: the fixture check has to resolve all four exactly the way the run will. Copy the pair, or run `docent.js` where it sits. Needs `playwright`, `js-yaml`, and `ffmpeg`. At UCSC these live in one pinned shared install at `/hive/groups/browser/uiTest/pw` (`browsers` for Chromium, `node_modules` for the modules), which every browser-driving test in the tree uses; its `README.md` records the pinned versions. Point the two variables above anywhere you have them. Outputs, relative to the script's own directory: - mp4 → `../.mp4` (override with `mp4:` or a second argument) - stills → `stills//.png` (override with `stills:`) - sessions → `sessions//.txt` (override the parent with `sessions:`) `docent.mk` in this directory has the make rules — include it from a project that keeps a set of scripts and it rebuilds only the ones whose source changed. See the usage comment at the top of that file. ## Top of file (all optional) ```yaml target: genome-test # or rr, hgwdev, hgwbeta, hgwdev-, or a full https://.../cgi-bin db: hg38 # source assembly position: chr7:155.8M # starting position (tracked from then on; `go:` sets it too) reset: true # cartReset first (clean cart + fresh quickLift hub) size: [1000, 760] # viewport pix: 850 # browser image width pace: 1.2 # seconds to dwell after each step shotHold: 2.2 # extra seconds the video pauses at a shot trackAnim: false # skip the visible dropdown gesture on `track:` (nav only) fast: false # true = figures only: no dwells, no cursor animation, no mp4 scale: 1 # >1 = print-resolution stills (see Print resolution) pinMouseovers: false # record every mouseover for `pinShot:` (see below) mp4: ../myname.mp4 # override output paths if you want stills: stills/AP1 sessions: sessions # PARENT of the session dirs; files land in // sessionUrlBase: https://hgwdev-you.gi.ucsc.edu/~you/docent/sessions # see Sessions ``` `target:` defaults to genome-test, so a script that forgets to say where it runs will not quietly hit someone's personal sandbox. ## Print resolution A screen still is about 120 dpi across a journal's 7-inch column — fine on a monitor, too coarse to print. `scale:` (or `DOCENT_SCALE=k`, or `make hires`) renders **the same tour with k times the pixels**, which is not the same thing as enlarging the stills afterwards: a still only ever has the pixels it was drawn with, so every layer is asked to draw more of them. ``` make hires # every scenario at 3x -> stills.hires// make hires SCALE=2 # 2x make hires BASES=BP1 # one scenario DOCENT_SCALE=3 DOCENT_STILLS=stills.hires DOCENT_FAST=1 node docent.js BP1.docent.yaml ``` What k does: - **deviceScaleFactor: k**, with the viewport left at its 1x CSS size. The page lays out exactly as at 1x — same line breaks, same jQuery-dialog width, same tooltip placement — and all of it rasterizes with k times the pixels. - **`pix` × k, `textSize` stepped up with it** (hgTracks' size ladder: 6, 8, 10, 12, 14, 18, 24, 34; 3x lands exactly on 24). The server draws a genuinely wider image, and the bigger font keeps its layout decisions proportional — tick spacing, room for labels, how features pack into rows. A wider image with an 8px font would be a different picture, not a bigger one. - **`zoom: 1/k` on the image table**, handing that wider image the 1x amount of layout space, so one image pixel lands on one device pixel. Native resolution, no resampling. - **The tooltip font is pinned back to its 1x size.** hgTracks takes the tooltip's font-size from the browser text size (`window.browserTextSize` → `hg/js/utils.js` `addMouseover`), which `textSize × k` has just tripled — and then the device pixel ratio scales the same text a second time. Left alone, a 3x still gets tooltips 3x too big: the popups swamp the figure and the last one pinned falls off the crop. - **Every drawn row asks for k times its height** (`.heightPer`, sent after each view change for the rows the page actually drew). A track with a FIXED PIXEL height does not follow `pix` or `textSize` — a bigLolly or wiggle row is a pixel count from trackDb/the cart — so a 128px row that was 15% of an 850px image would be 5% of a 2550px one. That is how ClinVar's lollipop row came out a sliver with unreadable y-axis labels. It is asked of the **drawn** row names, which is the only way to reach a **lifted/quickLift view**, whose tracks are hub tracks under names trackDb never saw. Each track's own `maxHeightPixels` still clamps the request, so **a fixed-height track needs a ceiling in trackDb to grow**: without one the default (128) is also the maximum and the request is silently clamped. Everything hgTracks reports about the image — map-box coords, mouseOver spans, `insideX` — is in the pixels the server drew, so it goes through the image's natural-to-displayed ratio before it becomes a page coordinate. That is why a `mouseover: {item: ...}` lands on the same feature at 3x as at 1x. Two things to expect: - **No mp4.** A scaled run is a figure run; build the video from an unscaled run of the same script. - **Densely packed tracks come out shorter.** hgTracks' font metrics are not perfectly linear across the size ladder, so a 24px font is a little narrower than three 8px ones and `pack` fits the same features into fewer rows. Same window, same tracks, same items, slightly roomier picture. Use `DOCENT_STILLS=` (what `make hires` does) so a print render lands beside the screen stills instead of overwriting them. ## Steps `steps:` is an ordered list. Each item is a bare verb or a one-key map. | Step | What it does | |------|--------------| | `gateway` | Open hgGateway for `db`. | | `go` | Click **GO** with the default position (no typing). | | `go: chr7:155799529-155812871` | Go straight to a position in hgTracks. | | `goShow: BRCA1` | **Demonstrates the position change through the UI** (vs. `go:` which navs there directly): the cursor glides to the **position box** and the term is **typed on screen**, then the page finishes the job — **Search** on hgTracks, the arrow on hgGateway — so one verb covers either page. Takes a **gene name** or a **position** (or anything else the box accepts: HGVS, an accession). Bare string, or map form `goShow: {gene: SHH, shot: source}` / `goShow: {position: "chr7:155,799,529-155,812,871", shot: source}` — quote coords in the map form (unquoted commas split a `{..}` flow map). | | `goShow: {gene: TP53, pick: "NR_176326"}` | A **gene name** goes through the browser's own **suggestion menu**, the way a user does it: Docent waits out the hgSuggest ajax, then clicks the matching row, so you land on the gene (not the search-results page) and the video shows the dropdown. Default match is the exact gene symbol; `pick:` selects a specific row by a substring of its text when the term is ambiguous (a `pick:` that matches nothing warns and falls back to the plain gene match). If nothing ever matches, the typed term is submitted as-is — that may land on the **search-results page**, which is legal, and a `click:` can take it from there. | | `hide: all` | Click **Hide all**. | | `track: {mane: pack}` | Set a track's visibility (`hide/dense/squish/pack/full`). The mouse visibly glides to that track's control dropdown, opens it, and picks the mode (then the state is applied). | | `track: {dbSnp155Common: pack}` | A subtrack by name: trackDb's view and composite above it come along, so this is the whole "common dbSNP" config. (`dbSnp155` is a *different*, off-by-default subtrack — all variants — not an alias for the composite.) | | `track: {clinvar: pack, clinvarCnv: hide}` | Composite: the container's mode reaches its subtracks, so name only the deviations. Docent adds the containers above and the `_sel` checkbox from trackDb — see **Track names come from trackDb**. | | `track: {varsInPubs: hideKids, pubtator: pack}` | **superTrack: show one member only.** `hideKids` is not a visibility — it means *hide everything under this container*, so the child named alongside it is left the only one drawn. A superTrack needs this because, unlike a composite, its own mode does **not** reach its children: each comes up at its own trackDb visibility, so `{varsInPubs: show}` draws all eight of its members and an earlier `hide: all` does not stick. The expansion skips any child the step names itself and is sent in a round of its own **after** the rest — a subtrack hide travelling in the same request as its container can be dropped by the cart (#37953), so the container goes on first and the hides follow. Works on a composite or view too, where it deselects (`_sel=0`) rather than just hiding. `hideKids` alone (no child named) simply empties the container. The expansion stops at the first container that propagates its own visibility, so a composite under the superTrack is hidden as a unit rather than enumerated: `{cCREs: hideKids}` on hg38 sends two variables, not the 1701 that naming all 850 subtracks of the ENCODE4 Core Collection would take (which overran Apache's request-line limit and returned a 414 page in place of the view). | | `track: {anyTrackName: dense}` | Any track by its **trackDb** name. A name trackDb doesn't have (hub, custom track, quickLift target) is sent literally as `name=mode`. | | `mouseover: {track: dbSnp155Common, item: rs28406051}` | Hover a **named** item to raise its tooltip. `item:` matches the item's map box (its `&i=` HREF or its TITLE) and hovers that rectangle's center — so it lands on the **correct row** even when items are stacked, which a bare coordinate can't do. It then waits for **that item's own tooltip**, not merely for a tooltip: the browser shows one 500 ms after `mouseenter` and hides it 500 ms after `mouseleave` (`hg/js/utils.js`), so the cursor gliding in crosses other items and one of THEIR tooltips can still be on screen on arrival — which is how an Alignment Differences mismatch got pinned as its neighbour's "identical". The expected text comes from the item's own map box, rendered the way the tooltip renders it (the attribute holds markup and undecoded entities), and `DOCENT_ROWS=1` warns if it never appears. `title: "SHH"` matches the TITLE text only; `value: "..."` matches a JSON `mouseOver` span (wig/dense tracks). | | `mouseover: {track: mane, at: chr7:155805900}` | Hover by **position** when you don't need a specific item: genomic coord (`at:`), a fraction across the view (`frac: 0.5`), or a raw pixel (`x: 400`); the y is forced to the middle of that track's row (cannot disambiguate stacked items). Optional on any `mouseover`: `hold: 2.5` (seconds to dwell) and `shot: tip_mane` (capture the image **plus** the tooltip in one still). | | `mouseover: {track: dbSnp155Common, item: rs28406051, pin: true}` | `pin: true` **records** that tooltip (its text + position) so a later `pinShot:` can show several mouseovers open together in one figure. Nothing is added to the recorded page, so the **mp4 is unaffected** (it still shows only the transient native tooltip). Set `pinMouseovers: true` at the top of the file to record every mouseover by default (`pin: false` opts one out). Records accumulate within a view and are cleared on the next nav. | | `pinShot: all_tips` | Write `.png` with **all recorded (pinned) tooltips open at once**, each with a **cursor drawn at the point it was raised from** — so the figure says which feature every tooltip belongs to instead of leaving the reader to infer it from the anchor. Rendered on a throwaway page that shares the session (same cart/view) — never on the recorded page — so it never appears in the mp4. Consumes the recorded set (clears it). Place it after the `mouseover` steps whose tooltips you want shown together, before any nav/zoom. Map form `pinShot: {name: all_tips, cursors: false}` drops the pointers. | | `click: {track: mane, item: "NM_000546.6"}` | **Click a track item** and follow its own map-box link, which is how the hgc details page is reached — a raw mouse click on the data area is swallowed by hgTracks' drag-select handler. Addressed the same way `mouseover:` is: by identity (`item:`/`title:`/`value:`) or by **position** (`at:`/`frac:`/`x:`), which takes the item box nearest that point. Position is the only way in for a track whose items cannot be named at all — every GIAB Problematic Regions subtrack is `type bigBed 3`, so hgTracks writes an empty `i=` into the hgc href and gives every box the same title, and `click: {track: alldifficultregions, frac: 0.5}` is then the way to open one. A bare string is a plain CSS selector click instead (`click: 'a:has-text("placed on its chromosome")'`), for a link on a page that has no track image. | | `click: {track: mane, item: "NM_000546.6", raw: true}` | **Press the mouse where a user presses it** and let the page answer, instead of following the item's link. A different gesture, not a slower route to the same page: hgTracks answers a real item click with an ajax **dialog** (`popUpHgcOrHgGene.hgc`), and a whole class of bug lives in that dialog rather than on the hgc page -- hgTracks hanging on the SECOND click of the same item (#36805). Following the href never opens a dialog, so it can never see one. With no item name, `raw:` is a **bare point on the row** (`frac:`/`at:`/`x:`), which is the only way to click a row that carries no hgc map boxes at all: a click on the **ruler** (#27113). Docent waits for whichever of the three answers arrives -- a navigation, a dialog, or a new image in place -- so it needs no sleep. Note that jQuery UI **hides** a dialog on close rather than removing it, so assert `has: "#hgcDialog:visible"`, not `has: "#hgcDialog"`. | | `convert: {to: GCA_018466845.2, quicklift: true, hideDefaults: true}` | View→Convert, then **type the target into the page's own "Search for target genome" bar** and click the suggestion (`search:` overrides what is typed, `pick:` disambiguates the menu); the Assembly dropdown is checked afterwards and only opened by hand if the search didn't land there. QuickLift on, **re-checks Hide-defaults** (it reverts when the Assembly menu reloads), Submit. `to:` accepts an accession or a label fragment (`2257.pat`, matched against the dropdown text). A bare string is `to:`, so `convert: hs1` is a plain coordinate convert to hs1 -- `quicklift: true` is never implied. | | `convert: {to: ..., shot: convert_filled}` | `shot:` inside `convert:` captures the Convert page, which no other verb can reach. A bare name is the **filled-in page just before Submit**. The map form names up to three moments: `shot: {opened: a, filled: b, result: c}` — `opened` as the page comes up, `filled` ready to Submit, `result` the conversion-result page (whose coordinate link `open: lift` clicks). These are viewport stills, so they show the page from the top. | | `hub: https://example.org/hub.txt` | **Quick, silent** attach of a track hub by URL (`hgTracks?hubUrl=...`): connects the hub so its tracks are available at their hub-declared visibility. Follow with `track:` to turn specific ones on. Map form `hub: {url: ..., db: hg38, position: chr7:...}` overrides the db/position (default: current `db` + last position). | | `addHub: https://example.org/hub.txt` | **Demonstrates the attach through the UI** (for the figure/video): opens My Data → Track Hubs, clicks the **Connected Hubs** tab, types the URL into the box, and clicks **Add Hub** — cursor glides and the URL is typed on screen. Then, on the "Hub Connect Successful" page, it **clicks the `Open:` link for `db`** so the demo ends on the browser with the hub loaded. Map form `addHub: {url: ..., db: hg38, shot: loaded}` sets which assembly to open and captures the still on that tracks view. Use `hub:` instead when you just need the hub attached without showing the steps. | | `addCustomTrack: ` | **Demonstrates loading a custom track via the UI**: opens My Data → Custom Tracks, types the track data (or a data URL) into the paste box, clicks **Submit**, then clicks through to the browser (**Go to first annotation**). Bare string is the data or URL; map form `addCustomTrack: {data: "track ...\nchr7 ...", db: hg38, goto: first, shot: loaded}` (use `url:` for a data URL, `goto: current` to land on **Return to current position** instead). Data is inserted literally (tabs/newlines preserved). In YAML, a multi-line track uses a block scalar: `addCustomTrack: |` then the indented lines. | | `addPublicHub: GTEx` | **Demonstrates connecting a PUBLIC hub via the UI**: opens My Data → Track Hubs, the **Public Hubs** tab, types the search terms, clicks **Search Public Hubs**, then clicks **Connect** on the matching hub row, and finally **clicks the `Open:` link for `db`** to land on the browser. Bare string is the search term (also used to match the row). A search usually returns several hubs, so use the map form `addPublicHub: {search: "GTEx", match: "GTEx Analysis Hub", db: hg38, shot: loaded}` to pick the exact hub by a substring of its row text (`match:` defaults to `search:`) and the assembly to open. If no row matches it **won't connect** (warns and stops) rather than pick the wrong hub. | | `drag: chr7:155,805,900-155,806,950` | Emulates the **Shift+drag-select** gesture: the cursor sweeps across the selection (a visible selection box is drawn) and the browser's own drag-select dialog is raised, then a button is clicked. The argument is one genomic region, `chrom:start-end`; a bare range **zooms**. For any other action, or to pass other keys, put the region under `range:` and quote it — unquoted commas split a `{..}` flow map: `drag: {range: "chr7:155,805,900-155,806,950", shot: dragselect, then: highlight}`. Endpoints that are not genomic coordinates are given as a fraction (`fromFrac:`/`toFrac:`) or raw px (`fromX:`/`toX:`) instead. Optional `track:` picks the row the box is drawn over; default is the top of the image. `shot: dragselect` captures the open dialog. `then:` = `zoom` (default → **Zoom In**) \| `highlight` (→ Single Highlight) \| `cancel` (Escape, view unchanged). (A real button-held drag would just pan, so the dialog is driven directly.) | | `open: lift` | Click the returned coordinate link → the lifted view. | | `zoom: out` / `zoom: in` | One zoom step (2×). | | `montage: {name: figure1, shots: [source, lifted]}` | Compose stills already written this run into **one multi-panel PNG**, which is what a journal wants for a figure with parts (A), (B), and so on. Panels are stacked in order and lettered automatically; `labels: [Before, After]` overrides the letters, `labels: false` drops them, `direction: horizontal` puts them side by side, and `gap:` / `labelSize:` tune the spacing and lettering. Composed at deviceScaleFactor 1 with every panel at its **natural pixel size**, so the composite is pixel-for-pixel its inputs: a `make hires` montage is print resolution because the panels were, not because anything was upscaled. Panels narrower than the widest are left-aligned and padded, never stretched. A named shot that was never taken is warned about and skipped. Put it last, after the `shot:`/`pinShot:` steps it names. | | `login` | **Sign in through hgLogin**, for the pages that refuse a visitor who is not logged in -- hgCollection above all (`hgCollection.c` doMiddle: *You must be logged in to edit collections*), and the saving half of hgSession. The login cookie is validated against a salted hash (`login.cookieSalt`, `hg/lib/wikiLink.c`), so there is no cookie to hand the browser: a script that needs one of those pages has to sign in the way a person does. **The step takes no credentials and cannot be given any.** They are read from `~/.docentLogin` (override with `DOCENT_LOGIN_FILE`), **one section per hgcentral database** -- an account is a row in `gbMembers` in one of them, so that is the key, not the server and not the sandbox:

`[hgcentraltest]`
`user=docentTest`
`password=...`

genome-test, hgwdev, every `hgwdev-` sandbox and every ticket park read hgcentraltest, so one account covers all of them; hgwbeta reads hgcentralbeta and the RR reads hgcentral. Which central a server reads is **read from its hg.conf**, not assumed from the host, because a sandbox can say so for itself -- 45 of the personal confs on hgwdev set `central.db=hgcentraltest` and two do not. A server whose conf is on another machine falls back to a small table (the RR, the two mirrors, hgwbeta), and `[default]` catches the rest. A run redirected with `DOCENT_TARGET` looks up the server it is really driving. The file is refused unless it is mode 0600 -- the rule `hg/lib/hgConfig.c` applies to `hg.conf`. `DOCENT_LOGIN_USER` + `DOCENT_LOGIN_PASSWORD` override the file for one run. Nothing prints a password. A wrong password fails the step rather than carrying on logged out, because hgLogin answers one by drawing the same form again, which is a perfectly good page. Map form `login: {shot: signed_in}`. `make preflight` reports the account and the central it resolved, so a missing password is caught before the browser starts. | | `loadSession: https://example.org/settings.txt` | Start from a **saved state** instead of a clean cart, so one tour can begin where another ended and a bug report that arrives as a session link becomes a starting position. Four forms: a **settings file by URL** (as above), a **share link** (`loadSession: https://genome.ucsc.edu/s/Braney/hg38`), a **named session** (`loadSession: {user: Braney, name: hg38}`), or a **local file** written by an earlier `session:` (`loadSession: {file: saved}` → `sessions//saved.txt`, sent up through hgSession's own upload form, so the project's sessions need not be published at all). Quick and silent, like `hub:` — this is setup, not something the tour demonstrates; add `shot:` to capture where it lands. Whatever the form, the load is issued against `target:` — see **Sessions** for why a share link is not simply followed. | -| `expect: {rows: [ruler, mane]}` | **The one verb that can fail a run.** Everything else renders happily whatever it is handed, so a wrong figure is written over a right one and only an eye catches it. State the expectation instead and the run stops, non-zero, at the step that broke it. Checks, any combination: `rows:` (these were drawn — plain names, matched by suffix so a lifted `hub__mane` counts), `exact: true` (…and nothing else), `ordered: true` (…and in that order, top to bottom), `noRows:` (these were not), `height: 2000` (the still is no taller than that in pixels; `"<1200"`, `">=300"` for another comparison), `tip: "mismatch A->C"` (the tooltip now up says this), `text:` / `noText:` (the page does / does not contain this — `noText: "Too Long"` catches the Apache 414 that renders as a perfectly good page), `url:` / `noUrl:` (the current address does / does not contain this — which CGI a click reached, or what a form put in the query string; `noUrl: "%E2%80%8B"` is the only way to see that a search term's zero-width space was stripped, since it is invisible in the page), `color:` (the color a track's row is actually **drawn** in -- `{track: crm4, is: "0,0,255"}`, or `not:` for one it must not be; `part: label` asks about the center label instead of the items, `at:`/`frac:`/`x:` about one item instead of the whole row, and a **list** states several rows in one step. The only check that reads the IMAGE, for a bug that leaves the page identical -- same rows, same height, same names, same tooltips), `has:` / `noHas:` (a CSS selector matches / matches nothing — for a bug whose whole signature is WHERE something sits, like a center label attached to the wrong row: same rows, same height, same pixels. Reach for these last, since an assertion on hgTracks' own ids breaks easily for reasons that are not bugs). A failure names every check that failed **and the rows actually drawn**. `warn: true` downgrades it to a warning for a check worth logging but not worth stopping a build over. | +| `expect: {rows: [ruler, mane]}` | **The one verb that can fail a run.** Everything else renders happily whatever it is handed, so a wrong figure is written over a right one and only an eye catches it. State the expectation instead and the run stops, non-zero, at the step that broke it. Checks, any combination: `rows:` (these were drawn — plain names, matched by suffix so a lifted `hub__mane` counts), `exact: true` (…and nothing else), `ordered: true` (…and in that order, top to bottom), `noRows:` (these were not), `height: 2000` (the still is no taller than that in pixels; `"<1200"`, `">=300"` for another comparison), `tip: "mismatch A->C"` (the tooltip now up says this), `text:` / `noText:` (the page does / does not contain this, one string or a **list** of them — `noText: "Too Long"` catches the Apache 414 that renders as a perfectly good page), `url:` / `noUrl:` (the current address does / does not contain this — which CGI a click reached, or what a form put in the query string; `noUrl: "%E2%80%8B"` is the only way to see that a search term's zero-width space was stripped, since it is invisible in the page), `color:` (the color a track's row is actually **drawn** in -- `{track: crm4, is: "0,0,255"}`, or `not:` for one it must not be; `part: label` asks about the center label instead of the items, `at:`/`frac:`/`x:` about one item instead of the whole row, and a **list** states several rows in one step. The only check that reads the IMAGE, for a bug that leaves the page identical -- same rows, same height, same names, same tooltips), `has:` / `noHas:` (a CSS selector matches / matches nothing — for a bug whose whole signature is where something sits in the page's TREE, like a center label attached to the wrong row: same rows, same height, same pixels. Reach for these last, since an assertion on hgTracks' own ids breaks easily for reasons that are not bugs), `box:` (where an element sits on the SCREEN — `{sel: "#topRightLinks", inside: "#main-menu-whole"}`, plus `clear:` for what it must not overlap and `height:`/`width:` for its own size. The only check that reads a bounding box, for a bug that leaves every selector matching and every word of the page in place). A failure names every check that failed **and the rows actually drawn**. `warn: true` downgrades it to a warning for a check worth logging but not worth stopping a build over. | | `session: source` | Write `sessions//.txt`: the **whole cart at this step**, in the format hgSession's "save settings to a local file" produces, so anyone can load it and get this exact view. Every track's visibility, the attached hubs, the custom tracks, the window. Off the video and off the page — it is fetched over the tour's own cookies, so the tour is not disturbed and nothing appears in the mp4. With `sessionUrlBase:` set at the top of the file, the run also prints the ready-made load URL. See **Sessions**. | | `shot: source` | Write `.png` **and** pause the video here. On a tracks page the still is the track image (`#imgTbl`), plus any open tooltip/dialog. On any other page (an hgc detail page, an external page a link led to) it is the **viewport only — the top of the page**, never the whole scrolling document. | Escape hatches for anything the verbs don't cover: `goto: `, `click: `, `hover: `, `wait: `, `sleep: `. ## Sessions A `shot:` gives a picture of the view. A `session:` gives the view itself: ```yaml sessionUrlBase: https://hgwdev-you.gi.ucsc.edu/~you/docent/sessions steps: - hide: all - track: {clinvar: pack} - session: clinvar_view - shot: clinvar_view ``` That writes `sessions//clinvar_view.txt` and prints the URL that loads it: ``` SESSION sessions/BP1/clinvar_view.txt (221 settings) https://genome-test.gi.ucsc.edu/cgi-bin/hgTracks?hgS_doLoadUrl=submit&hgS_loadUrlName=https%3A%2F%2F... ``` The file is what hgSession's **save settings to a local file** button produces, and it is made the same way (`doSaveLocal` in `hg/hgSession/hgSession.c`), so **no login is involved** and the file holds the whole cart: every track's visibility, the attached hubs, the custom tracks, the window. It is a **file rather than a live `hgsid` link** because the hgsid cart goes on changing as the tour runs. A link handed out at step 5 would open on whatever step 20 left behind. The file is a snapshot of the step it was taken at, which is what `shot:` already means — so a figure and a working session can be produced at the same moment and cannot disagree. The fetch goes through the tour's own cookies but never through the page, so **the tour is not disturbed and nothing appears in the mp4**. hgSession drops its own `hgS_*` variables before it checks the cart out, so the next step starts from an unchanged state. Publishing is the project's business, the way stills already are. `sessionUrlBase:` is the http URL of the **parent** of the `/` directories; without it, only the path is printed. hgTracks fetches the file itself, so it has to be reachable over http by the server named in `target:` — a `~/public_html` path on hgwdev is enough, and a private one is not. ### Starting from one `loadSession:` is the other direction, and it is what makes a session useful for work rather than only for sharing. A bug report usually arrives as a session link, and a tour that has to rebuild that state by hand is guessing at it: ```yaml steps: - loadSession: https://genome.ucsc.edu/s/Braney/hg38 - expect: {noText: "Can't find session"} - shot: as_reported ``` The load always goes to the server named in `target:`, even when the link names another host. Every later step navigates to `target:` by absolute URL, so following the link would leave the tour on the other server's cart and the next step would silently abandon it. **Named sessions are per-server** — the RR reads `hgcentral`, hgwdev and genome-test read `hgcentraltest` — so a link copied off genome.ucsc.edu only loads if that session also exists on the machine being driven. Docent warns when the hosts differ. A settings *file* by URL has no such problem, since hgTracks simply fetches it, and no warning is given. `loadSession: {file: }` reads a file written by an earlier `session:` in the same project and sends it up through hgSession's upload form, so a script can save a state, wander off, and come back to it without publishing anything. ### Two things to know - **A session taken on a lifted view is short-lived.** quickLift builds its hub under `trash/`, and the file records it by that path (`assumesHub 192070=../trash/quickLift/...`). It reloads correctly while the hub is there; it stops working when trash is cleaned. The GenArk hub beside it is a stable `/gbdb` path and does not expire. So a lifted session is good for showing a colleague what you are looking at this week, not for a paper. - **`make hires` writes to `sessions.hires/`.** A print run's cart carries `pix=2550` and `textSize=24`, which is not a state anyone wants handed to them, so it is kept out of the way of the screen run's files rather than overwriting them. ## Expectations Every other verb renders happily whatever it is handed. A superTrack that came up whole makes an image 7,581 px tall, a subtrack that never hid stays in the figure, a pinned tooltip grabs the neighbouring item, an Apache 414 arrives as a perfectly valid page saying *Request-URI Too Long*. All of those have shipped at least once, all were caught by eye, and all of them are mechanically checkable. `expect:` is the only verb that looks at any of it: ```yaml - track: {varsInPubs: hideKids, pubtator: pack} - expect: {rows: [ruler, mane, pubtator], exact: true, height: 2000, noText: "Too Long"} - mouseover: {track: quickLiftChain, item: "4.3.157828209.157828210", pin: true} - expect: {tip: "mismatch A->C"} - expect: {color: {track: crm4, is: "0,0,255", not: "0,255,0"}} + - expect: {box: {sel: "#topRightLinks", inside: "#main-menu-whole"}} ``` A failure prints every check that failed **and the rows that were actually drawn**, then exits non-zero at that step, so `make` fails instead of writing a wrong figure over a right one and the stills that would have followed are never taken: ``` step 5 (expect) failed: rows not drawn: clinvarMain; image is 69px, wanted <50 drawn: ruler, mane ``` Notes: - **Other verbs can still fail a run, they just do not check anything.** A verb throws when it cannot do what it was told: `mouseover:` cannot find the item, `loadSession:` cannot find the file, `drag:` gets endpoints it cannot read, `convert:` cannot find the assembly in the dropdown. Any step's throw becomes `step N (verb) failed` and exit 1. What none of them does is judge the result, and `track:` does not even judge its own input -- a name no assembly has is sent as `name=mode` and the run exits 0. That is why a script with no `expect:` in it is a tour and not a test. - **Row names are matched by suffix**, the way `mouseover:` resolves a track, so `mane` is satisfied by a lifted view's `hub_192070_mane` and a script does not have to know the per-run hub number. - **`rows:` is a set test unless you say `ordered: true`.** `exact: true` only adds "and nothing else", so neither of them can fail because the rows came out in the wrong order. When the order is the thing under test — a quickLift target that should read in source order, a superTrack whose members should follow their priorities — add `ordered: true` and the named rows have to appear top to bottom in the order given. Rows that were not drawn at all are reported once, by `rows:`, and skipped by the order check rather than failing twice. A failure names the first pair that is inverted: step 12 (expect) failed: rows out of order: mane should be drawn above gtexGeneV8 drawn: ruler, hub_274823_quickLiftChain, hub_274823_gtexGeneV8, hub_274823_mane - **`height:` is in still pixels**, which is the CSS height times the device pixel ratio — the number someone means by "7,581 px tall". A print run makes it k times bigger, so a ceiling that passes at 1x will fail at 3x. Scale it with `scale:` or leave the check to the screen run. - **A bare number is a ceiling** (`height: 2000` means no taller than 2000), which is the check anyone actually wants. `"<1200"`, `">=300"` and `"=850"` are there when it is not. +- **`text:`, `noText:`, `has:`, `noHas:`, `rows:` and `noRows:` all take one value or a + list.** That matters most for the two text checks. A check that stringifies its argument + turns `["a", "b"]` into `"a,b"`, which no page contains, so a list handed to one that did + not accept it would pass on anything — and pass *silently*, which is worse than failing. +- **`box:` is where an element sits on the screen, and it is the only check that can ask.** + `has:`/`noHas:` say what is in the page and never where it is; `color:` reads pixels but + only inside a track's row. #38251 moved the narrow-window menu icon out of the blue bar + and slid it across the menu items, with every selector still matching and every word of + the page still there: + + expect: {box: {sel: "#topRightLinks", inside: "#main-menu-whole"}} + expect: {box: {sel: "#topRightLinks", clear: "ul.nice-menu > li > a", gap: 8}} + expect: {box: {sel: "#main-menu-whole", height: "<=40", width: ">=1000"}} + expect: {box: [{sel: "#a", inside: "#bar"}, {sel: "#b", inside: "#bar"}]} + + `inside:` wants every edge within that element's box, with `tolerance:` px of slack + (default 1) for a border that rounds the wrong way. `clear:` wants no overlap with + anything that selector matches, and `gap:` asks for that many pixels of clear space + rather than merely for no touching. `height:`/`width:` take the same comparison grammar + as the image `height:` above. **Every element `sel:` matches has to satisfy every clause**, + so `{sel: "ul.nice-menu > li", inside: "#main-menu-whole"}` is "every menu item is in the + bar"; `inside:` and `clear:` take the first match of their own selector, since a container + is one element. An element with no box at all — `display:none`, never laid out — is + skipped rather than treated as a zero-sized box at the origin, which would sit "inside" + anything. Boxes are read in document coordinates, so a page scrolled between two reads + still compares. A failure gives the measurement: + + step 2 (expect) failed: #topRightLinks is not inside #main-menu: 32px below it + -- it is at 964,114 30x32, #main-menu at 30,114 970x0 + + One viewport per run: `size:` is read once, when the browser context is made, so a script + cannot sweep widths. A layout bug that only appears at certain widths needs one script per + width. - **`color:` reads the pixels, and it is the only check that does.** A bug about color changes nothing else: #36212 draws the items of a track that sets both `itemRgb on` and `color` from the color setting rather than from the file's own RGB column, and the rows, the height, the item names and the tooltips are identical either way. It names the color a row is mostly drawn in: expect: {color: {track: crm4, is: "0,0,255"}} # the items are blue expect: {color: {track: crm4, part: label, is: "0,255,0"}} # the center label green expect: {color: {track: crm4, at: "chr1:1004500"}} # one item, not the row expect: {color: [{track: crm3, is: "0,255,0"}, {track: crm4, is: "0,0,255"}]} White is background and is not counted; everything else is, black included, since a track with no color of its own draws black items. `is:` and `not:` take `r,g,b` or `#rrggbb` and match within `tolerance:` (default 8) -- **not** a CSS color name, because trackDb's `color 0,255,0` is not CSS `green`. `at:`/`frac:`/`x:` narrow the sample to a column `wide:` px across (default 5) at that position, which is how one item in a row of several colors gets asked about on its own. A row that drew nothing at all fails saying so rather than reporting a color, and the failure always lists the top three colors with their share of the row: step 8 (expect) failed: crm4 is drawn 0,255,0, wanted 0,0,255 -- the row holds 0,255,0 (98%), 127,255,127 (1%), 190,255,190 (1%) The side labels are a separate image and are never included: "what color is this row" must not be answered by the label text. - **`warn: true`** logs the failure and carries on, for a check worth recording but not worth stopping a build over. ## Speed A full run is a real browser against a real server, and most of its wall clock is the pacing that makes the *video* watchable, not work. Measured on BP1 (15 steps, 4 stills): **64 s**, of which ~40 s is dwells, cursor animation and dropdown theatrics, ~23 s is page loads, and ~1 s is the mp4 transcode. So when you are iterating on the **figures**, skip the video: make FAST=1 BP1 # or DOCENT_FAST=1, or `fast: true` in the script FAST drops the dwells (`pace`, `shotHold`, mouseover holds), moves the cursor in one jump, skips the dropdown open/highlight, and records no video, so there is no mp4 to transcode either — **24 s** instead of 64 s for BP1. The stills carry the same content; a couple of things about them are incidental rather than identical, so build normally for the figures you publish: - the drawn cursor rests wherever the last real action left it, which is not where the skipped animation would have left it; - a `pinShot:` crop is the bounding box of the image plus the pinned tooltips, so it can come out a few pixels taller or shorter. `DOCENT_TIME=1` prints where the time actually went, slowest step first — worth a look before assuming a script is slow for a reason you can fix. `DOCENT_DERIVE=1` skips the browser altogether and prints what each `track:` step turns into — the rounds, and the cart variables in each. That is where Docent's own decisions are made, and it answers "what will this send" in about a second instead of a full run: DOCENT_DERIVE=1 node docent.js urllen.docent.yaml step 3 track {"cCREs":"hideKids"} round 1 (3 vars): cCREregistry=hide coreCcres=hide encodeCcreCombined=hide db comes from the top of the file; a script that changes assembly mid-tour is not followed, and it says so. `tests/` keeps baselines of this for a few scripts (`make derive`), which is what catches a change to the derivation that a rendered page would hide. Two more things that help: - `make -j6` builds scenarios in parallel. Each run has its own browser, context and cart, so they don't interfere; the trackDb cache is written via a temp file and renamed, so concurrent runs can't read a half-written one. - The trackDb cache holds the *derived* index (a few MB), not hubApi's reply (~30 MB for hg38), so it is quick to re-read on every run. It refreshes daily; delete `$TMPDIR/docent-tdb-*` to force it. ## Gotchas The renderer lints for one YAML trap before it opens a browser: in a flow map a colon needs a trailing space, so `{item:name5568747}` parses as the single key `"item:name5568747"` and the intended `item:` argument is silently dropped — the verb then quietly falls back to a default. Any argument key containing a `:` gets a warning. Quote genomic coordinates inside a `{...}` flow map. An unquoted comma in `chr7:155,805,900` splits the map. `goShow:` with a gene name can't just wait for the suggestion menu to appear and click a row. The box shows **Recent** positions immediately, so an early click picks a previously-visited position instead of the gene, and when the hgSuggest results arrive the menu is re-rendered, which invalidates any row picked before that. Docent therefore polls until a row actually matches the term *and* the menu has stopped changing, ranks real hits above Recent rows, and addresses the row by index at click time. ## Track names come from trackDb Name tracks exactly as trackDb does — `mane`, `clinvar`, `clinvarCnv`, `dbSnp155Common`. Docent keeps **no table** of per-track cart variables; it fetches the trackDb of the server it is driving (hubApi `/list/tracks`, cached in `$TMPDIR` for a day) and derives what a `track:` step has to send: - **containers above the track** come along — a composite or view gets the mode you asked for, a superTrack gets `show`; - **the subtrack checkbox** (`_sel`) is stated explicitly, 1 for a visible mode and 0 for `hide`, because for a composite child the checkbox is what actually decides; - **nothing is pushed downward**: a container's visibility already reaches its selected children (`clinvar: pack` draws the SNVs, the CNVs and the submitted interpretations), so a script only names its *deviations* from trackDb — e.g. `track: {clinvar: pack, clinvarCnv: hide}`; - **`hide` never propagates upward** — hiding one subtrack must not turn its composite off. A step that names both a composite and something under it is applied in **two requests**, container first. hgTracks reshapes a composite when its container visibility changes, and that wipes per-subtrack overrides arriving in the same request (`clinvar=pack&clinvarCnv=hide` alone leaves `clinvarCnv_sel=1` and the CNV row still drawn). Names trackDb doesn't know — attached hubs, custom tracks, and everything on a quickLift target (a hub genome, which is not in the listing at all) — fall back to a literal `name=mode`, which is all Docent can honestly do for them. Set `DOCENT_ROWS=1` to log the rows hgTracks actually drew after each step (and the suggestion menus a `goShow:`/`convert:` search saw). That answers "why is that subtrack still there" and "why is my `mouseover:` track not shown" directly, instead of by guessing: DOCENT_ROWS=1 make BP1 track: clinvar=pack rows: ruler, mane, clinvarMain, clinvarCnv, clinvarSubLolly track: clinvarCnv=hide clinvarCnv_sel=0 rows: ruler, mane, clinvarMain, clinvarSubLolly It does the same for `mouseover:` — which map boxes matched the name, where the cursor went, the tooltip that was expected from the box, and the one that actually came up: item "4.3.157828209.157828210" in quickLiftChain: 1 map box(es) match (1 in row), centers x=[288] -> hovering (288,216), expecting tip "mismatch A->C" tip at (288,216): "mismatch A->C" **After a Convert/quickLift** the target's tracks are served from a hub, so their DOM ids gain a dynamic `hub__` prefix (e.g. `hub_191568_quickLiftChain`) that changes every run. `mouseover:` resolves the track by **suffix**, so you still just write the plain name — e.g. `track: quickLiftChain` (the Alignment Differences track) hits `hub_191568_quickLiftChain` automatically. ## Where this is headed Today a step reaches the browser as CGI variables — `hideTracks=1&clinvar=pack&...` — which is why the verbs have to know each composite's parameter set by heart. Once the cart can be handed a structured blob (#37838, "Store cart contents as JSON"), a Docent step can name the state it wants instead of assembling a pile of `_sel=1` variables, and the shortcut table above mostly goes away. Docent is meant to be an early consumer of that work. The same reason argues for the tour, not the video, being the durable artifact: a script that describes a state and a route through it can just as well drive a live in-browser tutorial as an mp4. ## Scripts in the wild The Current Protocols quickLift paper keeps six of these (`BP1`, `AP1`–`AP3`, `SP1`, `SP2`) in its own repo, under `currentProtocols/figures/scripts/`, and builds them with the `docent.mk` rules here. `BP1.docent.yaml` is a good one to read first: gateway → hide all → set two tracks → drag-select and zoom → Convert with quickLift → open the lifted view → pin two tooltips into one figure → click through to hgc and out to dbSNP.