679b48855a32f8ea1e1ea8ca3c095a0c2c7f6c7a
max
  Tue Sep 8 10:08:41 2026 -0700
detailsScript: add a scatterPlot plot type, and use it for pcLAI

Clicking a pcLAI window now shows where that window sits in the ancestry space
it was placed in: a scatterplot of the 1000 Genomes reference haplotypes with
the window's own PCA coordinate and its segment's coordinate marked on it. The
numbers were already on the details page and told a reader almost nothing.

New plot type scatterPlot (hg/js/hgc.scatterPlot.js), driven the same way as
histogram. Background points come from a JSON or TSV file named by dataUrl and
may carry a category, which colors them and builds a legend, and a label, which
is shown on mouseover. The cloud is drawn on a canvas, since these files hold
thousands of points and that many <circle> elements make the page crawl; axes
and the highlighted points stay SVG on top. Point lookup for the mouseover goes
through a cell index so a large file stays smooth.

Two additions serve every plot type, not just this one:

- exportFields, a config key listing further bigBed fields whose values are
passed to the module as a fieldValues object. Without it a plot needing two
coordinates would need them packed into one field, and pcLAI keeps them in
pca and pcaSegment. Only fields that exist in the bigBed are exported, at
most 32, and the JSON types are checked rather than asserted because
jsonListVal and jsonStringVal errAbort and this JSON is written by a hub.

- a config key ending in Url is treated as a file, by the convention
trackSettingIsFile() already uses, and a relative one is resolved against the
track's own bigDataUrl. The module does not fetch it directly; it asks
hgTrackUi for it, the route facetedComposite uses for its metadata. That
checks the canonicalized path against the hubs on the cart and reads it with
udc, so a hub-relative path works even for a hub loaded from a local path
(the GenArk /gbdb hubs), no CORS header is needed, and a file outside a
connected hub cannot be read. Verified that /etc/passwd, file://, a dot-dot
escape, an unattached hub and an unrelated host are all refused with 400.
When the session has file caching off, hgc now exports udcTimeout the way
hgTrackUi does and the module POSTs, so the browser cannot answer from cache.

Fixes a crash reachable from any hub: "detailsScript.<plotType>.<field> null"
segfaulted hgc, because jsonObjectVal returns NULL for a JSON null and the hash
routines dereference it. This hit the shipped histogram type too.

trackDbSettingsGen.py stopped reading a setting's description at the first
"Example:" paragraph and never read <ul> at all, so it dropped everything after
the first example and every list item. That silently truncated 226 of the 264
descriptions, including spectrum's minGrayLevel/scoreMin/scoreMax bullets, and
would have dropped this whole scatterPlot section. It now skips the Example
label instead of stopping, and folds list items in. No setting loses a word and
none gains or loses an example.

pcLAI wiring: the background file is the authors' published reference panel
(github.com/AI-sandbox/hprc-pclai reference_pca_metadata.tsv), converted by
hprc2annotMakePclaiRefPanel.py -- 3122 haplotypes, 21 populations, 94 KB, one
file for the collection since it is the reference space rather than per-assembly
data. The four values pcaSegment takes across all 460 assemblies turn out to be
the four continental cluster centres, so the highlighted segment dot always
lands on one of them.

genark: addContrib now rewrites a "...Url" inside a detailsScript value the same
way it rewrites bigDataUrl, and symlinks the collection's shared root-level data
files next to the docs, so contrib/<name>/<file> resolves in the deeper GenArk
layout. It writes the alpha tier only, leaving the assembly's default hub alone,
and clears any unmarked copy of the collection's stanzas that the assembly build
baked in, which would otherwise leave the hub declaring each track twice.

refs #35415

diff --git src/hg/htdocs/goldenPath/help/trackDb/trackDbSettingsGen.py src/hg/htdocs/goldenPath/help/trackDb/trackDbSettingsGen.py
index 3a3e857e1d6..cd6c6795498 100644
--- src/hg/htdocs/goldenPath/help/trackDb/trackDbSettingsGen.py
+++ src/hg/htdocs/goldenPath/help/trackDb/trackDbSettingsGen.py
@@ -1,302 +1,317 @@
 #!/usr/bin/env python3
 """Generate machine-readable trackDb settings data from the trackDb help docs.
 
 trackDbLibrary.shtml gives each setting's name, types, description and example.
 trackDbHub.v3.html says which ones hubs support, and at what level, by which table the
 row sits in.
 
   --import  read those two pages, write trackDbSettings.yaml
   --json    read trackDbSettings.yaml, write trackDbSettings.json
 
 Run "make settings" after editing the docs. Do not hand-edit the yaml: --import rewrites
 it, so corrections go in this script. Longer term the yaml should be the source the reader
 HTML is built from too; today it only feeds the json.
 """
 import sys
 import os
 import re
 import json
 import argparse
 from bs4 import BeautifulSoup
 import yaml
 
 HERE = os.path.dirname(os.path.abspath(__file__))
 HUB_VERSION = "v3"
 LIB = os.path.join(HERE, "trackDbLibrary.shtml")
 HUB = os.path.join(HERE, "trackDbHub.%s.html" % HUB_VERSION)
 YAML_OUT = os.path.join(HERE, "trackDbSettings.yaml")
 JSON_OUT = os.path.join(HERE, "trackDbSettings.json")
 
 # Stanzas a setting can appear in. hub.txt and genomes.txt are stanzas of their own, so a
 # trackDb "common" setting spans the four track roles, not all six.
 ROLE_ORDER = ["hub", "genome", "super", "composite", "view", "leaf"]
 ALL_ROLES = ["super", "composite", "view", "leaf"]
 
 # Doc sections named for their anchor because the keyword collides with a trackDb setting
 # of the same name. The keyword is what goes in a hub file, so map them back.
 DOC_NAME_TO_KEYWORD = {
     "hubShortLabel": "shortLabel",
     "hubLongLabel": "longLabel",
     "hubGenome": "genome",
     "trackDbFile": "trackDb",
     "groupsFile": "groups",
     "type_for_hubs": "type",
 }
 
 # The docs use HTML entities that decode to non-ASCII punctuation. Fold them: "format" is
 # a syntax template someone pastes into a hub file, and the generated files are ASCII.
 UNICODE_TO_ASCII = {
     "\u00a0": " ",     # nbsp
     "\u2013": "-",     # ndash
     "\u2014": "-",     # mdash
     "\u2018": "'",     # lsquo
     "\u2019": "'",     # rsquo
     "\u201c": '"',     # ldquo
     "\u201d": '"',     # rdquo
     "\u2026": "...",   # hellip
 }
 
 
 def toAscii(text):
     """Fold the docs' punctuation to ASCII, warning about anything left over."""
     for uni, plain in UNICODE_TO_ASCII.items():
         text = text.replace(uni, plain)
     extra = sorted(set(c for c in text if ord(c) > 126))
     if extra:
         print("warning: no ASCII spelling for %s, add it to UNICODE_TO_ASCII" %
               ", ".join("%r %s" % (c, hex(ord(c))) for c in extra), file=sys.stderr)
     return text
 
 
 def prettify(tableId):
     """Turn a settingsTable id into a human category label."""
     return tableId.replace("_-_", " - ").replace("_", " ").strip()
 
 
 def roleHints(tableId):
     """Guess which stanzas a table's settings apply to, from its id."""
     t = tableId.lower()
     if "supertrack" in t:
         return ["super"]
     if "view" in t and "composite" in t:
         return ["view"]
     if "composite" in t or "faceted" in t or "subgroup" in t:
         return ["composite"]
     if "aggregate" in t or "overlay" in t:
         return ["composite", "view", "leaf"]
     if "common" in t:
         return list(ALL_ROLES)
     if "hub_file" in t:
         return ["hub"]
     if "genomes_file" in t:
         return ["genome"]
     if "deprecated" in t:
         return []  # unknown, doImport falls back to the types
     return ["leaf"]  # type-specific tables (bam, bigBed, bigWig, vcfTabix, ...)
 
 
 def contextOf(tableId):
     """Which file a table's settings go in: hub.txt, genomes.txt, or a trackDb file."""
     t = tableId.lower()
     if "genomes_file" in t:
         return "genomes"
     if "hub_file" in t:
         return "hub"
     return "trackDb"
 
 
 def firstSentence(text, cap=160):
     """First sentence, length-capped, for the one-line summary."""
     text = re.sub(r"\s+", " ", text).strip()
     if not text:
         return ""
     m = re.search(r"\.(\s|$)", text)
     s = text[:m.start() + 1] if m else text
     if len(s) > cap:
         s = s[:cap - 3].rstrip() + "..."
     return s
 
 
 def parseLibrary():
     """name -> {types, format, required, description, summary, examples}."""
     with open(LIB, encoding="utf-8") as f:
         soup = BeautifulSoup(f.read(), "html.parser")
     blurbs = {}
     for span in soup.find_all("span", class_="types"):
         div = span.parent
         classes = div.get("class") or []
         if not classes:
             continue
         name = classes[0]
         if name.endswith("_intro") or name.endswith("_example"):
             continue
         types = [c for c in span.get("class", []) if not c.startswith("types")]
         if not types:
             types = ["all"]
         fmtDiv = div.find("div", class_="format")
         fmtCode = fmtDiv.find("code") if fmtDiv else None
         fmt = toAscii(fmtCode.get_text(" ", strip=True)) if fmtCode else name
 
         req = div.find("p", class_="isRequired")
         reqText = req.get_text(" ", strip=True).lower() if req else ""
         required = "yes" in reqText or "for hubs" in reqText
 
+        # Walk p and ul in document order. Stopping at the first "Example" paragraph
+        # would lose every later paragraph, which matters for a setting that documents
+        # more than one variant, so the label is skipped rather than ended on. List
+        # items carry real content, so <ul> is folded in as well.
         descParts = []
-        for p in div.find_all("p"):
-            if "isRequired" in (p.get("class") or []):
+
+        def addPart(txt, prefix=""):
+            # The source wraps paragraphs over several lines, so squeeze each block to
+            # one line; blocks are then joined by newlines to keep them apart.
+            txt = re.sub(r"\s+", " ", txt).strip()
+            if txt:
+                descParts.append(prefix + txt)
+
+        for el in div.find_all(["p", "ul"]):
+            if el.name == "ul":
+                for li in el.find_all("li"):
+                    addPart(li.get_text(" ", strip=True), "- ")
                 continue
-            txt = p.get_text(" ", strip=True)
+            if "isRequired" in (el.get("class") or []):
+                continue
+            txt = el.get_text(" ", strip=True)
             if txt.lower().startswith("example"):
-                break
-            if txt:
-                descParts.append(txt)
-        description = toAscii(re.sub(r"\s+", " ", " ".join(descParts)).strip())
+                continue
+            addPart(txt)
+        description = toAscii("\n".join(descParts).strip())
         examples = []
         for pre in div.find_all("pre"):
             ex = toAscii(pre.get_text().strip())
             if ex:
                 examples.append(ex)
 
         blurbs[name] = {
             "types": types,
             "format": fmt,
             "required": required,
             "description": description,
             "summary": firstSentence(description),
             "examples": examples,
         }
     return blurbs
 
 
 def parseHubSpec():
     """Walk the hub spec tables in order.
 
     Returns (order, home, hints): setting names in document order, home[name] =
     {category, context, level}, hints[name] = set of roles.
     """
     with open(HUB, encoding="utf-8") as f:
         soup = BeautifulSoup(f.read(), "html.parser")
     # Scan the whole document rather than the #specification subtree: html.parser
     # closes that div early at a malformed tag, hiding the later container tables.
     order = []
     home = {}
     hints = {}
     for table in soup.find_all("table", class_="settingsTable"):
         tid = table.get("id", "")
         if tid == "toc":
             continue
         category = prettify(tid)
         context = contextOf(tid)
         tHints = roleHints(tid)
         for td in table.find_all("td"):
             classes = td.get("class") or []
             if not classes:
                 continue
             name = classes[0]
             code = td.find("code")
             if code is None:
                 continue
             level = None
             for c in (code.get("class") or []):
                 if c.startswith("level-"):
                     level = c[len("level-"):]
             if name not in home:
                 order.append(name)
                 home[name] = {"category": category, "context": context, "level": level}
                 hints[name] = set()
             hints[name].update(tHints)
     return order, home, hints
 
 
 def doImport():
     blurbs = parseLibrary()
     order, home, hints = parseHubSpec()
     entries = []
     missing = []
     for name in order:
         b = blurbs.get(name)
         if b is None:
             missing.append(name)
             continue
         info = home[name]
         roles = [r for r in ROLE_ORDER if r in hints[name]]
         if not roles and info["context"] == "trackDb":
             if b["types"] == ["all"]:
                 # A copy, or safe_dump writes yaml aliases to one shared list.
                 roles = list(ALL_ROLES)
             else:
                 roles = ["leaf"]
         entries.append({
             "name": name,
             "types": b["types"],
             "roles": roles,
             "category": info["category"],
             "context": info["context"],
             "level": info["level"],
             "required": b["required"],
             "summary": b["summary"],
             "description": b["description"],
             "format": b["format"],
             "examples": b["examples"],
         })
 
     with open(YAML_OUT, "w", encoding="utf-8") as f:
         f.write("# Generated by trackDbSettingsGen.py --import, which rewrites the whole file.\n")
         f.write("# Edit the trackDb help docs instead, then run 'make settings'.\n")
         f.write("# 'roles' is a guess from which doc table a setting sits in; check the\n")
         f.write("# container-only ones.\n\n")
         yaml.safe_dump(entries, f, sort_keys=False, width=100)
     print("imported %d settings -> %s" % (len(entries), YAML_OUT))
     if missing:
         print("  (%d hub rows had no library blurb, skipped): %s" %
               (len(missing), ", ".join(missing)))
 
 
 def doJson():
     with open(YAML_OUT, encoding="utf-8") as f:
         entries = yaml.safe_load(f)
     categories = []
     settings = []
     for e in entries:
         if not e["roles"]:
             continue
         cat = e["category"]
         if cat not in categories:
             categories.append(cat)
         example = e["examples"][0] if e["examples"] else ""
         # "fmt" is the syntax to fill in, "ex" the doc's example of it filled in.
         settings.append({
             "key": DOC_NAME_TO_KEYWORD.get(e["name"], e["name"]),
             "category": cat,
             "roles": e["roles"],
             "types": "all" if e["types"] == ["all"] else e["types"],
             "level": e["level"],
             "fmt": e["format"],
             "ex": example,
             "desc": e["summary"],
         })
     out = {"version": HUB_VERSION, "categories": categories, "settings": settings}
     with open(JSON_OUT, "w", encoding="utf-8") as f:
         json.dump(out, f, indent=1)
         f.write("\n")
     print("wrote %d settings in %d categories -> %s" %
           (len(settings), len(categories), JSON_OUT))
 
 
 def main():
     ap = argparse.ArgumentParser(description=__doc__)
     ap.add_argument("--import", dest="doImport", action="store_true",
                     help="write trackDbSettings.yaml from the help docs")
     ap.add_argument("--json", dest="doJson", action="store_true",
                     help="write trackDbSettings.json from trackDbSettings.yaml")
     args = ap.parse_args()
     if not (args.doImport or args.doJson):
         ap.error("specify --import and/or --json")
     if args.doImport:
         doImport()
     if args.doJson:
         doJson()
 
 
 if __name__ == "__main__":
     main()