b066328906fe2c8fbbfada16f18e8cc2fb9dc1cf braney Sat Sep 12 09:58:46 2026 -0700 sessionCartAudit: match a catalog row that names a whole cart variable, refs #37979 peel() only ever considered suffixes that begin after a separator, so a catalog row anchored on a fixed prefix could never match the variable it describes: by the time the walk reached a separator, the hgta_ that anchors hgta_fs.check.<db>.<table>.<field> was gone. A plain literal row was hit just as hard, since dbRIP.genoRegion could only be tested as its own tail, "genoRegion". 4,299 hgta_ names were reported as covered by nothing but a catch-all because of it. The #37838 catalog already says which kind of row it is and the audit was throwing that away. A row whose separator is "." or "_" names a suffix that follows a track name; anything else names the whole cart variable, with the separator in front - "" for hgTables and the old per-dataset variables, cgs_<track>_ for chromGraph. trackVarNames() now returns the separator with the name, the whole-variable rows are compiled apart, and peel() tries the whole name before walking suffixes, so the longer and correct match wins. Two shorthands the catalog already uses are read rather than expanded by hand: a comma list is several variables sharing one description, and a trailing * is a family. A row that is prose rather than one pattern can be matched by nothing, so --check names it instead of letting it count for nothing; there is one today. The bare wildcard lists are sorted, because they were built by walking a set and two runs of a published report diffed for no reason. Over 6,631 saved sessions this moves 2,016 names: 1,990 out of the catch-all bucket, 25 out of the bare-track-name bucket (filter text boxes whose stored value is empty, which the visibility heuristic had been reading as track names), and tfbsConsSitesCutoff out of unknown. Nothing leaves the catalogued buckets. 1,208 hgta_fs.check names now match the row that describes them rather than <wigTrack>.<wigVar> or <filterName>Type. diff --git src/hg/utils/sessionCartAudit/sessionCartAudit.py src/hg/utils/sessionCartAudit/sessionCartAudit.py index 25af3cf282a..657d4dbe193 100755 --- src/hg/utils/sessionCartAudit/sessionCartAudit.py +++ src/hg/utils/sessionCartAudit/sessionCartAudit.py @@ -1,985 +1,1081 @@ #!/usr/bin/env python3 """sessionCartAudit.py - check the cart catalogs against the sessions people saved. Refs #37838 and #37923. Third sibling of hg/utils/cartTrackVarCatalog (which catalogs track-scoped cart variables) and hg/utils/urlCommandCatalog (which catalogs URL parameters). Both of those were built by reading the source. This one reads the other direction: it pulls every named session out of namedSessionDb, takes apart the stored var=val blob, and asks which of the names in it neither catalog can account for. Why it is worth having. A catalog derived from the source can only be as complete as the greps that built it. It cannot see a variable that only JavaScript writes, it cannot see a name assembled at run time from a track name plus a field name, and it cannot tell you which of the leaks it predicted are actually sitting in somebody's saved session. The sessions can. They are a transcript of everything the browser has ever been asked to remember. cartTrackVarCatalog.py source -> what a track variable may be called urlCommandCatalog.py source -> what may go on a URL sessionCartAudit.py data -> what is actually in the carts, and which of it the other two do not cover The catch, and the reason the matching below is fussier than it looks: the track catalog contains bare wildcard entries (<species>, <filterName>, <wigTrack>.<wigVar>, <attribute><Value>, <view>.<anyTypeVar>, _<origin>) that match literally any token. Score them as matches and every unrecognised name in the corpus is absorbed, the audit comes back clean, and it has proved nothing. They are matched separately and reported as "wildcard only", which is the honest answer: covered by a catch-all, not actually catalogued. -Matching is right-anchored - longest known suffix at a '.' or '_' boundary - -never left-anchored at the first separator. That is not a style choice. Track -and species names in the real data contain dots (GenArk accessions such as -GCF_020740605.2), slashes, spaces and parentheses, so the left-hand side cannot -be delimited by anything except a complete list of what the right-hand side may -be. +Matching a track-scoped name is right-anchored - longest known suffix at a '.' +or '_' boundary - never left-anchored at the first separator. That is not a +style choice. Track and species names in the real data contain dots (GenArk +accessions such as GCF_020740605.2), slashes, spaces and parentheses, so the +left-hand side cannot be delimited by anything except a complete list of what +the right-hand side may be. + +Not every catalog row is a suffix, though, and that is the other half of the +matching. A row's separator says which it is: "." or "_" means the name +follows a track name, and anything else means the row already spells out the +whole cart variable, as hgTables' rows do (hgta_fs.check.<db>.<table>.<field>) +and as chromGraph's do with a prefix in front (cgs_<track>_pixels). A whole +name has to be matched whole. Testing only suffixes leaves a left-anchored row +unable to match the variable it describes, because the prefix that anchors it +is gone before the first candidate is cut - which is why the whole name is +tried first, and why for a long time 4,299 hgta_ names were reported as covered +by nothing but a catch-all. Usage: sessionCartAudit.py --check # coverage counts sessionCartAudit.py --unknown # names neither catalog knows sessionCartAudit.py --wildcard # names only a catch-all matches sessionCartAudit.py --leaks # #37923 leak claims vs the data sessionCartAudit.py --findings # the curated findings, with live counts sessionCartAudit.py --json out.json sessionCartAudit.py --html out.html --dump FILE read a previously saved contents dump instead of the db --save-dump F write the raw dump so later runs can skip the query --central DB hgcentral database to read (default from hg.conf) Reading the sessions takes a minute and a couple of hundred MB of query output, so --save-dump / --dump is worth using while iterating. """ import argparse import collections import html import importlib.util import json import os import re import subprocess import sys import urllib.parse HERE = os.path.dirname(os.path.abspath(__file__)) UTILS = os.path.dirname(HERE) VIS_VALUES = {"hide", "dense", "squish", "pack", "full", "show", ""} PLACEHOLDER = re.compile(r"<[a-zA-Z]+>") # ---------------------------------------------------------------- the sessions def defaultCentral(): """Read central.db out of ~/.hg.conf, the same place the CGIs get it.""" path = os.path.expanduser("~/.hg.conf") try: for line in open(path): if line.startswith("central.db="): return line.split("=", 1)[1].strip() except IOError: pass return "hgcentraltest" def dumpSessions(central): """One row per named session, the raw contents blob. mysql batch mode escapes embedded newlines, so a row is a line. The blob itself is CGI-encoded, so & and = inside a value arrive as %26 and %3D and the split below is safe.""" cmd = ["hgsql", "-N", "-B", central, "-e", "select contents from namedSessionDb"] out = subprocess.run(cmd, stdout=subprocess.PIPE, check=True).stdout return out.decode("utf8", "replace") def extract(text, sampleMax=4): """text -> {name: [sessionCount, totalCount, [sample values]]}, sessionCount.""" counts = collections.Counter() inSessions = collections.Counter() samples = collections.defaultdict(set) nSess = 0 for line in text.split("\n"): if not line: continue nSess += 1 seen = set() for pair in line.split("&"): if not pair: continue rawName, _, rawVal = pair.partition("=") name = urllib.parse.unquote(rawName.replace("+", " ")) counts[name] += 1 seen.add(name) if len(samples[name]) < sampleMax: samples[name].add(urllib.parse.unquote(rawVal.replace("+", " "))[:60]) for name in seen: inSessions[name] += 1 names = {} for name, total in counts.items(): names[name] = [inSessions[name], total, sorted(samples[name])] return names, nSess # ---------------------------------------------------------------- the catalogs def loadSibling(subdir, module): """Import a sibling catalog by path so this works from any cwd.""" path = os.path.join(UTILS, subdir, module + ".py") if not os.path.exists(path): sys.exit("cannot find sibling catalog %s\n" "expected it next door at %s" % (module, path)) spec = importlib.util.spec_from_file_location(module, path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) return mod def trackVarNames(): - """Every variable name in the #37838 catalog, however deeply nested.""" + """Every variable in the #37838 catalog, however deeply nested. + + Returns the bare names and the (name, sep) rows. The separator matters: + it is what says whether a row names a suffix that follows a track name + ("." or "_") or a whole cart variable ("" for hgTables and the old + per-dataset variables, "cgs_<track>_" for chromGraph, which puts its + prefix in front). peel() needs both, and a set of bare names cannot tell + them apart.""" mod = loadSibling("cartTrackVarCatalog", "cartTrackVarCatalog") cat = mod.build() - names = set() + names, rows = set(), [] def walk(node): if isinstance(node, dict): name = node.get("name") if isinstance(name, str) and ("type" in node or "src" in node): names.add(name) + rows.append((name, node.get("sep", "."))) for value in node.values(): walk(value) elif isinstance(node, list): for value in node: walk(value) walk(cat) - return names + return names, rows def urlCommandNames(): """Every parameter name in the #37923 catalog, plus the ones it calls leaks.""" mod = loadSibling("urlCommandCatalog", "urlCommandCatalog") cat = mod.build() names, leaks = set(), set() def walk(node): if isinstance(node, dict): name = node.get("name") if isinstance(name, str) and ("kind" in node or "value" in node): names.add(name) if node.get("leaks"): leaks.add(name) for value in node.values(): walk(value) elif isinstance(node, list): for value in node: walk(value) walk(cat) return names, leaks def compilePatterns(catVars): """Split the catalog into literals, real families and bare wildcards. A family pattern still has something to anchor on once the <placeholders> come out (filter.<field> keeps "filter."). A bare wildcard does not (<species> keeps nothing), so it would match any token at all and has to be kept out of the ordinary match path.""" literals, families, wildcards = set(), [], [] for var in catVars: if not PLACEHOLDER.search(var): literals.add(var) continue parts = PLACEHOLDER.split(var) regex = re.compile("^" + "[^.]+".join(re.escape(p) for p in parts) + "$") residue = PLACEHOLDER.sub("", var).strip("._") if len(residue) < 3: wildcards.append((var, regex)) else: families.append((var, regex)) return literals, families, wildcards +# A catalog row may stand for more than one variable. Both shorthands are +# already in use in the #37838 catalog and both are machine-readable, so the +# audit reads them rather than making a person expand them: a comma list is +# several variables that share one description, and a trailing * is a family +# whose members are listed in the note. +GLOB = re.compile(r"<[a-zA-Z]+>|\*") + + +def absRegex(pattern): + """Compile a whole-variable pattern. <placeholder> is one token, * is any.""" + out, pos = [], 0 + for match in GLOB.finditer(pattern): + out.append(re.escape(pattern[pos:match.start()])) + out.append("[^.]*" if match.group(0) == "*" else "[^.]+") + pos = match.end() + out.append(re.escape(pattern[pos:])) + return re.compile("^" + "".join(out) + "$") + + +def absolutePatterns(catRows): + """The rows that spell out a whole cart variable rather than a suffix. + + A row whose sep is "." or "_" describes what follows a track name, which is + what peel()'s suffix walk is for. Every other sep means the row already + names the variable as it appears in the cart, and the sep goes in front of + it: sep "" leaves the name alone, and chromGraph's "cgs_<track>_" prefixes + it. These can only ever be matched against the whole name. + + Returns (literals, families, skipped). A row that is prose rather than a + pattern cannot be matched by anything and is returned in skipped, so it can + be reported instead of quietly counting for nothing.""" + literals, families, skipped = set(), [], [] + for name, sep in catRows: + if sep in (".", "_"): + continue + for part in name.split(","): + full = (sep + part.strip()).strip() + if not full: + continue + if " " in full: + skipped.append(full) + elif GLOB.search(full): + families.append((full, absRegex(full))) + else: + literals.add(full) + return literals, families, skipped + + # ---------------------------------------------------------------- classify class Audit(object): def __init__(self, names, nSess): self.names = names self.nSess = nSess - self.catVars = trackVarNames() + self.catVars, catRows = trackVarNames() self.urlNames, self.catLeaks = urlCommandNames() self.literals, self.families, self.wildcards = compilePatterns(self.catVars) + self.absLiterals, self.absFamilies, self.absSkipped = absolutePatterns(catRows) # Track-name vocabulary learned from the corpus itself: a bare name # whose every observed value is a visibility word is a track. self.trackNames = set(n for n, v in names.items() if v[2] and all(x in VIS_VALUES for x in v[2])) self.buckets = collections.defaultdict(list) self.wildcardSuffix = collections.Counter() self._dbs = None self.classify() def sess(self, name): return self.names[name][0] if name in self.names else 0 def dbVocabulary(self): """Assembly names, learned from the position.<db> variables in the corpus. Every session that has ever been at an assembly leaves one, so this is a better list than anything hard-coded, and it comes from the same data being audited.""" if self._dbs is None: self._dbs = set(n.split(".", 1)[1] for n in self.names if n.startswith("position.") and n.count(".") == 1) return self._dbs def famMatch(self, suffix, patterns): for src, regex in patterns: if regex.match(suffix): return src return None def peel(self, name): - """Longest catalogued suffix at a separator. -> (stem, suffix, how)""" + """Longest catalogued pattern covering the name. -> (stem, suffix, how) + + The whole name is tried first, then the longest suffix at a separator. + The whole-name step is not an optimisation: every candidate the suffix + walk considers begins after a separator, so a row that is left-anchored + on a fixed prefix can never match the variable it describes. + hgta_fs.check.<db>.<table>.<field> is the clearest case - by the time + the walk reaches a separator the hgta_ that anchors the row is gone - + and so is a plain literal like dbRIP.genoRegion, which without this + could only ever be tested as its own tail, "genoRegion".""" + if name in self.absLiterals: + return "", name, "literal" + hit = self.famMatch(name, self.absFamilies) + if hit: + return "", name, hit fallback = None for i in range(len(name) - 1): if name[i] not in "._": continue suffix = name[i + 1:] if suffix in self.literals or (name[i] + suffix) in self.literals: return name[:i], suffix, "literal" hit = self.famMatch(suffix, self.families) if hit: return name[:i], suffix, hit if fallback is None: hit = self.famMatch(suffix, self.wildcards) if hit: fallback = (name[:i], suffix, "wildcard:" + hit) return fallback def classify(self): for name in self.names: if name in self.urlNames: self.buckets["urlKnown"].append(name) continue peeled = self.peel(name) if peeled and not peeled[2].startswith("wildcard:"): self.buckets["trackKnown"].append(name) continue if name in self.trackNames: self.buckets["trackVis"].append(name) continue if peeled: self.buckets["wildcardOnly"].append(name) self.wildcardSuffix[peeled[1]] += 1 continue self.buckets["unknown"].append(name) def counts(self): return collections.OrderedDict( (k, len(self.buckets[k])) for k in ("trackVis", "trackKnown", "wildcardOnly", "urlKnown", "unknown")) def leakReport(self): seen = sorted(((self.sess(n), n) for n in self.catLeaks if n in self.names), reverse=True) unseen = sorted(self.catLeaks - set(self.names)) return seen, unseen def sorted_(self, bucket): rows = [(self.names[n][0], self.names[n][1], n, self.names[n][2]) for n in self.buckets[bucket]] rows.sort(reverse=True) return rows # ---------------------------------------------------------------- findings # # The prose is curated, the numbers are not. Each finding carries a probe that # recomputes its counts from the corpus, so the write-up cannot drift away from # the data the way a hand-typed number would. Same division of labour as the # two sibling catalogs: a human says what it means, the script says how much. def nameStats(audit, pred): hits = [n for n in audit.names if pred(n)] if not hits: return {"names": 0, "sessions": 0, "busiest": None, "busiestSess": 0} # tie-break on the name so repeated runs report the same example busiest = max(hits, key=lambda n: (audit.names[n][0], n)) return {"names": len(hits), "sessions": sum(audit.names[n][0] for n in hits), "busiest": busiest, "busiestSess": audit.names[busiest][0]} def dbSuffixFamily(audit, var): """Names of the form <var>_<db>. The suffix has to be checked against a real assembly list. Left to just startswith(), hgPcrResult_imgOrd counts as an assembly named imgOrd, which is the very ambiguity this finding is about.""" dbs = audit.dbVocabulary() return nameStats(audit, lambda n: (n.startswith(var + "_") and n[len(var) + 1:] in dbs)) def innerPositionCarts(audit, text=None): """The var=val blob stored inside the value of position.<db>.""" inner = collections.Counter() forms = collections.Counter() dbs = set() if text is None: return inner, forms, dbs for match in re.finditer(r"position\.([A-Za-z0-9_]+)=([^&]*)", text): dbs.add(match.group(1)) value = urllib.parse.unquote(match.group(2)) if "=" not in value: forms["bare position"] += 1 continue forms["nested cart"] += 1 for pair in value.split("&"): inner[pair.partition("=")[0]] += 1 return inner, forms, dbs def labelFields(audit): fields = collections.Counter() for name in audit.names: match = re.search(r"\.label\.([A-Za-z0-9_]+)$", name) if match: fields[match.group(1)] += audit.names[name][0] catalogued = set(v.split(".", 1)[1] for v in audit.catVars if v.startswith("label.") and "<" not in v) return fields, catalogued NOISE_CLASSES = [ ("Form buttons", "The name of the button the user clicked, saved as cart state.", lambda n: re.search(r"(Submit|Button)$", n) is not None), ("HTML-escaped ampersands", "A link copied out of an email or a web page, so & became part of the " "next parameter's name. These reach inside the nested position.<db> blob " "too, so the damage is stored two levels down.", lambda n: n.startswith("amp;")), ("Question mark glued to a name", "A second ? in the URL, or a hand-built link that lost its separator.", lambda n: "?" in n), ("Names containing a space", "Usually a track or species name with a stray space in it, which no " "left-to-right parse survives.", lambda n: " " in n), ("Non-ASCII bytes in the name", "Corruption rather than input.", lambda n: any(ord(ch) > 126 for ch in n)), ("Empty name", "A bare = in the query string.", lambda n: n == ""), ] TYPOS = ["measureTIming", "meassureTiming", "measureTiminjf", "udcTimout", "udcTimeour", "ucbTimeout", "hdcTimout", "UDCtimeout", "emAltHighligh", "hgTracksCOnfigurePage", "ignoreCookies", "intereraction", "executeQury", "disableAdvancedJavascript", "hgTracksConfigurePage"] # Global cart variables that belong to no track and are not documented URL # parameters. Neither catalog has a section for these. Grouped by what they # are for, because the groups behave differently: the first is user preference # worth sharing, the second is view state the browser recomputes anyway, the # last is one-request form scratch that should never have been saved. GLOBAL_GROUPS = [ ("Configure page display settings", "hg/hgTracks/config.c, read back in hgTracks.c:10620ff", ["leftLabels", "centerLabels", "trackControlsOnMain", "nextItemArrows", "nextExonArrows", "exonNumbers", "ideogram", "enableHighlightingDialog", "showDinkButtons", "doHgcInPopUp", "textFont", "textStyle", "fontType", "lineWidth", "tooltipTextSize", "theme", "displaySubtracks"]), ("Position and navigation state", "hg/hgTracks/hgTracks.c", ["lastPosition", "oldPosition", "dinkL", "dinkR", "rulerBaseZoom", "newWinWidth", "winStart", "winEnd", "chromName", "insideX", "rulerClickHeight", "dragSelection", "dragZooming", "prevHlColor"]), ("Multi-region state", "hg/hgTracks/hgTracks.c, hg/lib/web.c", ["virtMode", "virtModeType", "lastVirtModeType", "lastVirtModeExtraState", "emPadding", "emAltHighlight", "gmPadding", "singleAltHaploId", "virtWinFull", "autoRearr"]), ("Hub state", "hg/lib/hubConnect.c", ["trackHubs", "assumesHub", "hubText", "urlHub", "hubSearchTerms", "hubDbFilter", "tsIncludePublicHubs"]), ("Track search", "hg/hgTracks/searchTracks.c", ["tsGroup", "tsName", "tsDescr", "tsCurTab", "tsSimple", "tsType", "tsSort"]), ("Housekeeping", "hg/lib/cart.c, hg/inc/cart.h", ["defaultsSet", "cartVersion", "hgS_DataTableState", "hgPS_DataTableState", "sessionTable_length"]), ] # The four ways the tree scopes a variable to an assembly. None of them is in # either catalog, and the first cannot be told apart from the legacy # <track>_<var> form by parsing - only by knowing which half is the variable. DB_SCOPES = [ ("<var>_<db>", "hg/inc/cart.h:49 _cartVarDbName()", "ctfile_hg38"), ("position.<db>", "hg/lib/cart.c:4076 cartSetDbPosition()", "position.hg38"), ("customComposite-<db>", "hg/lib/sessionData.c:452", "customComposite-hg38"), ("hubQuickLift-<db>", "hg/inc/quickLift.h:9, sessionData.c:452", "hubQuickLift-hg38"), ] DB_SUFFIX_VARS = [ ("ctfile", "pointer to the user's custom track file for that assembly " "(customTrack.c:918); there is a hub form too, ctfile_hub_<id>, " "cart.c:1415"), ("complement", "complement the bases by the ruler (hui.h:117)"), ("hgt.baseTitle", "title over the base position track"), ("hgt.revCmplDisp", "reverse-complement the whole view"), ("hgGenome_threshold", "Genome Graphs significance threshold"), ("mvCtfile", "myVariants custom track file"), ("hgPcrResult", "in-cart PCR result"), ("hgSearch_categs", "search result categories"), ] # Track-scoped names the #37838 catalog does not carry. Each probe is a # predicate over the corpus so the counts stay live. MISSING_TRACK_VARS = [ ("<track>.showCfg", lambda n: n.endswith(".showCfg"), "Per-view config disclosure, written at hg/lib/hui.c:8682 and read at " "hui.c:8718. Also appears three deep as <composite>.<view>.showCfg."), ("<track>.childShowCfg", lambda n: n.endswith(".childShowCfg"), "Exists only in JavaScript, hg/js/hui.js:645. No C code anywhere mentions " "it, which is exactly why a harvester that reads C never saw it."), ("<track>.label.<field>", lambda n: ".label." in n, "hui.c:4743 composes label.%s from whatever trackDb labelFields names, so " "the field set is open-ended by construction and cannot be enumerated. It " "has to be a family."), ("<track>.<field>FilterMin / Max", lambda n: re.search(r"Filter(Min|Max)$", n) is not None, "The camel twins of filter.<field>Min/Max. The catalog has " "<field>FilterLimits and friends but not these."), ("<track>.filterBy.<a.b>", lambda n: re.search(r"\.(filter|filterBy|highlightBy)\.[^.]+\.", n) is not None, "The field part can itself contain dots: filterBy.attrs.transcriptType, " "filterBy.vep.Consequence, filter.src.SP."), ("<mafTrack>.<accession species>", lambda n: re.search(r"\.GC[AF]_\d+\.\d+$", n) is not None, "A maf species column that is a GenArk accession, so the species token " "itself contains a dot. Any [^.]+ for the species is wrong."), ] HGTA_SHAPES = [ ("hgta_fil.v.<db>.<table>.<field>.<op>", lambda n: n.startswith("hgta_fil.v.") and n.count(".") == 5, - "catalogued with .pat only; .cmp and .dd are also in use"), + "catalogued: pat, dd and cmp per field, and the table-wide rawLogic, " + "rawQuery and maxOutput, whose field slot is empty or _"), ("hgta_fs.check.<db>.<table>.<field>", lambda n: n.startswith("hgta_fs.check.") and n.count(".") == 4, "catalogued"), ("hgta_fs.linked.<db>.<table>", lambda n: n.startswith("hgta_fs.linked."), - "not catalogued"), + "catalogued"), ("hgta_fil.linked.<db>.<table>", lambda n: n.startswith("hgta_fil.linked."), - "not catalogued"), + "catalogued"), ] # ---------------------------------------------------------------- text reports def reportCheck(audit, out=sys.stdout): print("sessions read %d" % audit.nSess, file=out) print("distinct names %d" % len(audit.names), file=out) print("name instances %d" % sum(v[1] for v in audit.names.values()), file=out) print("track vocabulary %d (bare names whose values are all visibilities)" % len(audit.trackNames), file=out) print(file=out) for bucket, n in audit.counts().items(): print("%-16s %8d" % (bucket, n), file=out) print(file=out) print("catalog patterns literals %d families %d bare wildcards %d" % (len(audit.literals), len(audit.families), len(audit.wildcards)), file=out) - print("bare wildcards %s" % ", ".join(v for v, _ in audit.wildcards), file=out) + print("bare wildcards %s" + % ", ".join(sorted(v for v, _ in audit.wildcards)), file=out) + print("whole-name rows literals %d families %d" + % (len(audit.absLiterals), len(audit.absFamilies)), file=out) + if audit.absSkipped: + print(file=out) + print("catalog rows that are prose rather than one pattern, so nothing " + "can match them (%d):" % len(audit.absSkipped), file=out) + for row in sorted(audit.absSkipped): + print(" %s" % row, file=out) def reportRows(audit, bucket, out=sys.stdout, limit=None): rows = audit.sorted_(bucket) if limit: rows = rows[:limit] for nSess, total, name, values in rows: print("%6d\t%7d\t%s\t%s" % (nSess, total, name, " | ".join(values)), file=out) print("# %d names in bucket %s" % (len(audit.buckets[bucket]), bucket), file=out) def reportWildcard(audit, out=sys.stdout, limit=60): for suffix, n in audit.wildcardSuffix.most_common(limit): print("%6d\t%s" % (n, suffix), file=out) print("# %d distinct suffixes matched only by a catch-all" % len(audit.wildcardSuffix), file=out) def reportLeaks(audit, out=sys.stdout): seen, unseen = audit.leakReport() print("#37923 predicted %d leaks from reading the source." % len(audit.catLeaks), file=out) print("%d of them are in saved sessions right now:" % len(seen), file=out) for nSess, name in seen: print(" %6d %s" % (nSess, name), file=out) print(file=out) print("not seen in any session (%d). Absence is not safety, only that " "nobody saved a session after using them:" % len(unseen), file=out) print(" %s" % ", ".join(unseen), file=out) def reportFindings(audit, text, out=sys.stdout): def line(label, stats): print(" %-38s names %-7d sessions %-8d busiest %s (%d)" % (label, stats["names"], stats["sessions"], stats["busiest"], stats["busiestSess"]), file=out) print("== four ways to scope a variable to an assembly ==", file=out) for shape, src, example in DB_SCOPES: print(" %-24s %-44s %s in %d sessions" % (shape, src, example, audit.sess(example)), file=out) print(file=out) print("== the <var>_<db> family in use ==", file=out) for var, _note in DB_SUFFIX_VARS: line(var + "_<db>", dbSuffixFamily(audit, var)) print(file=out) print("== global variables neither catalog covers ==", file=out) for title, src, vars_ in GLOBAL_GROUPS: present = [(audit.sess(v), v) for v in vars_ if v in audit.names] present.sort(reverse=True) print(" %s (%s)" % (title, src), file=out) print(" %s" % ", ".join("%s(%d)" % (v, c) for c, v in present), file=out) print(file=out) print("== track variables the #37838 catalog lacks ==", file=out) for label, pred, _note in MISSING_TRACK_VARS: line(label, nameStats(audit, pred)) fields, catalogued = labelFields(audit) print(" label.<field>: %d distinct fields observed, %d of them catalogued" % (len(fields), len(set(fields) & catalogued)), file=out) print(file=out) print("== hgTables shapes ==", file=out) for label, pred, note in HGTA_SHAPES: stats = nameStats(audit, pred) print(" %-42s %6d names (%s)" % (label, stats["names"], note), file=out) print(file=out) print("== the nested cart inside position.<db> ==", file=out) inner, forms, dbs = innerPositionCarts(audit, text) print(" %d assemblies, value forms: %s" % (len(dbs), dict(forms)), file=out) for var, n in inner.most_common(12): print(" %-26s %d" % (var, n), file=out) print(file=out) print("== nothing rejects a name ==", file=out) for title, _note, pred in NOISE_CLASSES: stats = nameStats(audit, pred) print(" %-32s %5d names, busiest %s (%d sessions)" % (title, stats["names"], stats["busiest"], stats["busiestSess"]), file=out) present = [(audit.sess(t), t) for t in TYPOS if t in audit.names] print(" %-32s %5d of the %d known typos are in saved sessions: %s" % ("Typos, frozen forever", len(present), len(TYPOS), ", ".join(t for _c, t in sorted(present, reverse=True))), file=out) # ---------------------------------------------------------------- json def asJson(audit, text): inner, forms, dbs = innerPositionCarts(audit, text) fields, catalogued = labelFields(audit) seen, unseen = audit.leakReport() return { "ticket": "#37838, #37923", "what": "audit of the two cart catalogs against every named session", "corpus": {"sessions": audit.nSess, "distinctNames": len(audit.names), "nameInstances": sum(v[1] for v in audit.names.values())}, "coverage": audit.counts(), - "bareWildcards": [v for v, _ in audit.wildcards], + "bareWildcards": sorted(v for v, _ in audit.wildcards), + "wholeNameRows": {"literals": len(audit.absLiterals), + "families": sorted(v for v, _ in audit.absFamilies), + "unusable": sorted(audit.absSkipped)}, "dbScopes": [{"shape": s, "src": c, "example": e, "sessions": audit.sess(e)} for s, c, e in DB_SCOPES], "dbSuffixFamily": {v: dbSuffixFamily(audit, v) for v, _n in DB_SUFFIX_VARS}, "globalGroups": [{"title": t, "src": s, "vars": {v: audit.sess(v) for v in vs if v in audit.names}} for t, s, vs in GLOBAL_GROUPS], "missingTrackVars": [{"shape": l, "note": n, "stats": nameStats(audit, p)} for l, p, n in MISSING_TRACK_VARS], "labelFields": {"observed": len(fields), "catalogued": len(catalogued), "uncatalogued": sorted(set(fields) - catalogued)}, "hgtaShapes": [{"shape": l, "status": n, "names": nameStats(audit, p)["names"]} for l, p, n in HGTA_SHAPES], "positionInnerCart": {"assemblies": len(dbs), "valueForms": dict(forms), "vars": dict(inner)}, "noise": [{"title": t, "note": n, "stats": nameStats(audit, p)} for t, n, p in NOISE_CLASSES], "leaks": {"predicted": len(audit.catLeaks), "observed": {n: c for c, n in seen}, "notObserved": unseen}, "unknown": {n: audit.names[n][0] for n in audit.buckets["unknown"]}, } # ---------------------------------------------------------------- html CSS = """ body { font: 14px/1.5 -apple-system, "Segoe UI", Helvetica, Arial, sans-serif; margin: 0; color: #12191f; background: #fff; } header { background: #14385c; color: #fff; padding: 18px 28px; } header h1 { margin: 0 0 4px; font-size: 20px; font-weight: 600; } header p { margin: 0; font-size: 13px; color: #c5d6e8; max-width: 90ch; } main { max-width: 1080px; margin: 0 auto; padding: 22px 28px 60px; } h2 { font-size: 17px; margin: 34px 0 6px; padding-bottom: 5px; border-bottom: 2px solid #14385c; } h3 { font-size: 15px; margin: 22px 0 4px; color: #14385c; } p, li { max-width: 88ch; } p.what { margin: 4px 0 10px; color: #4a5764; font-size: 13px; max-width: 88ch; } table { border-collapse: collapse; width: 100%; margin: 6px 0 14px; font-size: 13px; } th { text-align: left; background: #eef2f6; padding: 5px 8px; border-bottom: 1px solid #c9d3dd; font-weight: 600; } td { padding: 5px 8px; border-bottom: 1px solid #eceff2; vertical-align: top; } tr:hover td { background: #f7fafd; } code, .n { font-family: ui-monospace, Menlo, Consolas, monospace; font-size: 12.5px; } .n { font-weight: 600; color: #0b3d62; white-space: nowrap; } .v { color: #7a3ba8; font-family: ui-monospace, Menlo, Consolas, monospace; font-size: 12.5px; word-break: break-all; } .src { color: #6b7885; font-size: 11.5px; } .note { color: #4a5764; font-size: 12.5px; } .num { text-align: right; white-space: nowrap; font-variant-numeric: tabular-nums; } .legend { background: #f7fafd; border: 1px solid #dde5ec; padding: 12px 14px; font-size: 13px; margin: 12px 0 4px; } .finding { background: #fff8e1; border-left: 4px solid #e0a800; padding: 10px 14px; margin: 12px 0; font-size: 13px; } .bug { background: #fdf1f1; border-left: 4px solid #c05050; padding: 10px 14px; margin: 12px 0; font-size: 13px; } .ok { background: #eef7f1; border-left: 4px solid #3f8f5f; padding: 10px 14px; margin: 12px 0; font-size: 13px; } """ def esc(s): return html.escape(str(s)) def n(x): return "{:,}".format(x) def renderHtml(audit, text): o = [] add = o.append fields, catalogued = labelFields(audit) inner, forms, dbs = innerPositionCarts(audit, text) seen, unseen = audit.leakReport() counts = audit.counts() add("<title>What the saved sessions say the cart grammar is missing " "(#37838, #37923)</title>") add("<style>%s</style>" % CSS) add("<header><h1>What the saved sessions say the cart and URL grammar is " "missing</h1><p>Refs #37838 and #37923. Generated by " "hg/utils/sessionCartAudit/sessionCartAudit.py, not hand-edited. Every " "named session read back and matched against both catalogs.</p></header><main>") # method add("<h2>What was measured</h2>") add("<p class='what'>All %s named sessions in <code>namedSessionDb</code>, " "giving %s variable-name instances and <b>%s distinct names</b>. Each " "name was matched right to left against the longest suffix either " "catalog knows, at a <code>.</code> or <code>_</code> boundary. The bare " "wildcard entries in the #37838 catalog (%s) match any token at all, so " "they are scored separately; letting them match would absorb every " "unrecognised name and prove nothing.</p>" % (n(audit.nSess), n(sum(v[1] for v in audit.names.values())), n(len(audit.names)), - ", ".join("<code>%s</code>" % esc(v) for v, _ in audit.wildcards))) + ", ".join("<code>%s</code>" % esc(v) + for v in sorted(v for v, _ in audit.wildcards)))) add("<table><tr><th>bucket</th><th class='num'>distinct names</th>" "<th>meaning</th></tr>") for key, meaning in [ ("trackVis", "a bare track name whose value is hide/dense/squish/pack/full"), ("trackKnown", "matched a real entry in the #37838 catalog"), ("wildcardOnly", "matched only a catch-all pattern, so effectively uncatalogued"), ("urlKnown", "matched an entry in the #37923 catalog"), ("unknown", "matched nothing in either catalog")]: add("<tr><td>%s</td><td class='num'>%s</td><td class='note'>%s</td></tr>" % (key, n(counts[key]), meaning)) add("</table>") # 1 db scopes add("<h2>1. Four ways to scope a variable to an assembly, and the spec names none</h2>") add("<p class='what'>Both catalogs are organised around the track. Neither " "has a level for "this variable belongs to a database". The " "tree has four separate conventions for it, all in live use.</p>") add("<table><tr><th>shape</th><th>defined at</th><th class='num'>sessions</th></tr>") for shape, src, example in DB_SCOPES: add("<tr><td class='n'>%s</td><td class='src'>%s</td><td class='num'>%s " "<span class='src'>(%s)</span></td></tr>" % (esc(shape), esc(src), n(audit.sess(example)), esc(example))) add("</table>") add("<div class='finding'><b>Why this matters for the schema.</b> " "<code><var>_<db></code> and the legacy track form " "<code><track>_<var></code> are the same string shape with " "the parts in opposite order. <code>revCmplDisp_hg38</code> cannot be " "told from a track called <code>revCmplDisp</code> with a variable " "called <code>hg38</code> by any amount of parsing. Only a list of which " "names are variables separates them, which is what these two tickets " "are for.</div>") add("<h3>The <code><var>_<db></code> family in use</h3>") add("<table><tr><th>variable</th><th class='num'>sessions</th>" "<th>what it holds</th></tr>") for var, note in DB_SUFFIX_VARS: st = dbSuffixFamily(audit, var) add("<tr><td class='n'>%s_<db></td><td class='num'>%s " "<span class='src'>(%d assemblies)</span></td><td class='note'>%s</td></tr>" % (esc(var), n(st["sessions"]), st["names"], esc(note))) add("</table>") # 2 nested cart add("<h2>2. One cart variable holds an entire second cart</h2>") add("<p class='what'><code>cartSetDbPosition()</code> at hg/lib/cart.c:4078 " "calls <code>cartEncodeState()</code> and stores the whole result as the " "value of <code>position.<db></code>, so a var=val blob is CGI-encoded " "and stuffed inside one value of the outer var=val blob. %s of the %s " "sessions have one, across %d assemblies. Raw, it looks like this:</p>" % (n(audit.sess("position.hg38")), n(audit.nSess), len(dbs))) add("<p class='v' style='background:#f7fafd;padding:8px 10px;" "border:1px solid #dde5ec'>position.hg38=lastVirtModeType%3Ddefault" "%26lastVirtModeExtraState%3D%26virtModeType%3Ddefault%26virtMode%3D0" "%26nonVirtPosition%3D%26position%3Dchr4%253A110617423%252D110623077</p>") if inner: add("<p class='what'>The inner cart has its own variable set, none of it " "catalogued:</p>") add("<table><tr><th>inner variable</th><th class='num'>occurrences</th></tr>") for var, c in inner.most_common(): if c < 5: continue add("<tr><td class='n'>%s</td><td class='num'>%s</td></tr>" % (esc(var), n(c))) add("</table>") add("<p class='what'>Value forms seen: %s. Some are in an older bare form " "(just the position, no inner cart), and a handful are in a third form " "again, a plus-separated tuple such as " "<code>virt:69139406-69143353+chr8:11438769-11619126+exonMostly+emGeneTable</code>. " "Three formats in one variable.</p>" % esc(", ".join("%s %s" % (k, n(v)) for k, v in forms.items()))) add("<div class='bug'><b>Bug found on the way through.</b> hg/lib/cart.c:4063 " "writes <code>lastVirtModeExtra</code> into the fresh inner cart, but " "every reader in the tree wants <code>lastVirtModeExtraState</code> " "(hgTracks.c:10957, 11276, 11302; web.c:1130). The short name appears " "exactly once in the whole source, at that write.</div>") # 3 globals add("<h2>3. The global browser settings have no catalog at all</h2>") add("<p class='what'>#37838 covers variables scoped to a track. #37923 covers " "what may go on a URL. A large and heavily used group falls between them: " "cart variables that are neither. They are what the Configure page writes, " "and they are in nearly every session.</p>") add("<table><tr><th>group</th><th>variables</th><th class='num'>busiest</th></tr>") for title, src, vars_ in GLOBAL_GROUPS: present = sorted(((audit.sess(v), v) for v in vars_ if v in audit.names), reverse=True) if not present: continue add("<tr><td><b>%s</b><br><span class='src'>%s</span></td>" "<td class='note'>%s</td><td class='num'><code>%s</code><br>" "<span class='src'>in %s sessions</span></td></tr>" % (esc(title), esc(src), ", ".join("<code>%s</code>" % esc(v) for _c, v in present), esc(present[0][1]), n(present[0][0]))) add("</table>") add("<div class='finding'><b>Suggested shape.</b> These divide three ways: " "settings the user chose and expects to keep, view state the browser " "recomputes every request (<code>winStart</code>, <code>insideX</code>, " "<code>rulerClickHeight</code>), and one-request form scratch that should " "never have been saved (the track-search group is a form's field set). " "Only the first belongs in a shared session, and nothing in the current " "format can tell them apart.</div>") # 4 missing track vars add("<h2>4. Track-scoped variables the #37838 catalog does not have</h2>") add("<table><tr><th>variable</th><th class='num'>distinct names</th>" "<th class='num'>busiest</th><th>where it comes from</th></tr>") for label, pred, note in MISSING_TRACK_VARS: st = nameStats(audit, pred) add("<tr><td class='n'>%s</td><td class='num'>%s</td>" "<td class='num'><code>%s</code><br><span class='src'>%s sessions</span></td>" "<td class='note'>%s</td></tr>" % (esc(label), n(st["names"]), esc(st["busiest"] or ""), n(st["busiestSess"]), esc(note))) add("</table>") extra = sorted(set(fields) - catalogued, key=lambda f: -fields[f]) add("<div class='finding'><b><code>label.<field></code> cannot be " "enumerated.</b> The catalog lists %d fixed names. The sessions contain " "<b>%d distinct fields</b>, %d of them uncatalogued, from " "<code>%s</code> in %s sessions down to a long tail of one. It has to be " "a family, not a list.</div>" % (len(catalogued), len(fields), len(extra), esc(extra[0]) if extra else "", n(fields[extra[0]]) if extra else 0)) add("<div class='finding'><b>The JavaScript side was never harvested.</b> " "The #37838 notes recorded JS-only variables as something to come back " "to. <code>childShowCfg</code> is written under %s different track names " "and appears wherever the UI has views, yet it exists nowhere in the C. " "<code>hlColor</code> (hui.js:1566, hgTracks.js:2260) is the same class. " "A harvester that reads only C will keep missing all of it.</div>" % n(nameStats(audit, lambda x: x.endswith(".childShowCfg"))["names"])) # 5 separators add("<h2>5. Name components contain the separators, so the parse must be " "right-anchored</h2>") add("<p class='what'>The catalog's own notes warn that " "<code><track>.<var></code> is not a safe two-part parse " "because of composites. The sessions give a second, independent reason: " "the components themselves contain dots, spaces and slashes.</p>") add("<table><tr><th>example</th><th>problem</th></tr>") for example, problem in [ ("vgp577way.GCF_020740605.2", "A maf species column that is a GenArk accession, so the species token " "contains a dot. %s distinct names of this shape." % n(nameStats(audit, lambda x: re.search(r"\.GC[AF]_\d+\.\d+$", x) is not None)["names"])), ("strainName44way.CoV_BtRs-BetaCoV/YN2018D", "Species token with a slash."), ("hub_220_ts_regions .yLineOnOff", "Track name with a trailing space. %d names contain a space." % nameStats(audit, lambda x: " " in x)["names"]), ("hub_220_TS_3Seq_rep4_uniq_( )_imgOrd", "Parentheses and a space, then the legacy underscore separator."), ("(the empty string)", "One name is empty, in %d sessions. %d more contain non-ASCII bytes." % (audit.sess(""), nameStats(audit, lambda x: any(ord(ch) > 126 for ch in x))["names"])), ]: add("<tr><td class='n'>%s</td><td class='note'>%s</td></tr>" % (esc(example), problem)) add("</table>") add("<div class='finding'><b>Rule this implies.</b> Match right-anchored, " "against a known vocabulary of variable names, never left-anchored at " "the first separator. That is only possible once the vocabulary is " "complete, which is the argument for finishing both catalogs before " "publishing a schema rather than after.</div>") # 6 hgTables add("<h2>6. The hgTables namespace is deeper than catalogued</h2>") add("<table><tr><th>shape</th><th class='num'>names</th><th>status</th></tr>") for label, pred, status in HGTA_SHAPES: add("<tr><td class='n'>%s</td><td class='num'>%s</td><td class='note'>%s</td></tr>" % (esc(label), n(nameStats(audit, pred)["names"]), esc(status))) add("</table>") # 7 noise add("<h2>7. Nothing rejects a variable name, and the sessions prove it</h2>") add("<p class='what'>Any name in a query string becomes a cart variable and " "stays. The saved sessions are a record of everything ever typed at the " "browser. This is the strongest argument in the data for validating " "names against a registry on the way in.</p>") add("<table><tr><th>class</th><th class='num'>distinct</th><th>examples</th></tr>") for title, note, pred in NOISE_CLASSES: hits = sorted(((audit.sess(x), x) for x in audit.names if pred(x)), reverse=True) if not hits: continue add("<tr><td><b>%s</b><br><span class='src'>%s</span></td>" "<td class='num'>%d</td><td class='note'>%s</td></tr>" % (esc(title), esc(note), len(hits), ", ".join("<code>%s</code> (%s)" % (esc(x), n(c)) for c, x in hits[:8]))) present = sorted(((audit.sess(t), t) for t in TYPOS if t in audit.names), reverse=True) add("<tr><td><b>Typos, frozen forever</b><br><span class='src'>a slip in a " "URL, kept for years</span></td><td class='num'>%d</td>" "<td class='note'>%s</td></tr>" % (len(present), ", ".join("<code>%s</code>" % esc(t) for _c, t in present))) add("</table>") add("<div class='finding'><b>Direct bearing on #37838 phase 4.</b> The " "JSON-in path is a new front door. If it validates names against the " "#37923 registry and refuses the rest, it is the first entry point the " "browser has ever had that cannot accumulate this.</div>") # 8 leaks add("<h2>8. The leak audit checks out against real data</h2>") add("<div class='ok'><b>%d of the %d leaks predicted by " "<code>--reconcile</code> are sitting in saved sessions right now.</b> " "The prediction came from reading the source; this is independent " "confirmation from the data.</div>" % (len(seen), len(audit.catLeaks))) add("<table><tr><th>leaked parameter</th><th class='num'>sessions</th></tr>") for c, name in seen: add("<tr><td class='n'>%s</td><td class='num'>%s</td></tr>" % (esc(name), n(c))) add("</table>") add("<p class='what'>The %d not seen in any session: %s. Absence is not " "evidence they are safe, only that nobody saved a session after using " "them.</p>" % (len(unseen), ", ".join("<code>%s</code>" % esc(x) for x in unseen))) add("<p class='src' style='margin-top:30px'>Source: %s named sessions. " "Counts are distinct variable names unless labelled otherwise; " ""sessions" means the number of saved sessions containing that " "name.</p>" % n(audit.nSess)) add("</main>") return "\n".join(o) # ---------------------------------------------------------------- main def main(): ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("--central", default=None, help="hgcentral database to read (default from ~/.hg.conf)") ap.add_argument("--dump", help="read a saved contents dump instead of the db") ap.add_argument("--save-dump", dest="saveDump", help="write the raw dump for later runs") ap.add_argument("--check", action="store_true", help="coverage counts") ap.add_argument("--unknown", action="store_true", help="names neither catalog knows, one per line") ap.add_argument("--wildcard", action="store_true", help="suffixes matched only by a bare wildcard") ap.add_argument("--leaks", action="store_true", help="#37923 leak predictions against the data") ap.add_argument("--findings", action="store_true", help="the curated findings, with counts recomputed") ap.add_argument("--json") ap.add_argument("--html") args = ap.parse_args() if args.dump: text = open(args.dump, encoding="utf8", errors="replace").read() else: text = dumpSessions(args.central or defaultCentral()) if args.saveDump: with open(args.saveDump, "w") as f: f.write(text) print("wrote %s" % args.saveDump, file=sys.stderr) names, nSess = extract(text) audit = Audit(names, nSess) did = False if args.check: reportCheck(audit); did = True if args.unknown: reportRows(audit, "unknown"); did = True if args.wildcard: reportWildcard(audit); did = True if args.leaks: reportLeaks(audit); did = True if args.findings: reportFindings(audit, text); did = True if args.json: with open(args.json, "w") as f: json.dump(asJson(audit, text), f, indent=1) print("wrote %s" % args.json); did = True if args.html: with open(args.html, "w") as f: f.write(renderHtml(audit, text)) print("wrote %s" % args.html); did = True if not did: reportCheck(audit) return 0 if __name__ == "__main__": sys.exit(main())