b066328906fe2c8fbbfada16f18e8cc2fb9dc1cf braney Sat Sep 12 09:58:46 2026 -0700 sessionCartAudit: match a catalog row that names a whole cart variable, refs #37979 peel() only ever considered suffixes that begin after a separator, so a catalog row anchored on a fixed prefix could never match the variable it describes: by the time the walk reached a separator, the hgta_ that anchors hgta_fs.check.<db>.<table>.<field> was gone. A plain literal row was hit just as hard, since dbRIP.genoRegion could only be tested as its own tail, "genoRegion". 4,299 hgta_ names were reported as covered by nothing but a catch-all because of it. The #37838 catalog already says which kind of row it is and the audit was throwing that away. A row whose separator is "." or "_" names a suffix that follows a track name; anything else names the whole cart variable, with the separator in front - "" for hgTables and the old per-dataset variables, cgs_<track>_ for chromGraph. trackVarNames() now returns the separator with the name, the whole-variable rows are compiled apart, and peel() tries the whole name before walking suffixes, so the longer and correct match wins. Two shorthands the catalog already uses are read rather than expanded by hand: a comma list is several variables sharing one description, and a trailing * is a family. A row that is prose rather than one pattern can be matched by nothing, so --check names it instead of letting it count for nothing; there is one today. The bare wildcard lists are sorted, because they were built by walking a set and two runs of a published report diffed for no reason. Over 6,631 saved sessions this moves 2,016 names: 1,990 out of the catch-all bucket, 25 out of the bare-track-name bucket (filter text boxes whose stored value is empty, which the visibility heuristic had been reading as track names), and tfbsConsSitesCutoff out of unknown. Nothing leaves the catalogued buckets. 1,208 hgta_fs.check names now match the row that describes them rather than <wigTrack>.<wigVar> or <filterName>Type. diff --git src/hg/utils/sessionCartAudit/sessionCartAudit.py src/hg/utils/sessionCartAudit/sessionCartAudit.py index 25af3cf282a..657d4dbe193 100755 --- src/hg/utils/sessionCartAudit/sessionCartAudit.py +++ src/hg/utils/sessionCartAudit/sessionCartAudit.py @@ -16,36 +16,47 @@ transcript of everything the browser has ever been asked to remember. cartTrackVarCatalog.py source -> what a track variable may be called urlCommandCatalog.py source -> what may go on a URL sessionCartAudit.py data -> what is actually in the carts, and which of it the other two do not cover The catch, and the reason the matching below is fussier than it looks: the track catalog contains bare wildcard entries (<species>, <filterName>, <wigTrack>.<wigVar>, <attribute><Value>, <view>.<anyTypeVar>, _<origin>) that match literally any token. Score them as matches and every unrecognised name in the corpus is absorbed, the audit comes back clean, and it has proved nothing. They are matched separately and reported as "wildcard only", which is the honest answer: covered by a catch-all, not actually catalogued. -Matching is right-anchored - longest known suffix at a '.' or '_' boundary - -never left-anchored at the first separator. That is not a style choice. Track -and species names in the real data contain dots (GenArk accessions such as -GCF_020740605.2), slashes, spaces and parentheses, so the left-hand side cannot -be delimited by anything except a complete list of what the right-hand side may -be. +Matching a track-scoped name is right-anchored - longest known suffix at a '.' +or '_' boundary - never left-anchored at the first separator. That is not a +style choice. Track and species names in the real data contain dots (GenArk +accessions such as GCF_020740605.2), slashes, spaces and parentheses, so the +left-hand side cannot be delimited by anything except a complete list of what +the right-hand side may be. + +Not every catalog row is a suffix, though, and that is the other half of the +matching. A row's separator says which it is: "." or "_" means the name +follows a track name, and anything else means the row already spells out the +whole cart variable, as hgTables' rows do (hgta_fs.check.<db>.<table>.<field>) +and as chromGraph's do with a prefix in front (cgs_<track>_pixels). A whole +name has to be matched whole. Testing only suffixes leaves a left-anchored row +unable to match the variable it describes, because the prefix that anchors it +is gone before the first candidate is cut - which is why the whole name is +tried first, and why for a long time 4,299 hgta_ names were reported as covered +by nothing but a catch-all. Usage: sessionCartAudit.py --check # coverage counts sessionCartAudit.py --unknown # names neither catalog knows sessionCartAudit.py --wildcard # names only a catch-all matches sessionCartAudit.py --leaks # #37923 leak claims vs the data sessionCartAudit.py --findings # the curated findings, with live counts sessionCartAudit.py --json out.json sessionCartAudit.py --html out.html --dump FILE read a previously saved contents dump instead of the db --save-dump F write the raw dump so later runs can skip the query --central DB hgcentral database to read (default from hg.conf) Reading the sessions takes a minute and a couple of hundred MB of query output, @@ -129,48 +140,56 @@ # ---------------------------------------------------------------- the catalogs def loadSibling(subdir, module): """Import a sibling catalog by path so this works from any cwd.""" path = os.path.join(UTILS, subdir, module + ".py") if not os.path.exists(path): sys.exit("cannot find sibling catalog %s\n" "expected it next door at %s" % (module, path)) spec = importlib.util.spec_from_file_location(module, path) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) return mod def trackVarNames(): - """Every variable name in the #37838 catalog, however deeply nested.""" + """Every variable in the #37838 catalog, however deeply nested. + + Returns the bare names and the (name, sep) rows. The separator matters: + it is what says whether a row names a suffix that follows a track name + ("." or "_") or a whole cart variable ("" for hgTables and the old + per-dataset variables, "cgs_<track>_" for chromGraph, which puts its + prefix in front). peel() needs both, and a set of bare names cannot tell + them apart.""" mod = loadSibling("cartTrackVarCatalog", "cartTrackVarCatalog") cat = mod.build() - names = set() + names, rows = set(), [] def walk(node): if isinstance(node, dict): name = node.get("name") if isinstance(name, str) and ("type" in node or "src" in node): names.add(name) + rows.append((name, node.get("sep", "."))) for value in node.values(): walk(value) elif isinstance(node, list): for value in node: walk(value) walk(cat) - return names + return names, rows def urlCommandNames(): """Every parameter name in the #37923 catalog, plus the ones it calls leaks.""" mod = loadSibling("urlCommandCatalog", "urlCommandCatalog") cat = mod.build() names, leaks = set(), set() def walk(node): if isinstance(node, dict): name = node.get("name") if isinstance(name, str) and ("kind" in node or "value" in node): names.add(name) if node.get("leaks"): leaks.add(name) @@ -194,71 +213,134 @@ literals, families, wildcards = set(), [], [] for var in catVars: if not PLACEHOLDER.search(var): literals.add(var) continue parts = PLACEHOLDER.split(var) regex = re.compile("^" + "[^.]+".join(re.escape(p) for p in parts) + "$") residue = PLACEHOLDER.sub("", var).strip("._") if len(residue) < 3: wildcards.append((var, regex)) else: families.append((var, regex)) return literals, families, wildcards +# A catalog row may stand for more than one variable. Both shorthands are +# already in use in the #37838 catalog and both are machine-readable, so the +# audit reads them rather than making a person expand them: a comma list is +# several variables that share one description, and a trailing * is a family +# whose members are listed in the note. +GLOB = re.compile(r"<[a-zA-Z]+>|\*") + + +def absRegex(pattern): + """Compile a whole-variable pattern. <placeholder> is one token, * is any.""" + out, pos = [], 0 + for match in GLOB.finditer(pattern): + out.append(re.escape(pattern[pos:match.start()])) + out.append("[^.]*" if match.group(0) == "*" else "[^.]+") + pos = match.end() + out.append(re.escape(pattern[pos:])) + return re.compile("^" + "".join(out) + "$") + + +def absolutePatterns(catRows): + """The rows that spell out a whole cart variable rather than a suffix. + + A row whose sep is "." or "_" describes what follows a track name, which is + what peel()'s suffix walk is for. Every other sep means the row already + names the variable as it appears in the cart, and the sep goes in front of + it: sep "" leaves the name alone, and chromGraph's "cgs_<track>_" prefixes + it. These can only ever be matched against the whole name. + + Returns (literals, families, skipped). A row that is prose rather than a + pattern cannot be matched by anything and is returned in skipped, so it can + be reported instead of quietly counting for nothing.""" + literals, families, skipped = set(), [], [] + for name, sep in catRows: + if sep in (".", "_"): + continue + for part in name.split(","): + full = (sep + part.strip()).strip() + if not full: + continue + if " " in full: + skipped.append(full) + elif GLOB.search(full): + families.append((full, absRegex(full))) + else: + literals.add(full) + return literals, families, skipped + + # ---------------------------------------------------------------- classify class Audit(object): def __init__(self, names, nSess): self.names = names self.nSess = nSess - self.catVars = trackVarNames() + self.catVars, catRows = trackVarNames() self.urlNames, self.catLeaks = urlCommandNames() self.literals, self.families, self.wildcards = compilePatterns(self.catVars) + self.absLiterals, self.absFamilies, self.absSkipped = absolutePatterns(catRows) # Track-name vocabulary learned from the corpus itself: a bare name # whose every observed value is a visibility word is a track. self.trackNames = set(n for n, v in names.items() if v[2] and all(x in VIS_VALUES for x in v[2])) self.buckets = collections.defaultdict(list) self.wildcardSuffix = collections.Counter() self._dbs = None self.classify() def sess(self, name): return self.names[name][0] if name in self.names else 0 def dbVocabulary(self): """Assembly names, learned from the position.<db> variables in the corpus. Every session that has ever been at an assembly leaves one, so this is a better list than anything hard-coded, and it comes from the same data being audited.""" if self._dbs is None: self._dbs = set(n.split(".", 1)[1] for n in self.names if n.startswith("position.") and n.count(".") == 1) return self._dbs def famMatch(self, suffix, patterns): for src, regex in patterns: if regex.match(suffix): return src return None def peel(self, name): - """Longest catalogued suffix at a separator. -> (stem, suffix, how)""" + """Longest catalogued pattern covering the name. -> (stem, suffix, how) + + The whole name is tried first, then the longest suffix at a separator. + The whole-name step is not an optimisation: every candidate the suffix + walk considers begins after a separator, so a row that is left-anchored + on a fixed prefix can never match the variable it describes. + hgta_fs.check.<db>.<table>.<field> is the clearest case - by the time + the walk reaches a separator the hgta_ that anchors the row is gone - + and so is a plain literal like dbRIP.genoRegion, which without this + could only ever be tested as its own tail, "genoRegion".""" + if name in self.absLiterals: + return "", name, "literal" + hit = self.famMatch(name, self.absFamilies) + if hit: + return "", name, hit fallback = None for i in range(len(name) - 1): if name[i] not in "._": continue suffix = name[i + 1:] if suffix in self.literals or (name[i] + suffix) in self.literals: return name[:i], suffix, "literal" hit = self.famMatch(suffix, self.families) if hit: return name[:i], suffix, hit if fallback is None: hit = self.famMatch(suffix, self.wildcards) if hit: fallback = (name[:i], suffix, "wildcard:" + hit) return fallback @@ -467,58 +549,68 @@ "The camel twins of filter.<field>Min/Max. The catalog has " "<field>FilterLimits and friends but not these."), ("<track>.filterBy.<a.b>", lambda n: re.search(r"\.(filter|filterBy|highlightBy)\.[^.]+\.", n) is not None, "The field part can itself contain dots: filterBy.attrs.transcriptType, " "filterBy.vep.Consequence, filter.src.SP."), ("<mafTrack>.<accession species>", lambda n: re.search(r"\.GC[AF]_\d+\.\d+$", n) is not None, "A maf species column that is a GenArk accession, so the species token " "itself contains a dot. Any [^.]+ for the species is wrong."), ] HGTA_SHAPES = [ ("hgta_fil.v.<db>.<table>.<field>.<op>", lambda n: n.startswith("hgta_fil.v.") and n.count(".") == 5, - "catalogued with .pat only; .cmp and .dd are also in use"), + "catalogued: pat, dd and cmp per field, and the table-wide rawLogic, " + "rawQuery and maxOutput, whose field slot is empty or _"), ("hgta_fs.check.<db>.<table>.<field>", lambda n: n.startswith("hgta_fs.check.") and n.count(".") == 4, "catalogued"), ("hgta_fs.linked.<db>.<table>", lambda n: n.startswith("hgta_fs.linked."), - "not catalogued"), + "catalogued"), ("hgta_fil.linked.<db>.<table>", lambda n: n.startswith("hgta_fil.linked."), - "not catalogued"), + "catalogued"), ] # ---------------------------------------------------------------- text reports def reportCheck(audit, out=sys.stdout): print("sessions read %d" % audit.nSess, file=out) print("distinct names %d" % len(audit.names), file=out) print("name instances %d" % sum(v[1] for v in audit.names.values()), file=out) print("track vocabulary %d (bare names whose values are all visibilities)" % len(audit.trackNames), file=out) print(file=out) for bucket, n in audit.counts().items(): print("%-16s %8d" % (bucket, n), file=out) print(file=out) print("catalog patterns literals %d families %d bare wildcards %d" % (len(audit.literals), len(audit.families), len(audit.wildcards)), file=out) - print("bare wildcards %s" % ", ".join(v for v, _ in audit.wildcards), file=out) + print("bare wildcards %s" + % ", ".join(sorted(v for v, _ in audit.wildcards)), file=out) + print("whole-name rows literals %d families %d" + % (len(audit.absLiterals), len(audit.absFamilies)), file=out) + if audit.absSkipped: + print(file=out) + print("catalog rows that are prose rather than one pattern, so nothing " + "can match them (%d):" % len(audit.absSkipped), file=out) + for row in sorted(audit.absSkipped): + print(" %s" % row, file=out) def reportRows(audit, bucket, out=sys.stdout, limit=None): rows = audit.sorted_(bucket) if limit: rows = rows[:limit] for nSess, total, name, values in rows: print("%6d\t%7d\t%s\t%s" % (nSess, total, name, " | ".join(values)), file=out) print("# %d names in bucket %s" % (len(audit.buckets[bucket]), bucket), file=out) def reportWildcard(audit, out=sys.stdout, limit=60): for suffix, n in audit.wildcardSuffix.most_common(limit): print("%6d\t%s" % (n, suffix), file=out) print("# %d distinct suffixes matched only by a catch-all" @@ -592,31 +684,34 @@ # ---------------------------------------------------------------- json def asJson(audit, text): inner, forms, dbs = innerPositionCarts(audit, text) fields, catalogued = labelFields(audit) seen, unseen = audit.leakReport() return { "ticket": "#37838, #37923", "what": "audit of the two cart catalogs against every named session", "corpus": {"sessions": audit.nSess, "distinctNames": len(audit.names), "nameInstances": sum(v[1] for v in audit.names.values())}, "coverage": audit.counts(), - "bareWildcards": [v for v, _ in audit.wildcards], + "bareWildcards": sorted(v for v, _ in audit.wildcards), + "wholeNameRows": {"literals": len(audit.absLiterals), + "families": sorted(v for v, _ in audit.absFamilies), + "unusable": sorted(audit.absSkipped)}, "dbScopes": [{"shape": s, "src": c, "example": e, "sessions": audit.sess(e)} for s, c, e in DB_SCOPES], "dbSuffixFamily": {v: dbSuffixFamily(audit, v) for v, _n in DB_SUFFIX_VARS}, "globalGroups": [{"title": t, "src": s, "vars": {v: audit.sess(v) for v in vs if v in audit.names}} for t, s, vs in GLOBAL_GROUPS], "missingTrackVars": [{"shape": l, "note": n, "stats": nameStats(audit, p)} for l, p, n in MISSING_TRACK_VARS], "labelFields": {"observed": len(fields), "catalogued": len(catalogued), "uncatalogued": sorted(set(fields) - catalogued)}, "hgtaShapes": [{"shape": l, "status": n, "names": nameStats(audit, p)["names"]} for l, p, n in HGTA_SHAPES], "positionInnerCart": {"assemblies": len(dbs), "valueForms": dict(forms), "vars": dict(inner)}, "noise": [{"title": t, "note": n, "stats": nameStats(audit, p)} @@ -688,31 +783,32 @@ "missing</h1><p>Refs #37838 and #37923. Generated by " "hg/utils/sessionCartAudit/sessionCartAudit.py, not hand-edited. Every " "named session read back and matched against both catalogs.</p></header><main>") # method add("<h2>What was measured</h2>") add("<p class='what'>All %s named sessions in <code>namedSessionDb</code>, " "giving %s variable-name instances and <b>%s distinct names</b>. Each " "name was matched right to left against the longest suffix either " "catalog knows, at a <code>.</code> or <code>_</code> boundary. The bare " "wildcard entries in the #37838 catalog (%s) match any token at all, so " "they are scored separately; letting them match would absorb every " "unrecognised name and prove nothing.</p>" % (n(audit.nSess), n(sum(v[1] for v in audit.names.values())), n(len(audit.names)), - ", ".join("<code>%s</code>" % esc(v) for v, _ in audit.wildcards))) + ", ".join("<code>%s</code>" % esc(v) + for v in sorted(v for v, _ in audit.wildcards)))) add("<table><tr><th>bucket</th><th class='num'>distinct names</th>" "<th>meaning</th></tr>") for key, meaning in [ ("trackVis", "a bare track name whose value is hide/dense/squish/pack/full"), ("trackKnown", "matched a real entry in the #37838 catalog"), ("wildcardOnly", "matched only a catch-all pattern, so effectively uncatalogued"), ("urlKnown", "matched an entry in the #37923 catalog"), ("unknown", "matched nothing in either catalog")]: add("<tr><td>%s</td><td class='num'>%s</td><td class='note'>%s</td></tr>" % (key, n(counts[key]), meaning)) add("</table>") # 1 db scopes add("<h2>1. Four ways to scope a variable to an assembly, and the spec names none</h2>")