20bd278408f2907cf1965c7a8789dc887222b546 braney Sat Jul 25 13:43:30 2026 -0700 add a registry of hg.conf variables with a sunset mechanism for release gates refs #37925 Third of the browser configuration inventories, after hg/utils/cartTrackVarCatalog (#37838) and hg/utils/urlCommandCatalog (#37923), and built as a sibling of both: a curated registry plus a mechanical harvester it is reconciled against. harvestHgConf.py scans for every cfgOption* read. The accessors do not all take the setting name in the same argument, so it parses the argument list rather than matching the first string literal: cfgOptionEnv's first argument is an environment variable, not an hg.conf name, and cfgOption2 builds prefix.suffix from a prefix that is usually a runtime profile name. That last case is why ex.hg.conf documents archivecentral.password while the source contains no such literal. hgConfCatalog.py is the registry, with --json, --html, --check and --reconcile. What hg.conf needs that the other two did not is a way to retire a variable, so --sunset reports which release gates should be gone. The 58 boolean flags are split into gates, added to ship a feature dark and meant to be deleted once it is public, and knobs, which a mirror may set forever. --reconcile fails if any flag in the tree is left unclassified. Only that split is hand-written. Both lifecycle dates are read out of git: one history walk records when each variable was first read and when each flag's default first became TRUE, mapped onto the CGI_VERSION in effect at the time from the history of versionInfo.h. hgConfAges.json caches that walk, which otherwise costs two minutes. The audit found 18 shipped gates past a removal deadline, 9 more that have sat at a FALSE default without ever shipping (hgSession.shortLink since v374), and a documentation bug: ex.hg.conf tells mirrors to set userDbTableName but the code reads userDbName, so anyone who follows it is silently ignored. No makefile: this is a documentation generator, so it is left out of the utils DIRS list, same as hg/utils/otto and the two sibling catalogs. Nothing reads the registry yet and no CGI behavior changes. diff --git src/hg/utils/hgConfCatalog/hgConfCatalog.py src/hg/utils/hgConfCatalog/hgConfCatalog.py new file mode 100755 index 00000000000..5d4066ea9e7 --- /dev/null +++ src/hg/utils/hgConfCatalog/hgConfCatalog.py @@ -0,0 +1,1780 @@ +#!/usr/bin/env python3 +"""hgConfCatalog.py - the registry of hg.conf variables. + +Third of the browser configuration inventories, after cartTrackVarCatalog.py +(#37838, track-scoped cart variables) and urlCommandCatalog.py (#37923, CGI URL +parameters). This one covers hg.conf: the per-machine configuration file every +CGI reads at startup through the cfgOption* accessors in hg/lib/hgConfig.c. + +What makes hg.conf different from the other two, and the reason this file +needed a mechanism the others do not have: + + A cart variable belongs to a user and a URL parameter belongs to a request. + An hg.conf variable belongs to a machine, and a machine we do not control. + Mirrors, the GBiB and the GBiC all carry their own hg.conf, so a variable is + reachable long after the tree stops caring about it, and deleting one is a + compatibility decision rather than a cleanup. + + On top of that, the tree deliberately manufactures short-lived hg.conf + variables as part of the release process. A user-visible feature is expected + to ship dark behind cfgOptionBooleanDefault(name, FALSE) so it can sit on + master through QA without blocking the release train, then have its default + flipped to TRUE once it is released, then have the flag deleted. That last + step is the one nobody does. Nothing in the tree records that a flag was + meant to be temporary, so shipped gates accumulate: showTutorial has been + defaulting TRUE since v466, greyBarIcons since v492, and both are still + branch points in hgTracks today. + +So the registry separates two populations that look identical in the source: + + gate introduced to gate a release. Temporary by intent. Has a lifecycle + (added, flipped, sunset) and is expected to be deleted. + knob a genuine deployment switch a mirror is entitled to set forever. + isGbib, browser.dumpStack and hgta.disableAllTables are knobs. These + are exempt from sunsetting and saying so explicitly is what keeps the + report from crying wolf. + +The distinction cannot be made mechanically, which is why it is curated here +rather than harvested. Everything else about a gate's lifecycle is mechanical: +harvestHgConf.py --age dates each variable's first read, and each boolean +flag's first TRUE default, out of the git history, mapped onto the CGI_VERSION +in effect at the time. So `added` and `flipped` come from the tree and only +`sunset` is a judgement call. A gate with no explicit sunset gets +flipped + KEEP_AFTER_FLIP, which is the release after mirrors have had a cycle +to object. + +Sunset policy, all in versions (releases are about three weeks apart): + + KEEP_AFTER_FLIP = 4 once the default is TRUE the feature is public. Keep + the flag four more releases so a mirror or hgwbeta can + switch it back off without a code change, then delete + it along with every branch that reads it. + QA_GRACE = 6 a gate still defaulting FALSE this long after it landed + is not being gated, it is being forgotten. Either turn + it on or delete the feature. + +Verification status. Rows carry verified=True only where the classification +was confirmed by reading the code at the cited file:line. --check counts what +is left, because a variable described as a permanent knob when it is really a +forgotten gate defeats the purpose of the exercise. + +Usage: + hgConfCatalog.py --json out.json + hgConfCatalog.py --html out.html + hgConfCatalog.py --check # counts and internal consistency + hgConfCatalog.py --reconcile # diff the catalog against the tree + hgConfCatalog.py --sunset # what should be deleted, and when +""" + +import argparse +import html +import json +import os +import sys + +# Sunset policy, in releases. See the module docstring. +KEEP_AFTER_FLIP = 4 +QA_GRACE = 6 + + +# --------------------------------------------------------------------------- +# helpers +# --------------------------------------------------------------------------- + +def h(name, kind, src, default=None, note=None, public=False, verified=False, + role=None, sunset=None, env=None, deprecated=False, family=None, + required=False, ticket=None, debatable=None): + """One catalog entry. + + name the hg.conf setting name + kind what sort of setting: path, table, profile, credential, url, + email, limit, flag, branding, debug, internal, dead + src file:line where the tree reads it + default compiled-in default if the read supplies one + role for boolean flags only: "gate" (a release gate, temporary by + intent, subject to sunsetting) or "knob" (a permanent + deployment switch, exempt). Every cfgOptionBooleanDefault + flag in the tree must be one or the other; --reconcile + enforces that + sunset release by which a gate should be gone from the tree. Omit to + take the default of flipped + KEEP_AFTER_FLIP; the age cache + supplies the flip version + public documented for mirror operators in product/ex.hg.conf, or + belongs there + verified True if the classification was confirmed by reading src + env environment variable that overrides it, via cfgOptionEnv + family groups members of one prefix family, for the docs + required read with cfgVal, so the CGI dies if it is absent + deprecated the feature it configures is gone or going + ticket Redmine ticket that introduced or tracks it + debatable why this row's gate-or-knob call could reasonably go the other + way. Set it rather than picking a side quietly: the whole + point of separating the two is that someone has decided, and a + decision nobody argued with is not the same as a decision. + --check lists these as the review agenda + """ + d = {"name": name, "kind": kind, "src": src, "public": public, + "verified": verified} + for key, val in (("default", default), ("note", note), ("role", role), + ("sunset", sunset), ("env", env), ("family", family), + ("ticket", ticket), ("debatable", debatable)): + if val is not None: + d[key] = val + if required: + d["required"] = True + if deprecated: + d["deprecated"] = True + return d + + +# --------------------------------------------------------------------------- +# how hg.conf is read at all +# --------------------------------------------------------------------------- + +ACCESSORS = { + "cfgOption": "Returns the value or NULL. Absent means off.", + "cfgOptionDefault": "Returns the value or a compiled-in default.", + "cfgOptionBooleanDefault": + "Boolean with a compiled-in default. Accepts yes/no, on/off, " + "true/false. This is the accessor used to gate a release.", + "cfgVal": "Returns the value or errAborts. The CGI will not run without it.", + "cfgOptionEnv": + "Environment variable first, then hg.conf. Note the argument order: " + "the environment name comes first and is not an hg.conf setting.", + "cfgOptionEnvDefault": "As cfgOptionEnv, with a compiled-in default.", + "cfgOption2": + "Reads prefix.suffix. The prefix is usually a runtime value, which is " + "how one call site serves db.host, central.host and every other " + "profile at once.", + "cfgOptionDefault2": "As cfgOption2, with a compiled-in default.", +} + +BOUNDARY = ( + "hg.conf is read once per CGI invocation and is never written by the " + "browser. It is not user state: nothing in a session, a saved session or " + "a URL can change it, which is exactly why it is the right place to gate " + "a feature during a release. A mirror's copy is outside our control, so " + "removing a variable has to be treated as an interface change, not a " + "cleanup. Precedence for a value is: the environment (only for the " + "cfgOptionEnv settings, and only where the CGI allows it), then hg.conf, " + "then the compiled-in default." +) + +PROFILE_SUFFIXES = { + "what": + "A database profile is a set of hg.conf settings sharing one prefix, " + "read through cfgOption2(profileName, suffix) where profileName is a " + "runtime value. So these suffixes are legal under any profile prefix, " + "and none of the resulting names appears as a literal anywhere in the " + "tree. This is why product/ex.hg.conf documents " + "archivecentral.password while a search of the source finds nothing.", + "src": "hg/lib/jksql.c:231", + "suffixes": ["host", "port", "socket", "user", "password", "db", + "verifyServerCert", "ca", "caPath", "cert", "key", "cipher", + "crl", "crlPath"], + "knownProfiles": ["db", "central", "cart", "customTracks", "archivecentral", + "backupcentral", "myStuff", "myGenome", "rrcentral", "pq", + "rtdb", "cdw"], +} + + +# --------------------------------------------------------------------------- +# release gates: boolean flags that exist to hold a feature back +# --------------------------------------------------------------------------- +# These are the reason this catalog has a sunset mode. Each was added so a +# user-visible change could sit on master without shipping. Ordered by age so +# the backlog is visible at a glance. + +RELEASE_GATES = { + "what": "Boolean flags introduced to ship a feature dark during a release. " + "Temporary by intent: each should be deleted once the feature it " + "guards is public and mirrors have had a cycle to object.", + "vars": [ + h("showMouseovers", "flag", "hg/hgTracks/config.c:671", default="FALSE", + role="gate", verified=True, + note="Mouseover text on track items instead of the browser's own " + "title tooltips. Added v446 and still defaulting FALSE, which " + "makes it the oldest gate in the tree that never shipped. Four " + "call sites across config.c and imageV2.c. Either the feature " + "is wanted, in which case flip it, or it is not, in which case " + "the flag and the code behind it should go."), + h("storeUserFiles", "flag", "hg/hgHubConnect/hgHubConnect.c:1730", + default="FALSE", role="gate", verified=True, + note="Hub space, the user file store behind hgHubConnect's upload " + "wizard. Added v447 and briefly defaulted TRUE around v454 " + "before going back to FALSE, so the history shows a flip that " + "was reverted. Four call sites."), + h("trustTrackDb", "flag", "hg/lib/hdb.c:4130", default="FALSE", + role="gate", verified=True, + note="Skip the check that a trackDb row's table actually exists. A " + "speed optimisation for machines whose trackDb is known good. " + "Flipped TRUE at v458 and back to FALSE since.", + debatable="Reads as much like a knob as a gate: whether a machine's " + "trackDb can be trusted is a property of that machine, " + "not of a feature waiting to ship. If it is a knob it " + "should stop appearing in the stalled list."), + h("hgSession.shortLink", "flag", "hg/hgSession/hgSession.c:175", + default="FALSE", role="gate", verified=True, + note="Short session links. Added v374 and never flipped, which is " + "the longest-running dark feature here."), + h("showHubApiKey", "flag", "hg/hgHubConnect/hgHubConnect.c:571", + default="FALSE", role="gate", verified=True, + note="Expose the hub API key UI. Shares its call site with " + "storeUserFiles, so the two should be retired together."), + h("blatShowLocus", "flag", "hg/hgBlat/hgBlat.c:784", default="FALSE", + role="gate", verified=True, + note="Show the genomic locus alongside BLAT results."), + h("genarkLiftOver", "flag", "hg/lib/genark.c:413", default="FALSE", + role="gate", verified=True, + note="Offer liftOver between GenArk assemblies. Four call sites in " + "genark.c and hdb.c."), + h("showIgv", "flag", "hg/hgTracks/hgTracks.c:12116", default="FALSE", + role="gate", verified=True, + note="An IGV link in the track hamburger menus."), + h("groupDropdown", "flag", "hg/hgTracks/hgTracks.c:10152", + default="FALSE", role="gate", verified=True, + note="Track group chooser as a dropdown rather than the current " + "layout."), + h("gcOnTheFlyCoExist", "flag", "hg/hgTracks/hgTracks.c:7515", + default="FALSE", role="gate", verified=True, + note="Let the calculated GC track coexist with the stored one. A " + "sub-flag of gcOnTheFly, so it should be deleted with it " + "rather than outliving it."), + h("showAliases", "flag", "hg/hgTracks/hgTracks.c:9824", default="FALSE", + role="gate", verified=True, + note="Show chromosome alias names in the position box."), + h("showColorPicker", "flag", "hg/lib/hui.c:6066", default="FALSE", + role="gate", verified=True, + note="The track colour picker in track UI."), + h("doMyVariants", "flag", "hg/hgCustom/hgCustom.c:1226", + default="FALSE", role="gate", verified=True, + note="The My Variants track and its upload path. Thirteen call " + "sites across seven files, the widest gate in the tree, which " + "is a fair measure of what deleting a stale one costs."), + h("hguidIpTracking.enabled", "flag", "hg/lib/botDelay.c:157", + default="FALSE", role="gate", verified=True, + note="Per-hguid IP tracking for abuse detection. Its three " + "companion settings (maxIps, table, windowSeconds) are plain " + "values and are listed under abuse control."), + h("canColorItems", "flag", "hg/hgTracks/hgTracks.c:9124", + default="FALSE", role="gate", verified=True, + note="Added in the current release, so it is doing exactly what a " + "gate is supposed to do and has not earned a deadline yet."), + # Gates whose default has flipped TRUE. These are the deletable ones: + # the feature is public and the flag is now only an off switch. + h("showTutorial", "flag", "hg/hgCustom/hgCustom.c:180", default="TRUE", + role="gate", verified=True, + note="The interactive tutorials. Public since v466, five call " + "sites across four CGIs. Nothing is gating any more."), + h("canDupTracks", "flag", "hg/lib/dupTrack.c:257", default="TRUE", + role="gate", verified=True, + note="Duplicate-track feature. Public since v443."), + h("canSnake", "flag", "hg/hgc/hgc.c:3928", default="TRUE", role="gate", + verified=True, + note="Snake display for chain and alignment tracks. Public since " + "v467."), + h("showDownloadUi", "flag", "hg/hgTracks/hgTracks.c:8999", + default="TRUE", role="gate", verified=True, + note="The download-current-track UI. Public since v467."), + h("mergeRecommended", "flag", "hg/hgTracks/recTrackSets.c:194", + default="TRUE", role="gate", verified=True, + note="Merge behaviour for recommended track sets. Public since " + "v467."), + h("svgBarChart", "flag", "hg/hgc/barChartClick.c:584", default="TRUE", + role="gate", verified=True, + note="SVG rather than raster bar charts on the details page. " + "Public since v428, the longest-shipped gate still in place."), + h("canDoHgcInPopUp", "flag", "hg/hgTracks/config.c:792", default="TRUE", + role="gate", verified=True, + note="Details pages in a popup instead of a page load. Public " + "since v492. Three call sites."), + h("greyBarIcons", "flag", "hg/hgTracks/hgTracks.c:10404", + default="TRUE", role="gate", verified=True, + note="The grey side-bar icons on track images. Public since v492. " + "Four call sites in hgTracks.c and imageV2.c."), + h("bigBedOnePath", "flag", "hg/hgTracks/bigBedTrack.c:1110", + default="TRUE", role="gate", verified=True, + note="Single code path for bigBed fetching, replacing the older " + "split. Public since v492. Four call sites, and deleting it " + "removes a whole alternative path rather than just a branch."), + h("trackHubsCanAddGroups", "flag", "hg/lib/hubConnect.c:40", + default="TRUE", role="gate", verified=True, + note="Let hubs declare their own track groups. Public since v492."), + h("newBotDelay", "flag", "hg/lib/botDelay.c:215", default="TRUE", + role="gate", verified=True, + note="The reworked bot-delay logic. Public since v492."), + h("sleepOn429", "flag", "hg/lib/botDelay.c:423", default="TRUE", + role="gate", verified=True, + note="Sleep rather than reject on a rate-limit hit. Defaulted TRUE " + "from the start at v481.", + debatable="Never defaulted FALSE, so it never actually gated " + "anything. A flag born TRUE is either a knob or a " + "leftover from a change that was made unconditionally; " + "worth deciding which."), + h("gcOnTheFly", "flag", "hg/hgTracks/hgTracks.c:7514", default="TRUE", + role="gate", verified=True, + note="Calculate the GC percent track at draw time instead of " + "reading a stored table. Public since v496, so it is inside " + "its grace period."), + h("useBlatBigPsl", "flag", "hg/hgBlat/hgBlat.c:480", default="TRUE", + role="gate", verified=True, + note="bigPsl output from BLAT. Public since v348."), + h("alwaysItemRgb", "flag", "hg/cgilib/bedCart.c:34", default="TRUE", + role="gate", verified=True, + note="Honour a BED's itemRgb without requiring the track setting. " + "Defaulted TRUE from v466.", + debatable="Like sleepOn429, born TRUE and so never gated anything."), + h("hgHubConnect.validateHub", "flag", + "hg/hgHubConnect/hgHubConnect.c:1728", default="TRUE", role="gate", + verified=True, + note="Run hubCheck when a hub is attached. Public since v427.", + debatable="A mirror might legitimately want to skip hub validation " + "for speed or because it attaches only hubs it controls, " + "which would make this a knob."), + h("forceTwoBit", "flag", "hg/lib/hdb.c:1224", default="TRUE", + role="gate", verified=True, + note="Require 2bit sequence rather than falling back to nib. Public " + "since v456.", + debatable="Its companion allowNib is classified as a knob, and both " + "settings control the same nib fallback. One of the two " + "is filed wrong; they should be decided together."), + h("freeType", "flag", "hg/cgilib/trackLayout.c:65", default="TRUE", + role="gate", verified=True, + note="FreeType font rendering in track images rather than the built " + "in bitmap fonts. Public since v412. Its companions " + "freeTypeDir and freeTypeFont are plain values and stay.", + debatable="A mirror without the URW fonts installed has to be able " + "to turn this off, which is knob behaviour. Deleting the " + "flag would remove the bitmap font path entirely."), + ], +} + + +# --------------------------------------------------------------------------- +# mirror knobs: boolean flags that are meant to live forever +# --------------------------------------------------------------------------- + +MIRROR_KNOBS = { + "what": "Boolean flags that are legitimate, permanent deployment switches. " + "Listed explicitly so the sunset report does not nag about them.", + "vars": [ + h("isGbib", "flag", "hg/lib/hdb.c:3712", default="FALSE", role="knob", + public=True, verified=True, + note="This is the Genome Browser in a Box. Changes paths and " + "disables features that make no sense on a VM."), + h("isGbic", "flag", "hg/lib/hdb.c:3718", default="FALSE", role="knob", + public=True, verified=True, + note="This is a Genome Browser in the Cloud install."), + h("allowNib", "flag", "hg/lib/hdb.c:2770", default="TRUE", role="knob", + public=True, verified=True, + note="Permit nib sequence files. Ancient, but an old mirror may " + "still hold nib assemblies, so it stays."), + h("browser.dumpStack", "flag", "hg/lib/hCommon.c:370", default="FALSE", + role="knob", public=True, verified=True, + note="Dump a stack trace to the error log on a crash. A debugging " + "switch an operator turns on when needed."), + h("showEarlyErrors", "flag", "hg/lib/hgConfig.c:395", default="FALSE", + role="knob", public=True, verified=True, + note="Show errors that happen before the HTML header is written. " + "Off in production because it leaks internals; on when " + "debugging a CGI that dies immediately."), + h("suppressVeryEarlyErrors", "flag", "hg/lib/hgConfig.c:398", + default="FALSE", role="knob", verified=True, + note="The opposite switch, for hiding a broken hg.conf from users."), + h("hgta.disableAllTables", "flag", "hg/lib/hCommon.c:419", + default="FALSE", role="knob", public=True, verified=True, + note="Remove the all-tables option from the Table Browser. A load " + "control a mirror is entitled to set."), + h("hgta.disableSendOutput", "flag", "hg/hgTables/mainPage.c:449", + default="FALSE", role="knob", public=True, verified=True, + note="Remove the send-output-to-Galaxy destinations."), + h("udc.useLocalDiskCache", "flag", "hg/lib/hui.c:656", default="TRUE", + role="knob", public=True, verified=True, + note="Use the local UDC cache. A mirror on a read-only filesystem " + "turns this off."), + h("db.neverLocal", "flag", "hg/lib/jksql.c:2141", default="0", + role="knob", verified=True, + note="Never treat the database as local, so no local file " + "shortcuts. Deployment topology, not a feature."), + h("traceGbdb", "flag", "hg/lib/hdb.c:1594", default="FALSE", + role="knob", verified=True, + note="Log every /gbdb file the CGI opens. A diagnostic."), + h("drawDot", "flag", "hg/hgc/hgc.c:3452", default="FALSE", role="knob", + verified=True, + note="Emit graphviz dot output from the details page instead of a " + "rendered image. A developer diagnostic."), + h("login.https", "flag", "hg/lib/wikiLink.c:295", default="TRUE", + role="knob", public=True, verified=True, + note="Require https for login. A mirror without a certificate has " + "to be able to turn this off."), + h("login.basicAuth", "flag", "hg/lib/wikiLink.c:43", default="FALSE", + role="knob", public=True, verified=True, + note="Take identity from HTTP basic auth rather than the login " + "system."), + h("login.relativeLink", "flag", "hg/lib/hdb.c:3650", default="FALSE", + role="knob", public=True, verified=True, + note="Relative rather than absolute login links."), + h("login.acceptAnyId", "flag", "hg/lib/wikiLink.c:248", + default="FALSE", role="knob", verified=True, + note="Accept any identity token. Development only, and dangerous " + "on a public machine."), + h("login.acceptIdx", "flag", "hg/lib/wikiLink.c:255", default="FALSE", + role="knob", verified=True, note="Companion to login.acceptAnyId."), + h("login.pwdEyeIcon", "flag", "hg/hgLogin/hgLogin.c:1429", + default="TRUE", role="knob", verified=True, + note="Show-password eye icon on the login form."), + h("analytics.trackClicks", "flag", "hg/lib/googleAnalytics.c:63", + default="TRUE", role="knob", verified=True, + note="Report link clicks to analytics. A mirror with its own " + "privacy policy turns this off."), + h("analytics.trackButtons", "flag", "hg/lib/googleAnalytics.c:64", + default="TRUE", role="knob", verified=True, + note="Report button presses to analytics."), + h("wikiTrack.readOnly", "flag", "hg/lib/wikiTrack.c:292", + default="FALSE", role="knob", verified=True, deprecated=True, + note="Make the wiki annotation track read-only. The wiki track " + "itself is effectively retired."), + h("cdw.siteIsPublic", "flag", + "hg/cirm/cdw/cdwGetFile/cdwGetFile.c:62", default="FALSE", + role="knob", verified=True, deprecated=True, + note="CIRM data warehouse is public. Belongs to the cirm CGIs, " + "which are not part of the browser release."), + h("multiRegionButtonTop", "flag", "hg/hgTracks/config.c:990", + default="FALSE", role="knob", public=True, verified=True, + note="Put the multi-region button in the top bar.", + debatable="Read twice with two different compiled-in defaults, which " + "is a smell either way. If it was a layout gate that was " + "never flipped it belongs in the stalled list rather than " + "here."), + h("autoBlatBigPsl", "flag", "hg/hgBlat/hgBlat.c:2629", + default="autoBigPsl", role="knob", verified=True, + note="Its default is another variable rather than a literal, so the " + "harvester reports the identifier.", + debatable="Because the default is a variable, nothing here can say " + "whether it is on or off in practice. Needs a read of " + "hgBlat.c before it can be classified with any " + "confidence."), + h("ignoreDefaultKnown", "flag", "hg/lib/hdb.c:6144", default="FALSE", + role="knob", verified=True, + note="Ignore the default known-genes setting when resolving the gene " + "track. Used on machines with unusual gene tables.", + debatable="Filed as a knob on the strength of its comment. If it " + "was added for one assembly's migration it is really an " + "expired gate."), + ], +} + + +# --------------------------------------------------------------------------- +# database connections and profiles +# --------------------------------------------------------------------------- + +DATABASE = { + "what": "Where the CGIs find MySQL. A profile is a prefix; see the " + "profile suffix family for the settings legal under any of them.", + "vars": [ + h("db.host", "profile", "hg/qaPushQ/qaPushQ.c:2435", public=True, + verified=True, env="HGDB_HOST", family="db", + note="The main assembly database server."), + h("db.user", "profile", "hg/hgc/hgc.c:1295", public=True, + verified=True, env="HGDB_USER", family="db"), + h("db.password", "credential", "hg/hgc/hgc.c:1296", public=True, + verified=True, env="HGDB_PASSWORD", family="db"), + h("db.trackDb", "table", "hg/lib/hdb.c:352", public=True, + verified=True, env="HGDB_TRACKDB", family="db", + note="Comma-separated list of trackDb tables, searched in order. " + "This is how a developer layers a personal trackDb over " + "production, and why a sandbox hg.conf can be much slower than " + "the CGI's own."), + h("db.metaDb", "table", "hg/lib/mdb.c:945", verified=True, family="db"), + h("db.relatedTrack", "table", "hg/lib/hui.c:10772", + default='"relatedTrack"', verified=True, family="db"), + h("central.host", "profile", "hg/qaPushQ/qaPushQ.c:2448", public=True, + verified=True, family="central", + note="hgcentral, which holds sessions, users, hub status and the " + "assembly list."), + h("central.user", "profile", "hg/qaPushQ/qaPushQ.c:2449", public=True, + verified=True, family="central"), + h("central.password", "credential", "hg/qaPushQ/qaPushQ.c:2450", + public=True, verified=True, family="central"), + h("central.db", "profile", "hg/hgc/lowelab.c:2362", public=True, + verified=True, family="central"), + h("central.domain", "internal", "hg/hubApi/apiUtils.c:854", + public=True, verified=True, required=True, family="central", + note="Cookie domain for the central cookie. Read with cfgVal in " + "hubApi, so that CGI will not start without it."), + h("central.cookie", "internal", "hg/lib/hui.c:636", default='"hguid"', + public=True, verified=True, family="central", + note="Name of the user-identity cookie. Changing it logs every " + "user out."), + h("cart.host", "profile", "hg/lib/hdb.c:937", public=True, + verified=True, family="cart", + note="Optional separate server for cart traffic, which is the " + "heaviest write load in the browser."), + h("cart.user", "profile", "hg/lib/hdb.c:938", public=True, + verified=True, family="cart"), + h("cart.password", "credential", "hg/lib/hdb.c:938", public=True, + verified=True, family="cart"), + h("cart.db", "profile", "hg/lib/hdb.c:937", public=True, verified=True, + family="cart"), + h("customTracks.host", "profile", "product/ex.hg.conf", public=True, + family="customTracks", + note="Read through the profile mechanism, so it has no literal call " + "site in the tree."), + h("customTracks.user", "profile", "product/ex.hg.conf", public=True, + family="customTracks"), + h("customTracks.password", "credential", "product/ex.hg.conf", + public=True, family="customTracks"), + h("customTracks.tmpdir", "path", "hg/lib/customAdjacency.c:138", + default='"/data/tmp"', public=True, verified=True, + family="customTracks"), + h("customTracks.maxBytes", "limit", "hg/lib/customFactory.c:2448", + public=True, verified=True, family="customTracks"), + h("customTracks.useAll", "flag", "hg/lib/customTrack.c:167", + default="NULL", public=True, verified=True, family="customTracks"), + h("customTracks.botCheckMult", "limit", "hg/lib/customTrack.c:1043", + default='"1"', verified=True, family="customTracks", + note="Multiplier on the bot-delay penalty for custom track loads."), + h("showTableCache", "table", "hg/lib/jksql.c:850", + default='"tableList"', public=True, verified=True, required=True, + note="Table holding a cached list of table names, which avoids a " + "slow SHOW TABLES. One of the four settings read with cfgVal."), + ], +} + + +# --------------------------------------------------------------------------- +# hgcentral table names +# --------------------------------------------------------------------------- + +CENTRAL_TABLES = { + "what": "Names of the hgcentral tables. Nearly all are overridable from " + "the environment as well, which is how a test instance points at " + "its own copies without editing hg.conf.", + "vars": [ + h("dbDbTableName", "table", "hg/lib/hdb.c:87", public=True, + verified=True, env="HGDB_DBDBTABLE", default="dbDb", + note="The assembly list."), + h("defaultDbTableName", "table", "hg/lib/hdb.c:107", public=True, + verified=True, env="HGDB_DEFAULTDBTABLE", default="defaultDb"), + h("cladeTableName", "table", "hg/lib/hdb.c:117", public=True, + verified=True, env="HGDB_CLADETABLE", default="clade"), + h("genomeCladeTableName", "table", "hg/lib/hdb.c:97", public=True, + verified=True, env="HGDB_GENOMECLADETABLE", default="genomeClade"), + h("userDbName", "table", "hg/lib/cartDb.c:329", verified=True, + env="HGDB_USERDBTABLE", default="userDb", + note="Per-user cart storage."), + h("sessionDbName", "table", "hg/lib/cartDb.c:339", verified=True, + env="HGDB_SESSIONDBTABLE", default="sessionDb"), + h("defaultCartName", "table", "hg/lib/cartDb.c:319", public=True, + verified=True, env="HGDB_DEFAULTCARTTABLE"), + h("namedSessionDbName", "table", "hg/lib/cart.c:382", verified=True, + env="HGDB_NAMED_SESSION_DB", default="namedSessionDb", + note="Saved sessions."), + h("hub.publicTableName", "table", + "hg/hgHubConnect/hgHubConnect.c:1388", public=True, verified=True, + env="HGDB_HUB_PUBLIC_TABLE", family="hub"), + h("hub.statusTableName", "table", + "hg/hgHubConnect/hgHubConnect.c:1390", public=True, verified=True, + env="HGDB_HUB_STATUS_TABLE", family="hub"), + h("hub.genArkTableName", "table", "hg/lib/genark.c:347", + verified=True, env="HGDB_GENARK_STATUS_TABLE", family="hub"), + h("hub.assemblyListTableName", "table", "hg/lib/assemblyList.c:433", + verified=True, env="HGDB_ASSEMBLYLIST_STATUS_TABLE", family="hub"), + h("liftOverChainName", "table", "hg/lib/liftOver.c:1984", + verified=True, env="LIFTOVERCHAINNAME", default="liftOverChain"), + h("quickLiftChainName", "table", "hg/lib/quickLift.c:582", + verified=True, env="QUICKLIFTCHAINNAME", default="quickLiftChain", + ticket="37788"), + h("blatServersTbl", "table", "hg/hgPcr/hgPcr.c:122", + default='"blatServers"', verified=True), + h("hubSearchTextTable", "table", "hg/hgGateway/hgGateway.c:930", + default='"hubSearchText"', verified=True), + h("authTableName", "table", "hg/lib/hubSpaceKeys.c:132", + verified=True, note="Hub space API keys."), + h("ottoTable", "table", "hg/hubApi/apiUtils.c:884", verified=True, + note="Table the hub API reads to report otto track update times."), + h("genbankDb", "profile", "hg/hgVai/hgVai.c:803", public=True, + verified=True, env="GENBANKDB"), + h("cart.trace", "debug", "hg/lib/cart.c:106", verified=True, + note="Log every cart read and write. Very noisy; a debugging aid " + "for session problems."), + h("cartVersion", "internal", "hg/cgilib/cartRewrite.c:46", + default='"on"', verified=True, + note="Run the cart rewrite steps that migrate old sessions " + "forward. Turning it off strands old sessions."), + h("browser.sessionKey", "internal", "hg/lib/cartDb.c:76", + verified=True, + note="Read through cfgOption2 with a literal prefix, so it is one " + "of the few two-part names that does appear as a literal."), + ], +} + + +# --------------------------------------------------------------------------- +# paths and caches +# --------------------------------------------------------------------------- + +PATHS = { + "what": "Where things live on disk. These are the settings a mirror is " + "most likely to have to change.", + "vars": [ + h("gbdbLoc1", "path", "hg/lib/hdb.c:1551", public=True, verified=True, + note="Primary /gbdb location. Any C code opening a /gbdb path is " + "supposed to run it through hReplaceGbdb() so this takes " + "effect."), + h("gbdbLoc2", "path", "hg/lib/hdb.c:1582", public=True, verified=True, + note="Fallback /gbdb location, tried when a file is missing from " + "gbdbLoc1. This is how a mirror keeps part of /gbdb local and " + "the rest remote."), + h("udc.cacheDir", "path", "hg/lib/hui.c:660", default="udcDefaultDir()", + public=True, verified=True, + note="UDC cache for remote bigData files. Wants real disk: this is " + "where every hub file lands."), + h("udc.localDir", "path", "hg/lib/customFactory.c:204", public=True, + verified=True), + h("udcLog", "debug", "hg/hgTracks/hgTracks.c:12061", verified=True, + note="Log UDC fetches, which is the first thing to turn on when a " + "hub is slow."), + h("cacheTrackDbDir", "path", "hg/lib/trackDbCache.c:483", + default='"/dev/shm/trackDbCache"', verified=True, + note="Shared-memory cache of parsed trackDb. Setting it empty " + "forces a fresh read from MySQL every request, which is the " + "right way to test trackDb changes; deleting the directory is " + "not, since it is shared between users."), + h("sessionDataDir", "path", "hg/hgPcr/hgPcr.c:564", verified=True, + note="Where session-scoped data (custom tracks belonging to a saved " + "session) is kept so it survives cleanup."), + h("sessionDataDirOld", "path", "hg/lib/customFactory.c:180", + verified=True, note="Previous location, still read so old sessions " + "keep working."), + h("sessionDataDbPrefix", "internal", "hg/lib/sessionData.c:474", + verified=True), + h("sessionThumbnail.imgDir", "path", "hg/cgilib/sessionThumbnail.c:29", + public=True, verified=True, family="sessionThumbnail"), + h("sessionThumbnail.webPath", "url", + "hg/cgilib/sessionThumbnail.c:30", public=True, verified=True, + family="sessionThumbnail"), + h("sessionThumbnail.convertPath", "path", + "hg/hgSession/hgSession.c:1153", public=True, verified=True, + family="sessionThumbnail"), + h("sessionThumbnail.suppress", "flag", + "hg/hgSession/hgSession.c:1149", public=True, verified=True, + family="sessionThumbnail"), + h("freeTypeDir", "path", "hg/hgTracks/config.c:164", + default='"../htdocs/urw-fonts"', verified=True), + h("freeTypeFont", "internal", "hg/cgilib/trackLayout.c:67", + default='"Bitmap"', verified=True), + h("fonts.extra", "path", "hg/cgilib/trackLayout.c:49", default="NULL", + public=True, verified=True), + h("textSize", "internal", "hg/cgilib/trackLayout.c:80", + default='"small"', verified=True), + h("tusdDataDir", "path", "hg/lib/userdata.c:115", verified=True, + family="hubSpace", note="Hub space upload staging."), + h("tusdMountPoint", "path", "hg/lib/userdata.c:116", verified=True, + family="hubSpace"), + h("hubSpaceUrl", "url", "hg/lib/userdata.c:176", verified=True, + family="hubSpace"), + h("hubSpaceTusdEndpoint", "url", + "hg/hgHubConnect/trackHubWizard.c:260", default="NULL", + verified=True, family="hubSpace"), + h("myVariantsDataDir", "path", "hg/lib/myVariants.c:783", + verified=True, note="Goes with the doMyVariants gate."), + h("hgPhyloPlaceServerDir", "path", "hg/hgPhyloPlace/runUsher.c:1067", + verified=True), + h("browser.documentRoot", "path", "hg/lib/hui.c:692", + default="DOCUMENT_ROOT", public=True, verified=True), + h("browser.cgiRoot", "path", "hg/lib/hui.c:759", default="defaultDir", + public=True, verified=True), + h("browser.javaScriptDir", "path", "hg/lib/web.c:1441", + default='"js"', public=True, verified=True), + h("browser.styleDir", "path", "hg/lib/web.c:1381", default='"style"', + public=True, verified=True), + h("browser.styleImagesDir", "path", "hg/lib/web.c:1445", + default='"style/images"', public=True, verified=True), + h("browser.trixPath", "path", "hg/cgilib/search.c:19", + default='"/gbdb/$db/trackDb.ix"', public=True, verified=True, + note="Track search index. The $db is substituted at runtime."), + h("downloads.server", "url", "hg/lib/hui.c:642", + default='"hgdownload.soe.ucsc.edu"', public=True, verified=True), + h("cramRef", "path", "hg/hgTables/bam.c:180", public=True, + verified=True, note="Reference sequence cache for CRAM."), + h("grepIndex.default", "path", "hg/lib/hgFind.c:168", public=True, + verified=True, family="grepIndex"), + h("grepIndex.genbank", "path", "hg/lib/hgFind.c:1201", public=True, + verified=True, family="grepIndex"), + ], +} + + +# --------------------------------------------------------------------------- +# limits and load control +# --------------------------------------------------------------------------- + +LIMITS = { + "what": "Caps on what one request may consume. These are the settings " + "that decide whether a heavy request is answered slowly or " + "refused.", + "vars": [ + h("maxMem", "limit", "hg/lib/hgConfig.c:376", public=True, + verified=True, + note="Address-space cap applied by cfgSetMaxMem() at CGI startup. " + "Exceeding it is what produces the hogExit entries in the " + "error log."), + h("warnSeconds", "limit", "hg/hgTracks/hgTracks.c:10786", + verified=True, note="Log a warning for any hgTracks render slower " + "than this."), + h("maxItemsPossible", "limit", "hg/hgTracks/simpleTracks.c:830", + default='"100000"', public=True, verified=True), + h("BAMMaxItems", "limit", "hg/hgTracks/bamTrack.c:54", + default='"10000"', verified=True), + h("bigBedMaxItems", "limit", "hg/hgTracks/bigBedTrack.c:481", + default='"10000"', public=True, verified=True), + h("vcfMaxItems", "limit", "hg/hgTracks/vcfTrack.c:3023", + default='"10000"', public=True, verified=True), + h("maxTrackImageHeightPx", "limit", "hg/hgTracks/hgTracks.c:5321", + default='"32000"', verified=True, + note="Hard ceiling on image height, which is what stops a dense " + "track from producing a PNG no browser will render."), + h("maxDisplayPixelWidth", "limit", "hg/cgilib/trackLayout.c:20", + default="NULL", public=True, verified=True), + h("barbMergePixels", "limit", "hg/hgTracks/simpleTracks.c:4481", + default='"3"', public=True, verified=True), + h("quickLift.lengthLimit", "limit", "hg/lib/quickLift.c:446", + default='"10000"', verified=True, ticket="37788"), + h("liftDailyLimit", "limit", "hg/hubApi/apiUtils.c:908", verified=True, + note="Per-day liftOver cap for the hub API."), + h("hgBlat.maxSequenceCount", "limit", "hg/hgBlat/hgBlat.c:1776", + default="NULL", public=True, verified=True), + h("parallelFetch.threads", "limit", "hg/hgBlat/hgBlat.c:2442", + default='"20"', public=True, verified=True), + h("parallelFetch.timeout", "limit", "hg/hgBlat/hgBlat.c:2473", + default='"90"', public=True, verified=True), + h("logCgiVarMaxLen", "limit", "hg/lib/hgConfig.c:386", default='"0"', + public=True, verified=True, + note="Truncate logged CGI variables at this length. Zero disables " + "the logging that cfgSetLogCgiVars() would otherwise do."), + ], +} + + +# --------------------------------------------------------------------------- +# abuse control +# --------------------------------------------------------------------------- + +ABUSE = { + "what": "Bot delay, rate limiting and captchas. Mostly read from " + "hg/lib/botDelay.c.", + "vars": [ + h("bottleneck.host", "url", "hg/lib/botDelay.c:331", public=True, + verified=True, family="bottleneck", + note="The bottleneck server that tracks per-IP request rates."), + h("bottleneck.port", "internal", "hg/lib/botDelay.c:332", public=True, + verified=True, family="bottleneck"), + h("bottleneck.except", "internal", "hg/lib/botDelay.c:273", + verified=True, family="bottleneck", + note="IPs exempt from delay."), + h("hguidIpTracking.maxIps", "limit", "hg/lib/botDelay.c:170", + default='"10"', verified=True, family="hguidIpTracking"), + h("hguidIpTracking.windowSeconds", "limit", "hg/lib/botDelay.c:171", + default='"600"', verified=True, family="hguidIpTracking"), + h("hguidIpTracking.table", "table", "hg/lib/botDelay.c:172", + default='"hguidIpAccess"', verified=True, family="hguidIpTracking"), + h("cloudFlareSiteKey", "credential", "hg/lib/cart.c:1526", + verified=True, required=True, + note="Turnstile captcha site key. Read with cfgVal, so a machine " + "that enables the captcha path must set it."), + h("cloudFlareSecretKey", "credential", "hg/lib/cart.c:1501", + verified=True, required=True), + h("noCaptchaAgent.", "internal", "hg/lib/botDelay.c:307", + verified=True, + note="A prefix family rather than one setting: every " + "noCaptchaAgent.* value is a user-agent string exempt from the " + "captcha. Enumerated at runtime with cfgNamesWithPrefix()."), + ], +} + + +# --------------------------------------------------------------------------- +# logging and diagnostics +# --------------------------------------------------------------------------- + +LOGGING = { + "what": "Diagnostics. Several of these are safe to leave on and a few " + "are expensive, so they are worth telling apart.", + "vars": [ + h("browser.cgiTime", "debug", "hg/lib/cart.c:3866", default='"yes"', + public=True, verified=True, + note="Log per-CGI timing. On by default and cheap; this is what " + "the log analysis relies on."), + h("trackLog", "debug", "hg/hgTracks/hgTracks.c:9228", default='"off"', + verified=True, note="Log which tracks were drawn per request."), + h("noSqlInj.level", "internal", "hg/lib/cart.c:2732", + default='"abort"', verified=True, family="noSqlInj", + note="What to do when the SQL injection guard fires: abort, warn or " + "ignore. Production wants abort."), + h("noSqlInj.dumpStack", "debug", "hg/lib/cart.c:2735", verified=True, + family="noSqlInj"), + h("signalsHandler", "internal", "hg/lib/cart.c:2695", public=True, + verified=True, note="Install handlers that turn a segfault into a " + "logged error rather than a blank page."), + h("httpsCertCheck", "internal", "hg/lib/cart.c:2699", public=True, + verified=True, family="httpsCertCheck", + note="How strictly to verify certificates on outbound https, which " + "matters because hubs are fetched over it."), + h("httpsCertCheckVerbose", "debug", "hg/lib/cart.c:2702", + verified=True, family="httpsCertCheck"), + h("httpsCertCheckDepth", "internal", "hg/lib/cart.c:2705", + verified=True, family="httpsCertCheck"), + h("httpsCertCheckDomainExceptions", "internal", "hg/lib/cart.c:2708", + public=True, verified=True, family="httpsCertCheck"), + h("httpProxy", "url", "hg/lib/cart.c:2715", public=True, + verified=True, family="proxy"), + h("httpsProxy", "url", "hg/lib/cart.c:2718", public=True, + verified=True, family="proxy"), + h("ftpProxy", "url", "hg/lib/cart.c:2721", public=True, verified=True, + family="proxy"), + h("noProxy", "internal", "hg/lib/cart.c:2724", public=True, + verified=True, family="proxy"), + h("logProxy", "debug", "hg/lib/cart.c:2727", verified=True, + family="proxy"), + ], +} + + +# --------------------------------------------------------------------------- +# branding and site text +# --------------------------------------------------------------------------- + +BRANDING = { + "what": "Text, styling and links that differ between UCSC and a mirror.", + "vars": [ + h("browser.style", "internal", "hg/lib/cart.c:2993", public=True, + verified=True, note="Stylesheet override."), + h("browser.theme.", "internal", "hg/hgTracks/config.c:31", + public=True, verified=True, + note="A prefix family: each browser.theme.N.Name value defines a " + "selectable theme. Enumerated with cfgNamesWithPrefix(), " + "which is why the individual names never appear in the source."), + h("addJs", "internal", "hg/lib/web.c:1587", verified=True, + note="Extra JavaScript file to include on every page."), + h("help.html", "path", "hg/lib/hui.c:702", verified=True), + h("hgTracksNoteHtml", "internal", "hg/hgTracks/hgTracks.c:9895", + public=True, verified=True, + note="A banner on the browser page. This is where a mirror puts " + "its own notice."), + h("survey", "url", "hg/hgGateway/hgGateway.c:371", public=True, + verified=True, env="HGDB_SURVEY", + note="Survey link on the gateway. Set to 'off' to hide it."), + h("surveyLabel", "internal", "hg/hgGateway/hgGateway.c:375", + default='"Please take our survey"', public=True, verified=True, + env="HGDB_SURVEY_LABEL"), + h("surveyLabelImage", "url", "hg/hgGateway/hgGateway.c:377", + verified=True), + h("hubSurvey", "url", "hg/hgHubConnect/hgHubConnect.c:1675", + verified=True, env="HGDB_HUB_SURVEY"), + h("hubSurveyLabel", "internal", + "hg/hgHubConnect/hgHubConnect.c:1676", verified=True, + env="HGDB_HUB_SURVEY_LABEL"), + h("searchHelpUrl", "url", "hg/hgTracks/hgTracks.c:8867", + default='"../goldenPath/help/query.html"', verified=True), + h("searchHelpLabel", "internal", "hg/hgTracks/hgTracks.c:8868", + default='"Examples"', verified=True), + h("analyticsKey", "credential", "hg/lib/googleAnalytics.c:13", + public=True, verified=True), + h("mouseOverEnabled", "internal", "hg/hgTracks/hgTracks.c:11996", + default='"on"', verified=True, + note="Not the same thing as the showMouseovers gate: this one is on " + "by default and controls the existing tooltip behaviour. The " + "near-identical names are a trap worth fixing."), + h("bigWarn", "internal", "hg/hgTracks/bigWarn.c:56", default='"on"', + public=True, verified=True), + h("defaultGenome", "internal", "hg/lib/hdb.c:513", + default="DEFAULT_GENOME", public=True, verified=True), + h("browser.popularGenomes", "internal", + "hg/hgIntegrator/hgIntegrator.c:956", + default='"hg38,hg19,mm39,mm10..."', verified=True), + h("geneTracks", "internal", "hg/lib/hgFind.c:3795", verified=True), + h("browser.recTrackSets", "internal", "hg/hgTracks/recTrackSets.c:58", + verified=True, family="recTrackSets", + note="Recommended track sets. Goes with the mergeRecommended " + "gate."), + h("browser.recTrackSetsDetectChange", "internal", + "hg/hgTracks/recTrackSets.c:66", verified=True, + family="recTrackSets", + note="Drives the 'session changed' banner. The mergeRecommended " + "gate belongs to the same feature."), + h("browser.exportedDataHubs", "internal", + "hg/lib/exportedDataHubs.c:181", verified=True), + h("browser.cgiExpireMinutes", "limit", + "hg/hgHubConnect/hgHubConnect.c:404", default='"20"', verified=True), + h("curatedHubPrefix", "internal", "hg/lib/hubConnect.c:1214", + verified=True, note="Which curated hubs this machine shows."), + h("genarkHubPrefix", "internal", "hg/hgGateway/hgGateway.c:1128", + verified=True), + h("test.preview", "flag", "hg/lib/hdb.c:3726", verified=True, + note="Marks a preview machine, which changes some banners and " + "links. Part of the release plumbing, but a permanent part."), + h("restoreMapFind", "internal", "hg/hgTracks/imageV2.c:496", + verified=True), + ], +} + + +# --------------------------------------------------------------------------- +# geographic mirroring +# --------------------------------------------------------------------------- + +GEO = { + "what": "Settings for the geographically distributed mirrors, which route " + "users to a nearby machine.", + "vars": [ + h("browser.node", "internal", "hg/lib/geoMirror.c:43", public=True, + verified=True, + note="Which node this machine is. Drives the geo redirect."), + h("browser.geoSuffix", "internal", + "hg/geoIpToCountry/geoIpToCountry.c:68", default='""', verified=True, + note="Suffix appended to central table names so each node can have " + "its own copies."), + ], +} + + +# --------------------------------------------------------------------------- +# login and wiki +# --------------------------------------------------------------------------- + +LOGIN = { + "what": "The login system, and the older wiki-based identity it replaced. " + "Most of these are read through macros, which is why the harvester " + "cannot date them.", + "vars": [ + h("login.systemName", "internal", "hg/lib/wikiLink.c:31", public=True, + verified=True, family="login"), + h("login.browserName", "internal", "hg/hgLogin/hgLogin.c:67", + public=True, verified=True, family="login"), + h("login.browserAddr", "url", "hg/hgLogin/hgLogin.c:76", public=True, + verified=True, family="login"), + h("login.mailSignature", "internal", "hg/hgLogin/hgLogin.c:85", + public=True, verified=True, family="login"), + h("login.mailReturnAddr", "email", "hg/hgLogin/hgLogin.c:96", + public=True, verified=True, family="login"), + h("login.approvedReturn", "url", "hg/hgLogin/hgLogin.c:326", + default="NULL", verified=True, family="login"), + h("login.cookieSalt", "credential", + "hg/hgPhyloPlace/hgPhyloPlace.c:581", public=True, verified=True, + family="login", note="Salt for the login cookie. A secret."), + h("wiki.host", "url", "hg/lib/wikiLink.c:199", public=True, + verified=True, family="wiki", deprecated=True), + h("wiki.userNameCookie", "internal", "hg/lib/wikiLink.c:50", + default='"hgLoginUserName"', public=True, verified=True, + family="wiki"), + h("wiki.loggedInCookie", "internal", "hg/lib/wikiLink.c:51", + default='"hgLoginIdKey"', public=True, verified=True, family="wiki"), + h("wiki.sessionCookie", "internal", "hg/lib/wikiTrack.c:342", + public=True, verified=True, family="wiki", deprecated=True), + h("wikiTrack.URL", "url", "hg/hgGene/wikiTrack.c:38", default="NULL", + public=True, verified=True, family="wikiTrack", deprecated=True), + h("wikiTrack.browser", "internal", "hg/hgGene/wikiTrack.c:297", + default="DEFAULT_BROWSER", public=True, verified=True, + family="wikiTrack", deprecated=True), + h("wikiTrack.dbList", "internal", "hg/lib/wikiTrack.c:318", + public=True, verified=True, family="wikiTrack", deprecated=True), + h("wikiTrack.editors", "internal", "hg/hgc/variomeClick.c:203", + default="NULL", public=True, verified=True, family="wikiTrack", + deprecated=True), + ], +} + + +# --------------------------------------------------------------------------- +# external services +# --------------------------------------------------------------------------- + +EXTERNAL = { + "what": "Hosts, URLs and helper programs outside the browser. Each is a " + "dependency that can fail independently of us.", + "vars": [ + h("galaxyUrl", "url", "hg/hgTables/galaxy.c:40", public=True, + verified=True), + h("rnaPlotPath", "path", "hg/hgGene/rnaStructure.c:65", + default='"../cgi-bin/RNAplot"', public=True, verified=True), + h("hgc.psxyPath", "path", "hg/hgc/hgdpClick.c:277", public=True, + verified=True, family="hgc"), + h("hgc.ps2rasterPath", "path", "hg/hgc/hgdpClick.c:297", public=True, + verified=True, family="hgc"), + h("hgc.ghostscriptPath", "path", "hg/hgc/hgdpClick.c:298", + public=True, verified=True, family="hgc"), + h("nextstrainHost", "url", "hg/hgPhyloPlace/phyloPlace.c:1456", + verified=True), + h("microbeTraceHost", "url", "hg/hgPhyloPlace/phyloPlace.c:1532", + verified=True), + h("hgPhyloPlaceEnabled", "flag", "hg/hgPhyloPlace/phyloPlace.c:331", + verified=True, + note="Read with cfgOption rather than the boolean accessor, so " + "absent means off. It gates a whole CGI, which is why it is " + "not in the gate list: hgPhyloPlace is optional by design, not " + "pending release."), + h("resolvProts", "internal", "hg/lib/hui.c:672", public=True, + verified=True, family="resolv"), + h("resolvPrefix", "internal", "hg/lib/hui.c:673", public=True, + verified=True, family="resolv"), + h("resolvCmd", "path", "hg/lib/hui.c:674", public=True, verified=True, + family="resolv"), + h("hubApi.allowHtml", "flag", "hg/hubApi/hubApi.c:1703", + default='"off"', verified=True, family="hubApi"), + h("hubApi.showActive0", "flag", "hg/hubApi/apiUtils.c:554", + default='"off"', verified=True, family="hubApi"), + h("hubApi.blatDelayFraction", "limit", "hg/hubApi/blat.c:312", + default="NULL", verified=True, family="hubApi"), + h("hubApi.relaySecret", "credential", "hg/hubApi/apiUtils.c:1016", + verified=True, family="hubApi"), + h("apiFromEmail", "email", "hg/hubApi/liftOver.c:528", verified=True), + h("chainFileRequestEmail", "email", "hg/hubApi/liftOver.c:527", + verified=True), + h("newCustomTrackValidate", "flag", "hg/lib/customFactory.c:773", + verified=True, + note="Use the newer custom track validator. Reads like a gate but " + "is set per machine on purpose while the two validators " + "coexist; worth revisiting."), + h("suggest.mailToAddr", "email", + "hg/hgUserSuggestion/hgUserSuggestion.c:35", public=True, + verified=True, family="suggest"), + h("suggest.mailFromAddr", "email", + "hg/hgUserSuggestion/hgUserSuggestion.c:36", public=True, + verified=True, family="suggest"), + h("suggest.filterKeyword", "internal", + "hg/hgUserSuggestion/hgUserSuggestion.c:37", public=True, + verified=True, family="suggest"), + h("suggest.mailSignature", "internal", + "hg/hgUserSuggestion/hgUserSuggestion.c:38", public=True, + verified=True, family="suggest"), + h("suggest.mailReturnAddr", "email", + "hg/hgUserSuggestion/hgUserSuggestion.c:39", public=True, + verified=True, family="suggest"), + h("suggest.browserName", "internal", + "hg/hgUserSuggestion/hgUserSuggestion.c:40", public=True, + verified=True, family="suggest"), + h("suggest.siteKey", "credential", + "hg/hgUserSuggestion/hgUserSuggestion.c:191", verified=True, + family="suggest"), + h("suggest.secretKey", "credential", + "hg/hgUserSuggestion/hgUserSuggestion.c:539", verified=True, + family="suggest"), + h("suggest.humanThreshold", "limit", + "hg/hgUserSuggestion/hgUserSuggestion.c:543", default='"-0.1"', + verified=True, family="suggest"), + h("hgGateway.dbDbTaxonomy", "internal", "hg/hgGateway/hgGateway.c:407", + default="defaultDbDbTree", verified=True), + h("hgEncodeVocabDocBaseUrl", "url", + "hg/encode/hgEncodeVocab/hgEncodeVocab.c:67", default='""', + verified=True, deprecated=True), + h("namedSessionAlt.", "internal", "hg/lib/cart.c:666", verified=True, + note="A prefix family enumerated at runtime: alternative " + "namedSessionDb locations to search when resolving a shared " + "session."), + h("encpipeline_prod", "internal", "hg/hgTracks/hgTracks.c:9867", + verified=True, deprecated=True, + note="Read via cfgValsWithPrefix from the ENCODE pipeline era."), + ], +} + + +# --------------------------------------------------------------------------- +# retired: settings whose feature is gone or was never part of the browser +# --------------------------------------------------------------------------- + +RETIRED = { + "what": "Settings for features that are gone, or for CGIs that are not " + "part of the browser release. Kept in the catalog because the " + "reads are still in the tree and a mirror's hg.conf may still " + "carry them. Deleting the code is a separate job from " + "sunsetting a release gate, and these are candidates for it.", + "vars": [ + h("paypalServer", "url", "hg/gsid/gsidMember/gsidMember.c:237", + deprecated=True, verified=True, family="gsid", + note="The GSID member site took payments. That CGI is not built " + "for the browser."), + h("paypalIpnServer", "url", "hg/gsid/gsidMember/gsidMember.c:257", + deprecated=True, verified=True, family="gsid"), + h("paypalCommercialFee", "internal", + "hg/gsid/gsidMember/gsidMember.c:739", deprecated=True, + verified=True, family="gsid"), + h("paypalAcademicFee", "internal", + "hg/gsid/gsidMember/gsidMember.c:741", deprecated=True, + verified=True, family="gsid"), + h("paypalEmail", "email", "hg/gsid/gsidMember/gsidMember.c:771", + deprecated=True, verified=True, family="gsid"), + h("paypalCert", "credential", "hg/gsid/gsidMember/gsidMember.c:816", + deprecated=True, verified=True, family="gsid"), + h("gsidCertId", "credential", "hg/gsid/gsidMember/gsidMember.c:808", + deprecated=True, verified=True, family="gsid"), + h("gisaid.structDir", "path", "hg/hgc/virusClick.c:127", + deprecated=True, verified=True, family="gisaid"), + h("gisaid.structUrl", "url", "hg/hgc/virusClick.c:140", + deprecated=True, verified=True, family="gisaid"), + h("genomeSpace.{variable}", "internal", + "hg/hgTables/genomeSpace.c:105", deprecated=True, verified=True, + note="GenomeSpace is shut down. Read with cfgOption2 and a runtime " + "suffix, so the harvester reports it half-resolved."), + h("rtdb.server", "profile", "hg/rtdbWebUpdate/rtdbWebUpdate.c:87", + deprecated=True, verified=True, family="rtdb"), + h("rtdb.port", "profile", "hg/rtdbWebUpdate/rtdbWebUpdate.c:88", + deprecated=True, verified=True, family="rtdb"), + h("rtdb.databases", "profile", "hg/rtdbWebUpdate/rtdbWebUpdate.c:89", + deprecated=True, verified=True, family="rtdb"), + h("pq.host", "profile", "hg/qaPushQ/qaPushQ.c:3832", verified=True, + family="pq", note="qaPushQ, a QA tool rather than a browser CGI."), + h("pq.user", "profile", "hg/qaPushQ/qaPushQ.c:3833", verified=True, + family="pq"), + h("pq.password", "credential", "hg/qaPushQ/qaPushQ.c:3834", + verified=True, family="pq"), + h("pq.db", "profile", "hg/qaPushQ/qaPushQ.c:3831", verified=True, + family="pq"), + h("pq.crossHost", "internal", "hg/qaPushQ/qaPushQ.c:3619", + verified=True, family="pq"), + h("rrcentral.host", "profile", "hg/qaPushQ/qaPushQ.c:3322", + verified=True, family="rrcentral"), + h("rrcentral.user", "profile", "hg/qaPushQ/qaPushQ.c:3323", + verified=True, family="rrcentral"), + h("rrcentral.password", "credential", "hg/qaPushQ/qaPushQ.c:3324", + verified=True, family="rrcentral"), + h("rrcentral.db", "profile", "hg/qaPushQ/qaPushQ.c:3325", + verified=True, family="rrcentral"), + h("encodeDataWarehouse.dataRoot", "path", + "hg/encode3/encodeDataWarehouse/edwWebXSendFile/edwWebXSendFile.c:29", + deprecated=True, verified=True, family="encodeDataWarehouse"), + h("encodeDataWarehouse.key", "credential", + "hg/encode3/encodeDataWarehouse/edwWebXSendFile/edwWebXSendFile.c:35", + deprecated=True, verified=True, family="encodeDataWarehouse"), + ], +} + + +# --------------------------------------------------------------------------- +# names the scan could not resolve +# --------------------------------------------------------------------------- + +RUNTIME_NAMES = { + "what": "Reads whose setting name is built at run time, so there is no " + "fixed name to document. Each is a small family rather than a " + "single setting, and each is a place where a typo in hg.conf is " + "silently ignored.", + "vars": [ + h("{themeKey}", "internal", "hg/lib/cart.c:3006", verified=True, + note="browser.theme.<name>, resolved from the theme the user " + "picked."), + h("{cfgName}", "internal", "hg/hgTracks/hgTracks.c:8894", + verified=True), + h("{confName}", "internal", "hg/lib/hdb.c:5447", verified=True), + h("{confVariable}", "internal", "hg/hgTracks/quickLift.c:26", + verified=True, ticket="37788", + note="quickLift colour settings, named per use."), + h("{overlapKey}", "internal", "hg/hgc/myVariantsClick.c:579", + verified=True, note="Goes with the doMyVariants gate."), + h("{temp}", "internal", "hg/hgcentralTidy/hgcentralTidy.c:80", + verified=True), + h("{cdwSetting}", "internal", + "hg/cirm/cdw/cdwWebBrowse/cdwWebBrowse.c:341", verified=True, + deprecated=True), + ], +} + + +# --------------------------------------------------------------------------- +# assembly +# --------------------------------------------------------------------------- + +SECTIONS = [ + ("Release gates", RELEASE_GATES), + ("Mirror knobs", MIRROR_KNOBS), + ("Database connections", DATABASE), + ("hgcentral tables", CENTRAL_TABLES), + ("Paths and caches", PATHS), + ("Limits and load control", LIMITS), + ("Abuse control", ABUSE), + ("Logging and diagnostics", LOGGING), + ("Branding and site text", BRANDING), + ("Geographic mirroring", GEO), + ("Login and wiki", LOGIN), + ("External services", EXTERNAL), + ("Retired", RETIRED), + ("Runtime-built names", RUNTIME_NAMES), +] + + +def build(): + """The catalog as one structure.""" + return {"boundary": BOUNDARY, + "accessors": ACCESSORS, + "profileSuffixes": PROFILE_SUFFIXES, + "policy": {"keepAfterFlip": KEEP_AFTER_FLIP, + "qaGrace": QA_GRACE}, + "sections": [{"title": t, "what": s["what"], "vars": s["vars"]} + for t, s in SECTIONS]} + + +def all_vars(cat): + for sec in cat["sections"]: + for v in sec["vars"]: + yield v + + +def by_name(cat): + out = {} + for v in all_vars(cat): + out.setdefault(v["name"], v) + return out + + +def gates(cat): + return [v for v in all_vars(cat) if v.get("role") == "gate"] + + +def knobs(cat): + return [v for v in all_vars(cat) if v.get("role") == "knob"] + + +def counts(cat): + vs = list(all_vars(cat)) + return { + "sections": len(cat["sections"]), + "vars": len(vs), + "distinctNames": len({v["name"] for v in vs}), + "gates": len(gates(cat)), + "knobs": len(knobs(cat)), + "public": len([v for v in vs if v["public"]]), + "deprecated": len([v for v in vs if v.get("deprecated")]), + "verified": len([v for v in vs if v["verified"]]), + "unverified": len([v for v in vs if not v["verified"]]), + "required": len([v for v in vs if v.get("required")]), + "envOverridable": len([v for v in vs if v.get("env")]), + } + + +# --------------------------------------------------------------------------- +# the harvester, for --reconcile and --sunset +# --------------------------------------------------------------------------- + +def load_harvester(): + """Import harvestHgConf from next door.""" + sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + try: + import harvestHgConf + except ImportError: + return None + return harvestHgConf + + +# --------------------------------------------------------------------------- +# sunset report +# --------------------------------------------------------------------------- + +def gate_lifecycle(cat, ages): + """Join each gate with the version history. + + Returns a record per gate carrying what the tree knows (added, flipped, + current default) and what the catalog decided (sunset). Everything the + report says follows from this join, so a gate cannot be described as + healthy just because nobody updated its entry. + """ + first = ages.get("first", {}) + first_true = ages.get("firstTrue", {}) + cur = ages.get("current") + out = [] + for v in gates(cat): + name = v["name"] + added = (first.get(name) or {}).get("version") + flipped = (first_true.get(name) or {}).get("version") + shipped = v.get("default") == "TRUE" + # A flip date with a FALSE default now means the flip was reverted, so + # the flag is back to gating and the flip date must not drive a + # deadline. + reverted = bool(flipped) and not shipped + sunset = v.get("sunset") + if sunset is None and shipped and flipped: + sunset = flipped + KEEP_AFTER_FLIP + out.append({ + "name": name, "src": v["src"], "default": v.get("default"), + "note": v.get("note"), "added": added, "flipped": flipped, + "shipped": shipped, "reverted": reverted, "sunset": sunset, + "current": cur, + "age": (cur - added) if (cur and added) else None, + }) + return out + + +def sunset_report(cat, ages, sites=None, out=sys.stdout): + """What should be deleted, what has no deadline, what is stuck in QA.""" + cur = ages.get("current") + life = gate_lifecycle(cat, ages) + print("current tree version: v%s" % cur, file=out) + print("policy: keep a flag %d releases after its default flips TRUE; " + "a gate\nstill defaulting FALSE after %d releases is stalled.\n" + % (KEEP_AFTER_FLIP, QA_GRACE), file=out) + + def line(g): + bits = [] + if g["added"]: + bits.append("added v%d" % g["added"]) + if g["flipped"] and not g["reverted"]: + bits.append("flipped v%d" % g["flipped"]) + if g["reverted"]: + bits.append("flip v%d reverted" % g["flipped"]) + if g["sunset"]: + bits.append("sunset v%d" % g["sunset"]) + n = len((sites or {}).get(g["name"], [])) or None + tail = "%d call site%s" % (n, "" if n == 1 else "s") if n else "" + return " %-26s %-46s %s" % (g["name"], ", ".join(bits), tail) + + overdue = sorted([g for g in life if g["sunset"] and cur + and g["sunset"] <= cur], key=lambda g: g["sunset"]) + print("OVERDUE (delete the flag and every branch that reads it): %d" + % len(overdue), file=out) + for g in overdue: + print(line(g), file=out) + + due = sorted([g for g in life if g["sunset"] and cur + and g["sunset"] > cur], key=lambda g: g["sunset"]) + print("\nSCHEDULED (shipped, deadline not yet reached): %d" % len(due), + file=out) + for g in due: + print(line(g), file=out) + + stalled = sorted([g for g in life + if not g["shipped"] and g["age"] is not None + and g["age"] > QA_GRACE], key=lambda g: -g["age"]) + print("\nSTALLED IN QA (default still FALSE %d+ releases after it landed): " + "%d" % (QA_GRACE, len(stalled)), file=out) + print(" Either turn these on or delete the feature. A flag nobody " + "flips is not\n gating a release, it is hiding dead code.", + file=out) + for g in stalled: + print(line(g) + " %d releases old" % g["age"], file=out) + + fresh = [g for g in life if not g["shipped"] + and (g["age"] is None or g["age"] <= QA_GRACE)] + print("\nIN QA (recent, inside the grace period): %d" % len(fresh), + file=out) + for g in sorted(fresh, key=lambda g: -(g["age"] or 0)): + print(line(g) + (" %d releases old" % g["age"] if g["age"] is not None + else " age unknown"), file=out) + + nodate = [g for g in life if g["sunset"] is None and g["shipped"]] + if nodate: + print("\nNO DEADLINE (shipped but the flip version is unknown, so no " + "deadline\ncould be computed; needs a sunset= in the catalog): " + "%d" % len(nodate), file=out) + for g in nodate: + print(line(g), file=out) + return len(overdue) + + +# --------------------------------------------------------------------------- +# checks +# --------------------------------------------------------------------------- + +def check(cat, out=sys.stderr): + """Internal consistency. Returns the number of problems found.""" + problems = 0 + c = counts(cat) + print("=== counts ===", file=out) + for k in sorted(c): + print("%-16s %s" % (k, c[k]), file=out) + + print("\n=== consistency ===", file=out) + seen = {} + for sec in cat["sections"]: + for v in sec["vars"]: + seen.setdefault(v["name"], []).append(sec["title"]) + dupes = {n: s for n, s in seen.items() if len(s) > 1} + if dupes: + print("names in more than one section (%d):" % len(dupes), file=out) + for n, s in sorted(dupes.items()): + print(" %-34s %s" % (n, ", ".join(s)), file=out) + + for v in all_vars(cat): + if v.get("role") == "gate" and v.get("kind") != "flag": + print("gate not of kind flag: %s" % v["name"], file=out) + problems += 1 + if v.get("sunset") and not v.get("role") == "gate": + print("sunset on a non-gate: %s" % v["name"], file=out) + problems += 1 + + unver = [v["name"] for v in all_vars(cat) if not v["verified"]] + if unver: + print("\nunverified rows (%d): classification not confirmed against " + "the code" % len(unver), file=out) + for n in sorted(unver): + print(" %s" % n, file=out) + + arguable = [v for v in all_vars(cat) if v.get("debatable")] + if arguable: + print("\n=== gate or knob: the calls worth arguing about (%d) ===" + % len(arguable), file=out) + print("Everything else in the split is either obvious or was confirmed " + "at its\ncall site. These are the ones a second opinion should " + "settle, because\nfiling a gate as a knob hides it from the " + "sunset report forever.\n", file=out) + for v in sorted(arguable, key=lambda x: x["name"].lower()): + print(" %s (currently: %s, default %s)" + % (v["name"], v.get("role"), v.get("default")), file=out) + for line in wrap_text(v["debatable"], 72): + print(" %s" % line, file=out) + print(file=out) + + print("problems: %d" % problems, file=out) + return problems + + +def wrap_text(s, width): + """Wrap without pulling in textwrap for one caller.""" + words = s.split() + lines, cur = [], "" + for w in words: + if cur and len(cur) + 1 + len(w) > width: + lines.append(cur) + cur = w + else: + cur = (cur + " " + w).strip() + if cur: + lines.append(cur) + return lines + + +def reconcile(cat, out=sys.stdout): + """Diff the catalog against what the tree actually reads. + + Three questions: + 1. does the catalog list something the tree no longer reads + 2. does the tree read something the catalog has not classified + 3. is every boolean flag in the tree classified gate or knob + The third is the one that keeps the sunset report honest: an unclassified + flag is one nobody has decided the fate of. + """ + hh = load_harvester() + if hh is None: + print("harvestHgConf.py not importable; cannot reconcile", file=out) + return 1 + + found, _ = hh.harvest() + tree = hh.by_name(found) + cataloged = by_name(cat) + problems = 0 + + # Three kinds of name legitimately have no literal read to point at, and + # all three have to be excused or the report is nothing but false alarms: + # profile members (read through cfgOption2 with a runtime prefix), prefix + # families (enumerated with cfgNamesWithPrefix), and the members of such a + # family as spelled out in the example configs. + suffixes = set(PROFILE_SUFFIXES["suffixes"]) + prefix_scans = set(found["prefixScans"]) + + def is_profile_member(name): + return "." in name and name.rsplit(".", 1)[1] in suffixes + + def is_prefix_family(name): + if name in prefix_scans or name.endswith("."): + return True + return any(name.startswith(p) for p in prefix_scans) + + print("=== catalog vs tree ===", file=out) + + missing = sorted(n for n in tree + if n not in cataloged and not n.startswith("{")) + # {ident} names are carried in the catalog with the braces, so match those + # separately. + missing += sorted(n for n in tree + if n.startswith("{") and n not in cataloged) + if missing: + problems += len(missing) + print("\nread by the tree, not in the catalog (%d):" % len(missing), + file=out) + for n in missing: + print(" %-40s %s" % (n, sorted(tree[n]["sites"])[0]), file=out) + + stale = sorted(n for n in cataloged + if n not in tree and not is_profile_member(n) + and not is_prefix_family(n)) + if stale: + print("\nin the catalog, no literal read found (%d):" % len(stale), + file=out) + print(" (a prefix family or a profile member is expected here; " + "anything else\n is a catalog row whose read has gone away)", + file=out) + for n in stale: + print(" %-40s %s" % (n, cataloged[n]["src"]), file=out) + + print("\n=== boolean flags: every one must be a gate or a knob ===", + file=out) + tree_flags = {n for n, d in tree.items() if d["boolean"]} + classified = {v["name"] for v in gates(cat)} | {v["name"] for v in knobs(cat)} + unclassified = sorted(tree_flags - classified) + if unclassified: + problems += len(unclassified) + print("unclassified (%d): nobody has decided whether these are " + "temporary" % len(unclassified), file=out) + for n in unclassified: + print(" %-40s %s" % (n, sorted(tree[n]["sites"])[0]), file=out) + else: + print("all %d boolean flags in the tree are classified" % + len(tree_flags), file=out) + + phantom = sorted(classified - tree_flags) + if phantom: + print("\nclassified as a flag but not read with " + "cfgOptionBooleanDefault (%d):" % len(phantom), file=out) + for n in phantom: + print(" %-40s %s" % (n, cataloged[n]["src"]), file=out) + + print("\n=== product/ex.hg.conf ===", file=out) + docs = hh.parse_doc_files() + pub = {v["name"] for v in all_vars(cat) if v["public"]} + # A prefix family counts as documented if any member of it is. + def documented(name): + if name in docs: + return True + return name.endswith(".") and any(d.startswith(name) for d in docs) + undocumented = sorted(n for n in pub if not documented(n)) + if undocumented: + print("marked public in the catalog, absent from the example configs " + "(%d):" % len(undocumented), file=out) + print(" (these are settings a mirror operator would want and has no " + "way to\n discover)", file=out) + for n in undocumented: + print(" %-40s %s" % (n, cataloged[n]["src"]), file=out) + only_docs = sorted(n for n in docs + if n not in cataloged and not is_profile_member(n) + and not is_prefix_family(n)) + if only_docs: + print("\nin the example configs, nothing in the tree reads them (%d):" + % len(only_docs), file=out) + print(" (either the feature was deleted and the documentation was " + "not, or the\n documented spelling is wrong, in which case a " + "mirror that sets it is\n silently ignored)", file=out) + for n in only_docs: + print(" %-40s %s" % (n, docs[n]["sites"][0]), file=out) + + print("\nproblems: %d" % problems, file=out) + return problems + + +# --------------------------------------------------------------------------- +# HTML +# --------------------------------------------------------------------------- + +CSS = """ +body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, + Arial, sans-serif; margin: 0 auto; max-width: 1180px; padding: 1em 2em; + color: #222; line-height: 1.45; } +h1 { font-size: 1.6em; margin-bottom: 0.2em; } +h2 { font-size: 1.2em; margin-top: 1.8em; border-bottom: 2px solid #4b6c9e; + padding-bottom: 0.2em; color: #1a3a6b; } +h3 { font-size: 1.05em; margin-top: 1.4em; color: #1a3a6b; } +p.what { color: #444; margin: 0.4em 0 1em 0; } +table { border-collapse: collapse; width: 100%; font-size: 0.87em; + margin-bottom: 1.2em; } +th { background: #eaf0f8; text-align: left; padding: 5px 7px; + border-bottom: 2px solid #4b6c9e; font-weight: 600; } +td { padding: 5px 7px; border-bottom: 1px solid #dde3ec; + vertical-align: top; } +tr:hover td { background: #f6f9fd; } +code { font-family: Menlo, Consolas, monospace; font-size: 0.94em; } +td.name code { font-weight: 600; } +td.src { color: #666; font-size: 0.9em; white-space: nowrap; } +span.kind { display: inline-block; padding: 1px 6px; border-radius: 3px; + font-size: 0.82em; background: #e8e8e8; color: #444; } +span.gate { background: #fde9c8; color: #7a4a00; } +span.knob { background: #d9ead9; color: #24541f; } +span.overdue { background: #f8d3d3; color: #8a1f1f; font-weight: 600; } +span.stalled { background: #fdf0c0; color: #6b5300; font-weight: 600; } +span.dep { background: #eee; color: #777; } +span.req { background: #dce6f8; color: #1a3a6b; } +div.note { color: #555; margin-top: 3px; font-size: 0.95em; } +div.arguable { color: #6b4a00; background: #fdf6e3; border-left: 3px solid #d9a441; + padding: 3px 7px; margin-top: 4px; font-size: 0.93em; } +span.arguable { background: #fdf0c0; color: #6b5300; } +div.box { background: #f6f8fb; border-left: 4px solid #4b6c9e; + padding: 0.7em 1em; margin: 1em 0; } +div.policy { background: #fff8ec; border-left: 4px solid #d9a441; + padding: 0.7em 1em; margin: 1em 0; } +ul.toc { columns: 3; list-style: none; padding-left: 0; font-size: 0.92em; } +""" + + +def esc(s): + return html.escape(str(s), quote=False) + + +def var_rows(vs, life_by_name=None): + rows = [] + for v in sorted(vs, key=lambda x: x["name"].lower()): + tags = ['<span class="kind">%s</span>' % esc(v["kind"])] + role = v.get("role") + if role: + tags.append('<span class="kind %s">%s</span>' % (role, role)) + if v.get("required"): + tags.append('<span class="kind req">required</span>') + if v.get("deprecated"): + tags.append('<span class="kind dep">retired</span>') + life = (life_by_name or {}).get(v["name"]) + if life: + cur = life.get("current") + if life.get("sunset") and cur and life["sunset"] <= cur: + tags.append('<span class="kind overdue">overdue v%d</span>' + % life["sunset"]) + elif life.get("sunset"): + tags.append('<span class="kind">sunset v%d</span>' + % life["sunset"]) + if (not life.get("shipped") and life.get("age") is not None + and life["age"] > QA_GRACE): + tags.append('<span class="kind stalled">stalled %d</span>' + % life["age"]) + extra = "" + if life and life.get("added"): + extra = "added v%d" % life["added"] + if life.get("flipped"): + extra += ", flipped v%d" % life["flipped"] + note = "" + if v.get("note"): + note = '<div class="note">%s</div>' % esc(v["note"]) + if v.get("debatable"): + tags.append('<span class="kind arguable">gate or knob?</span>') + note += ('<div class="arguable"><b>Arguable:</b> %s</div>' + % esc(v["debatable"])) + env = "" + if v.get("env"): + env = '<div class="note">environment: <code>%s</code></div>' \ + % esc(v["env"]) + default = esc(v.get("default") or "") + rows.append( + "<tr><td class='name'><code>%s</code>%s%s</td>" + "<td>%s</td><td><code>%s</code></td>" + "<td class='src'><code>%s</code>%s</td></tr>" + % (esc(v["name"]), note, env, " ".join(tags), default, + esc(v["src"]), + ("<div class='note'>%s</div>" % extra) if extra else "")) + return "\n".join(rows) + + +def table_of(vs, life_by_name=None): + return ("<table><tr><th>setting</th><th>kind</th><th>default</th>" + "<th>read at</th></tr>\n%s\n</table>" + % var_rows(vs, life_by_name)) + + +def render_html(cat, ages=None, sites=None): + life_by_name = {} + sunset_html = "" + if ages: + life = gate_lifecycle(cat, ages) + life_by_name = {g["name"]: g for g in life} + cur = ages.get("current") + overdue = [g for g in life if g["sunset"] and cur + and g["sunset"] <= cur] + stalled = [g for g in life if not g["shipped"] + and g["age"] is not None and g["age"] > QA_GRACE] + sunset_html = ( + '<div class="policy"><b>Sunset status at v%s.</b> ' + '%d shipped gates are past their removal deadline and %d have been ' + 'sitting at a FALSE default for more than %d releases. ' + 'Policy: keep a flag %d releases after its default flips TRUE, ' + 'then delete it and every branch that reads it. ' + '<code>hgConfCatalog.py --sunset</code> prints the working list.' + '</div>' % (cur, len(overdue), len(stalled), QA_GRACE, + KEEP_AFTER_FLIP)) + + c = counts(cat) + parts = ["<h1>Genome Browser hg.conf variables</h1>", + "<p>%d settings the CGIs read from <code>hg.conf</code>, " + "generated from <code>hg/utils/hgConfCatalog/</code>. " + "%d are release gates and %d are permanent deployment knobs." + "</p>" % (c["distinctNames"], c["gates"], c["knobs"]), + '<div class="box">%s</div>' % esc(cat["boundary"]), + sunset_html] + + parts.append("<h2>Contents</h2><ul class='toc'>") + for sec in cat["sections"]: + parts.append("<li><a href='#%s'>%s</a></li>" + % (esc(sec["title"].replace(" ", "-")), esc(sec["title"]))) + parts.append("</ul>") + + parts.append("<h2>How a setting is read</h2>") + parts.append("<table><tr><th>accessor</th><th>behaviour</th></tr>") + for fn, what in sorted(ACCESSORS.items()): + parts.append("<tr><td><code>%s</code></td><td>%s</td></tr>" + % (esc(fn), esc(what))) + parts.append("</table>") + + ps = cat["profileSuffixes"] + parts.append("<h2>Database profile suffixes</h2>") + parts.append("<p class='what'>%s</p>" % esc(ps["what"])) + parts.append("<p>Suffixes: %s</p>" + % ", ".join("<code>%s</code>" % esc(s) for s in ps["suffixes"])) + parts.append("<p>Profiles in use: %s</p>" + % ", ".join("<code>%s.</code>" % esc(s) + for s in ps["knownProfiles"])) + + for sec in cat["sections"]: + parts.append("<h2 id='%s'>%s</h2>" + % (esc(sec["title"].replace(" ", "-")), esc(sec["title"]))) + parts.append("<p class='what'>%s</p>" % esc(sec["what"])) + parts.append(table_of(sec["vars"], life_by_name)) + + return ("<!DOCTYPE html>\n<html><head><meta charset='utf-8'>" + "<title>hg.conf variables</title><style>%s</style></head>" + "<body>\n%s\n</body></html>\n" % (CSS, "\n".join(parts))) + + +# --------------------------------------------------------------------------- +# main +# --------------------------------------------------------------------------- + +def main(): + ap = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--json") + ap.add_argument("--html") + ap.add_argument("--check", action="store_true") + ap.add_argument("--reconcile", action="store_true") + ap.add_argument("--sunset", action="store_true") + args = ap.parse_args() + + cat = build() + + ages = None + sites = None + if args.sunset or args.html: + hh = load_harvester() + if hh is None: + print("harvestHgConf.py not importable", file=sys.stderr) + return 1 + ages = hh.harvest_ages() + found, _ = hh.harvest() + sites = {n: d["sites"] for n, d in hh.by_name(found).items()} + + rc = 0 + if args.check: + rc |= 1 if check(cat) else 0 + if args.reconcile: + rc |= 1 if reconcile(cat) else 0 + if args.sunset: + sunset_report(cat, ages, sites) + if args.json: + with open(args.json, "w") as f: + json.dump(cat, f, indent=1) + print("wrote %s" % args.json) + if args.html: + with open(args.html, "w") as f: + f.write(render_html(cat, ages, sites)) + print("wrote %s" % args.html) + + if not any([args.check, args.reconcile, args.sunset, args.json, args.html]): + for k, v in sorted(counts(cat).items()): + print("%-16s %s" % (k, v)) + return rc + + +if __name__ == "__main__": + sys.exit(main())