97557b2dab79bd19b0345f1ea49a233c191300ae braney Sat Sep 12 06:59:07 2026 -0700 hgConfCatalog: cite skipMalformedCgiPairs where the tree actually reads it, refs #37925 The row cited lib/cheapcgi.c, which is where the flag takes effect but not where it is read: the kent libraries cannot read hg.conf at all. The read is hg/lib/hgConfig.c:227, which hands the setting to cheapcgi through cgiSkipMalformedPairs. The nightly reconcile had been reporting the row as citing a file the read has left. diff --git src/hg/utils/hgConfCatalog/hgConfCatalog.py src/hg/utils/hgConfCatalog/hgConfCatalog.py index 4d2713464ec..700bb694013 100755 --- src/hg/utils/hgConfCatalog/hgConfCatalog.py +++ src/hg/utils/hgConfCatalog/hgConfCatalog.py @@ -1,3181 +1,3181 @@ #!/usr/bin/env python3 """hgConfCatalog.py - the registry of hg.conf variables. Third of the browser configuration inventories, after cartTrackVarCatalog.py (#37838, track-scoped cart variables) and urlCommandCatalog.py (#37923, CGI URL parameters). This one covers hg.conf: the per-machine configuration file every CGI reads at startup through the cfgOption* accessors in hg/lib/hgConfig.c. What makes hg.conf different from the other two, and the reason this file needed a mechanism the others do not have: A cart variable belongs to a user and a URL parameter belongs to a request. An hg.conf variable belongs to a machine, and a machine we do not control. Mirrors, the GBiB and the GBiC all carry their own hg.conf, so a variable is reachable long after the tree stops caring about it, and deleting one is a compatibility decision rather than a cleanup. On top of that, the tree deliberately manufactures short-lived hg.conf variables as part of the release process. A user-visible feature is expected to ship dark behind cfgOptionBooleanDefault(name, FALSE) so it can sit on master through QA without blocking the release train, then have its default flipped to TRUE once it is released, then have the flag deleted. That last step is the one nobody does. Nothing in the tree records that a flag was meant to be temporary, so shipped gates accumulate: showTutorial has been defaulting TRUE since v466, greyBarIcons since v492, and both are still branch points in hgTracks today. So the registry separates two populations that look identical in the source: gate introduced to gate a release. Temporary by intent. Has a lifecycle (added, flipped, sunset) and is expected to be deleted. knob a genuine deployment switch a mirror is entitled to set forever. isGbib, browser.dumpStack and hgta.disableAllTables are knobs. These are exempt from sunsetting and saying so explicitly is what keeps the report from crying wolf. The distinction cannot be made mechanically, which is why it is curated here rather than harvested. Everything else about a gate's lifecycle is mechanical: harvestHgConf.py --age dates each variable's first read, and each boolean flag's first TRUE default, out of the git history, mapped onto the CGI_VERSION in effect at the time. So `added` and `flipped` come from the tree and only `sunset` is a judgement call. A gate with no explicit sunset gets flipped + KEEP_AFTER_FLIP, which is the release after mirrors have had a cycle to object. Sunset policy, all in versions (releases are about three weeks apart): KEEP_AFTER_FLIP = 4 once the default is TRUE the feature is public. Keep the flag four more releases so a mirror or hgwbeta can switch it back off without a code change, then delete it along with every branch that reads it. QA_GRACE = 6 a gate still defaulting FALSE this long after it landed is not being gated, it is being forgotten. Either turn it on or delete the feature. Verification status. Rows carry verified=True only where the classification was confirmed by reading the code at the cited source file. --check counts what is left, because a variable described as a permanent knob when it is really a forgotten gate defeats the purpose of the exercise. Usage: hgConfCatalog.py --json out.json hgConfCatalog.py --html out.html hgConfCatalog.py --check # counts and internal consistency hgConfCatalog.py --reconcile # diff the catalog against the tree hgConfCatalog.py --reconcile --verbose # ... with the standing drift hgConfCatalog.py --sunset # what should be deleted, and when hgConfCatalog.py --sunset-new # ... only what changed since the backlog hgConfCatalog.py --redundant # conf lines a flipped default made pointless hgConfCatalog.py --redundant --verbose # ... and the files it read hgConfCatalog.py --update-baseline # accept the current backlog hgConfCatalog.py --refresh --sunset-new # ... rebuilding the age cache hgConfCatalog.py --cache /tmp/ages.json --refresh --sunset-new # ... without writing into the tree --reconcile is the mode meant for a nightly cron: it prints nothing and exits 0 when the tree holds no setting the catalog has not classified, and exits 1 with the list when somebody has added one. --sunset-new is the same shape for the release side, and is what belongs in the weekly build's wrap-up: it reports the gates that went overdue or stalled since the backlog file was last accepted, and the ones that were cleaned up, and says nothing about the standing backlog. --sunset itself reports every overdue gate on every run whether or not anything changed. That is the right thing for a person working the list down and the wrong thing for anything automated, which is why it is a separate mode. --redundant is the third leg of the same idea and the one that closes the loop. Flipping a default to TRUE does not turn the feature on anywhere: it was already on, on every machine somebody had appended the flag to by hand. What the flip does is make those lines pointless, and a pointless line is not harmless, since while it is there the flag reads on locally whatever the tree says. So this mode reads the hg.conf files it can see from hgwdev, joins them against the catalog, and names the lines to delete. Like --reconcile it is silent and exits 0 when there is nothing to do, which is what lets the nightly cron run it and mail only the news. """ import argparse import html import json import os import re import sys # Sunset policy, in releases. See the module docstring. KEEP_AFTER_FLIP = 4 QA_GRACE = 6 # The gates already known to be overdue or stalled, so --sunset-new can report # what changed rather than the whole standing list. Regenerate with # --update-baseline and commit it; the diff is then the release-to-release # history of the backlog, in the git log rather than in somebody's memory. BACKLOG_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)), "hgConfGateBacklog.txt") # Floor on how many settings a working scan finds; well under the real count. # See the check in reconcile(). MIN_TREE_NAMES = 150 REDMINE = "https://redmine.gi.ucsc.edu/issues/%d" # --------------------------------------------------------------------------- # helpers # --------------------------------------------------------------------------- def h(name, kind, src, default=None, note=None, public=False, verified=False, role=None, sunset=None, env=None, deprecated=False, family=None, required=False, ticket=None, debatable=None): """One catalog entry. name the hg.conf setting name kind what sort of setting: path, table, profile, credential, url, email, limit, flag, branding, debug, internal, dead src source file the tree reads it in, with no line number. A line is derived data: it is recomputed for display from the harvester, so an edit above a read is not a citation to fix default compiled-in default if the read supplies one role for boolean flags only: "gate" (a release gate, temporary by intent, subject to sunsetting) or "knob" (a permanent deployment switch, exempt). Every cfgOptionBooleanDefault flag in the tree must be one or the other; --reconcile enforces that sunset release by which a gate should be gone from the tree. Omit to take the default of flipped + KEEP_AFTER_FLIP; the age cache supplies the flip version public documented for mirror operators in product/ex.hg.conf, or belongs there verified True if the classification was confirmed by reading src env environment variable that overrides it, via cfgOptionEnv family groups members of one prefix family, for the docs required read with cfgVal, so the CGI dies if it is absent deprecated the feature it configures is gone or going ticket Redmine ticket that introduced or tracks it debatable why this row's gate-or-knob call could reasonably go the other way. Set it rather than picking a side quietly: the whole point of separating the two is that someone has decided, and a decision nobody argued with is not the same as a decision. --check lists these as the review agenda """ d = {"name": name, "kind": kind, "src": src, "public": public, "verified": verified} for key, val in (("default", default), ("note", note), ("role", role), ("sunset", sunset), ("env", env), ("family", family), ("ticket", ticket), ("debatable", debatable)): if val is not None: d[key] = val if required: d["required"] = True if deprecated: d["deprecated"] = True return d # --------------------------------------------------------------------------- # how hg.conf is read at all # --------------------------------------------------------------------------- ACCESSORS = { "cfgOption": "Returns the value or NULL. Absent means off.", "cfgOptionDefault": "Returns the value or a compiled-in default.", "cfgOptionBooleanDefault": "Boolean with a compiled-in default. Accepts yes/no, on/off, " "true/false. This is the accessor used to gate a release.", "cfgVal": "Returns the value or errAborts. The CGI will not run without it.", "cfgOptionEnv": "Environment variable first, then hg.conf. Note the argument order: " "the environment name comes first and is not an hg.conf setting.", "cfgOptionEnvDefault": "As cfgOptionEnv, with a compiled-in default.", "cfgOption2": "Reads prefix.suffix. The prefix is usually a runtime value, which is " "how one call site serves db.host, central.host and every other " "profile at once.", "cfgOptionDefault2": "As cfgOption2, with a compiled-in default.", } BOUNDARY = ( "hg.conf is read once per CGI invocation and is never written by the " "browser. It is not user state: nothing in a session, a saved session or " "a URL can change it, which is exactly why it is the right place to gate " "a feature during a release. A mirror's copy is outside our control, so " "removing a variable has to be treated as an interface change, not a " "cleanup. Precedence for a value is: the environment (only for the " "cfgOptionEnv settings, and only where the CGI allows it), then hg.conf, " "then the compiled-in default." ) PROFILE_SUFFIXES = { "what": "A database profile is a set of hg.conf settings sharing one prefix, " "read through cfgOption2(profileName, suffix) where profileName is a " "runtime value. So these suffixes are legal under any profile prefix, " "and none of the resulting names appears as a literal anywhere in the " "tree. This is why product/ex.hg.conf documents " "archivecentral.password while a search of the source finds nothing. " "Any profile also has a failover profile, named with the \"slow-\" " "prefix in front of the main profile's name (sqlProfileGetFailover at " "jksql.c:409), which is where slow-db comes from. excludeDbs is the " "one suffix that belongs only to a failover profile, and the one that " "is not read through cfgOption2: jksql.c:1325 builds the whole name " "with safef and reads it with cfgOption, which is why no harvest of " "the tree finds it even though ex.hg.conf:36 documents " "slow-db.excludeDbs and goldenPath/help/gbib.html explains it twice. " "Its value is a comma-separated list of databases that exist only on " "the local server, so the failover connection is never opened for " "them.", "src": "hg/lib/jksql.c", "suffixes": ["host", "port", "socket", "user", "password", "db", "verifyServerCert", "ca", "caPath", "cert", "key", "cipher", "crl", "crlPath", "excludeDbs"], "knownProfiles": ["db", "slow-db", "central", "cart", "customTracks", "archivecentral", "backupcentral", "myStuff", "myGenome", "rrcentral", "pq", "rtdb", "cdw"], } # --------------------------------------------------------------------------- # release gates: boolean flags that exist to hold a feature back # --------------------------------------------------------------------------- # These are the reason this catalog has a sunset mode. Each was added so a # user-visible change could sit on master without shipping. Ordered by age so # the backlog is visible at a glance. RELEASE_GATES = { "what": "Boolean flags introduced to ship a feature dark during a release. " "Temporary by intent: each should be deleted once the feature it " "guards is public and mirrors have had a cycle to object.", "vars": [ h("showTxCodonNumbers", "flag", "hg/hgTracks/cds.c", default="FALSE", role="gate", verified=True, ticket="38298", note="A second codon number in the gene-track codon mouseover, counted in the " "transcript's own coordinates, for a transcript that aligns with bases the " "assembly does not have. The number the browser has always shown is counted " "along the genome, so for those transcripts it disagrees with the number NCBI " "reports, by one codon per three missing bases. It affects 58 coding " "transcripts on hg38's primary chromosomes but 656 on canFam3, and gets worse " "the poorer the assembly. With the gate on, such codons also draw in the " "browser's existing query-insertion orange with a \"!\" after the codon " "number. Read once in txCodonNumbersEnabled(); with it off no alignment is " "looked up, no codon carries a transcript number, and the rendering and the " "mouseover are byte-identical to before. Off during QA; flip to TRUE once " "released."), - h("skipMalformedCgiPairs", "flag", "lib/cheapcgi.c", default="FALSE", + h("skipMalformedCgiPairs", "flag", "hg/lib/hgConfig.c", default="FALSE", role="gate", verified=True, ticket="38340", note="Step over a CGI or cookie pair that has no =value, instead of losing the " "pair after it or aborting the request. Covers parseCookies in cheapcgi, " "hgSession's session backup, and refreshNamedSessionCustomTracks. The query " "string parsers were fixed unconditionally under #38335 and do not read this " "flag. The kent libraries cannot read hg.conf, so hgConfig.c hands the " "setting to cheapcgi through cgiSkipMalformedPairs. Off during QA; flip to " "TRUE once released."), h("collectionHubCopyOnWrite", "flag", "hg/lib/cart.c", default="FALSE", role="gate", verified=True, ticket="38273", note="Copy a track collection's generated hub file when the program that writes " "it (only hgCollection) asks for a copy, instead of on every session load. " "With it off, loading any session that carries a collection copies the hub to " "a new trash file and registers that copy in hgcentral.hubStatus, which is " "where 81% of the RR's 3.1M hubStatus rows came from. Read in " "cartCollectionHubCopyOnWrite() and branched on at four places in three " "files: cart.c makes the copy when cartRequestLocalHubCopy() was called, and " "skips a hub the cart already owns in copyLocalHubs(); " "cartCopyLocalHubsOnSessionLoad() does nothing (that one is the old behavior, " "and it plus its five callers in cart.c and hgSession.c go away with the " "gate); sessionData.c's saveTrackFile splits a hub that came from another " "session. Off during QA; flip to TRUE once released."), h("bigNet", "flag", "hg/lib/trackHub.c", default="FALSE", role="gate", verified=True, ticket="20824", note="The bigNet track type, a net of pairwise alignments in a " "bigBed, and quickLift support for it. Read once in " "trackHubBigNetEnabled(); with it off a hub that declares " "\"type bigNet\" fails to load with an unsupported-type error, " "quickLift refuses the type, and neither hubCheck nor the hubApi " "lists it among the supported types."), h("showMouseovers", "flag", "hg/hgTracks/config.c", default="FALSE", role="gate", verified=True, note="Mouseover text on track items instead of the browser's own " "title tooltips. Added v446 and still defaulting FALSE, which " "makes it the oldest gate in the tree that never shipped. Four " "call sites across config.c and imageV2.c. Either the feature " "is wanted, in which case flip it, or it is not, in which case " "the flag and the code behind it should go."), h("storeUserFiles", "flag", "hg/hgHubConnect/hgHubConnect.c", default="FALSE", role="gate", verified=True, note="Hub space, the user file store behind hgHubConnect's upload " "wizard. Added v447 and briefly defaulted TRUE around v454 " "before going back to FALSE, so the history shows a flip that " "was reverted. Four call sites."), h("hgSession.shortLink", "flag", "hg/hgSession/hgSession.c", default="FALSE", role="gate", verified=True, note="Short session links. Added v374 and never flipped, which is " "the longest-running dark feature here."), h("showHubApiKey", "flag", "hg/hgHubConnect/hgHubConnect.c", default="FALSE", role="gate", verified=True, note="Expose the hub API key UI. Shares its call site with " "storeUserFiles, so the two should be retired together."), h("autoBlatBigPsl", "flag", "hg/hgBlat/hgBlat.c", default="FALSE", role="gate", verified=True, ticket="32751", note="Always create a custom track from BLAT results, so a result " "page can be reopened and shared. The read at hgBlat.c:2972 " "overrides the file-scope autoBigPsl, which is initialised " "FALSE at hgBlat.c:60 under the comment \"DEFAULT VALUE change " "to TRUE in future\", and eleven branches downstream test it. " "Filed as a knob until that was read, on the strength of the " "default being an identifier the harvester could not resolve; " "a flag whose own source says to flip it later is a gate."), h("blatShowLocus", "flag", "hg/hgBlat/hgBlat.c", default="FALSE", role="gate", verified=True, note="Show the genomic locus alongside BLAT results."), h("modernAlignPage", "flag", "hg/hgc/hgc.c", default="FALSE", role="gate", verified=True, ticket="37893", note="Use the modern single-page base-by-base alignment view (the one " "hgBlat's new results page uses) for ordinary alignment clicks too " "- mRNA/EST, PSL and similar hgc details - instead of the classic " "two-frame . Off during QA; flip to TRUE once released."), h("blatNewPageBanner", "flag", "hg/hgBlat/hgBlat.c", default="FALSE", role="gate", verified=True, note="The banner on the classic BLAT results page that offers a " "one-click switch to the new sortable table display. Guards " "the advertisement, not the feature: turning it on makes the " "browser recommend the new page without releasing new CGIs, " "and users who already opted in or follow a direct link get " "the new page either way. Goes away with the banner, once the " "new page is the default. Born TRUE at c683ecb63d7 and put " "back to FALSE at 81d7cff6cb6 while the new page is still " "being tested, so it is gating again rather than shipped. " "Kept a gate rather than a knob because what it " "turns off is a sentence recommending another page, and no " "mirror needs that switch forever; the deadline the report " "computes for it is really a deadline on deciding whether the " "new page becomes the default, which is the conversation the " "nag is supposed to force."), h("blatResultsGroup", "flag", "hg/hgTracks/hgTracks.c", default="FALSE", role="gate", verified=True, ticket="38086", note="Put BLAT result custom tracks in their own \"BLAT Results\" track " "group (with a \"Delete all\" button) instead of Custom Tracks, and " "give headerless queries a useful default name (query size + top-hit " "gene) rather than \"blat YourSeq\". Read in three CGIs that must " "agree: hgBlat.c:1493 (naming), hgc.c:27514 (tags the track " "group=blat), hgTracks.c:7262 (synthesizes the group). Off during " "QA; flip to TRUE once released."), h("genarkLiftOver", "flag", "hg/lib/genark.c", default="FALSE", role="gate", verified=True, note="Offer liftOver between GenArk assemblies. Four call sites in " "genark.c and hdb.c."), h("showIgv", "flag", "hg/hgTracks/hgTracks.c", default="FALSE", role="gate", verified=True, note="An IGV link in the track hamburger menus."), h("showLiftRequest", "flag", "hg/hgConvert/hgConvert.c", default="FALSE", role="gate", verified=True, ticket="37973", note="A link from the Convert page to liftRequest.html, the page " "that requests a new whole-genome alignment. The assembly " "list only offers targets that already have a chain from the " "source, so the Convert page is where a user finds out theirs " "is missing, but nothing in the tree linked to the request " "page. Off until the request pipeline is confirmed ready to " "take traffic from the browser UI."), h("groupDropdown", "flag", "hg/hgTracks/hgTracks.c", default="FALSE", role="gate", verified=True, note="Track group chooser as a dropdown rather than the current " "layout."), h("gcOnTheFlyCoExist", "flag", "hg/hgTracks/hgTracks.c", default="FALSE", role="gate", verified=True, note="Let the calculated GC track coexist with the stored one. A " "sub-flag of gcOnTheFly, so it should be deleted with it " "rather than outliving it."), h("showAliases", "flag", "hg/hgTracks/hgTracks.c", default="TRUE", role="gate", verified=True, note="Show chromosome alias names in the position box. On by " "default; the flag stays so a mirror can switch it back off."), h("showColorPicker", "flag", "hg/lib/hui.c", default="TRUE", role="gate", verified=True, note="The track colour picker in track UI. On by default; the " "flag stays so a mirror can switch it back off."), h("doMyVariants", "flag", "hg/hgCustom/hgCustom.c", default="FALSE", role="gate", verified=True, note="The My Variants track and its upload path. Thirteen call " "sites across seven files, the widest gate in the tree, which " "is a fair measure of what deleting a stale one costs."), h("hguidIpTracking.enabled", "flag", "hg/lib/botDelay.c", default="FALSE", role="gate", verified=True, note="Per-hguid IP tracking for abuse detection. Its three " "companion settings (maxIps, table, windowSeconds) are plain " "values and are listed under abuse control."), h("canColorItems", "flag", "hg/hgTracks/hgTracks.c", default="FALSE", role="gate", verified=True, note="Added in the current release, so it is doing exactly what a " "gate is supposed to do and has not earned a deadline yet."), h("cspResponseHeader", "flag", "hg/lib/hCommon.c", default="FALSE", role="gate", verified=True, ticket="38151", note="Send the Content Security Policy as an http response header, " "on top of the meta tag the library already writes. It exists " "for the pages that build their own header block and so never " "reach that library code, which is why cspWriteResponseHeader " "is called from fourteen places: most inherit it from " "addHttpHeaders in cart.c, and six call it directly (the two " "\"too many requests\" pages, the captcha and its error page, " "the hubApi help redirect, the hgSearch redirect). Added at " "e6ddf60465d after v502 and on no release branch yet, so the " "deadline has not started. On in hgwdev's cgi-bin hg.conf. " "The introducing commit carries no refs #38151 on purpose, " "under the security-incident policy, so the ticket here is " "hand-written and the sunset report's git-derived ticket " "column will show it blank.", debatable="A knob if the policy ends up being set once in the Apache " "config instead, which #38151 lists as its open question. " "Then this switch survives as the way a mirror avoids " "emitting a second, conflicting policy, since two CSP " "headers intersect and both are enforced, and a proxy " "setup is a property of the machine. Filed as a gate " "because the CGI emits the header today, and because " "calling a gate a knob hides it from the sunset report " "for good."), h("browser.quickLiftAlignments", "flag", "hg/lib/quickLift.c", default="FALSE", role="gate", verified=True, ticket="38249", note="Let quickLift lift alignment tracks: psl, bigPsl, chain, " "bigChain, maf, bigMaf and wigMaf. Read once, in " "quickLiftAlignmentsEnabled(), and asked once, by " "validateOneTdb in trackHub.c, which is the only door: the " "quickLiftUrl and quickLiftDb pair that every lift path keys " "off is written by the quickLift hub writer and by nothing " "else, so with the flag off an alignment track never enters a " "quickLift hub and the lifting, drawing and details code behind " "it is unreachable. A hub built while it was on keeps working, " "since its stanzas are already in the hub file in trash. On in " "confs/hgwdev.hg.conf. Named browser.quickLiftAlignments to " "sit beside browser.quickLift, the gate on the rest of the " "feature, but read with a literal cfgOptionBooleanDefault " "rather than cartOrCfgOption: the harvester only sees the " "cfgOption* accessors, which is why browser.quickLift is not in " "this catalog. A cart variable of the same name still " "overrides it, so both answers can be had on one machine."), # Gates whose default has flipped TRUE. These are the deletable ones: # the feature is public and the flag is now only an off switch. h("showTutorial", "flag", "hg/hgCustom/hgCustom.c", default="TRUE", role="gate", verified=True, note="The interactive tutorials. Public since v466, five call " "sites across four CGIs. Nothing is gating any more."), h("canDupTracks", "flag", "hg/lib/dupTrack.c", default="TRUE", role="gate", verified=True, note="Duplicate-track feature. Public since v443."), h("canSnake", "flag", "hg/hgc/hgc.c", default="TRUE", role="gate", verified=True, note="Snake display for chain and alignment tracks. Public since " "v467."), h("showDownloadUi", "flag", "hg/hgTracks/hgTracks.c", default="TRUE", role="gate", verified=True, note="The download-current-track UI. Public since v467."), h("mergeRecommended", "flag", "hg/hgTracks/recTrackSets.c", default="TRUE", role="gate", verified=True, note="Merge behaviour for recommended track sets. Public since " "v467."), h("svgBarChart", "flag", "hg/hgc/barChartClick.c", default="TRUE", role="gate", verified=True, note="SVG rather than raster bar charts on the details page. " "Public since v428, the longest-shipped gate still in place."), h("canDoHgcInPopUp", "flag", "hg/hgTracks/config.c", default="TRUE", role="gate", verified=True, note="Details pages in a popup instead of a page load. Public " "since v492. Three call sites."), h("greyBarIcons", "flag", "hg/hgTracks/hgTracks.c", default="TRUE", role="gate", verified=True, note="The grey side-bar icons on track images. Public since v492. " "Four call sites in hgTracks.c and imageV2.c."), h("trackHubsCanAddGroups", "flag", "hg/lib/hubConnect.c", default="TRUE", role="gate", verified=True, note="Let hubs declare their own track groups. Public since v492."), h("newBotDelay", "flag", "hg/lib/botDelay.c", default="TRUE", role="gate", verified=True, note="The reworked bot-delay logic. Public since v492."), h("gcOnTheFly", "flag", "hg/hgTracks/hgTracks.c", default="TRUE", role="gate", verified=True, note="Calculate the GC percent track at draw time instead of " "reading a stored table. Public since v496, so it is inside " "its grace period."), h("useBlatBigPsl", "flag", "hg/hgBlat/hgBlat.c", default="TRUE", role="gate", verified=True, note="bigPsl output from BLAT. Public since v348."), h("alwaysItemRgb", "flag", "hg/cgilib/bedCart.c", default="TRUE", role="gate", verified=True, note="Honour a BED's itemRgb without requiring the track setting. " "Defaulted TRUE from v466. Born TRUE, so it never gated a " "release; it is the switch back to how the browser coloured " "BED items before v466, and unlike sleepOn429 the off position " "describes an old version of the browser rather than a " "property of the machine, which is what keeps it a gate. The " "last test in bedItemRgb is the only thing it controls, so " "deleting it is a two-line change."), h("hgHubConnect.validateHub", "flag", "hg/hgHubConnect/hgHubConnect.c", default="TRUE", role="gate", verified=True, note="The Hub Development tab on hgHubConnect, which is where a hub " "author runs hubCheck from the browser. Two call sites, the " "tab itself and hgHubConnectDeveloperMode below it. Public " "since v427. Described here as \"run hubCheck when a hub is " "attached\" until the call site was read: it gates the tab, " "not attachment, so a hub is validated on attach either way."), h("blatNewForm", "flag", "hg/hgBlat/hgBlat.c", default="FALSE", role="gate", verified=True, ticket="37893", note="Replaces hgBlat's classic C-generated search form with the " "JavaScript-built one in hg/js/hgBlat.js, which shares its " "styling and page header with the new BLAT results page. Off " "while the new form is being tested. Read through " "cartUsualBoolean, so this is only the default: the form's " "banner offers a per-user opt-out that sets blatNewForm=0 in " "the cart. It comes out once the new form replaces the old " "one, which is what makes it a gate rather than a knob."), h("blatNewFormBanner", "flag", "hg/hgBlat/hgBlat.c", default="follows blatNewForm", role="gate", verified=True, ticket="37893", note="Whether the classic BLAT search form carries a banner offering " "the new one. Defaults to whatever blatNewForm is set to: " "where the new form is enabled, a user who took its go-back " "link needs a way to return, and where it is disabled there is " "nothing to advertise. Set explicitly to override either way. " "The default is that nested read rather than a literal, so the " "sunset report cannot tell from the tree whether this one has " "shipped; it ships and goes away with blatNewForm. Sibling of " "blatNewPageBanner, which does the same job for the results " "page."), h("sessionNewPage", "flag", "hg/hgSession/hgSession.c", default="FALSE", role="gate", ticket="38157", note="Replaces hgSession's classic C-generated 'My Sessions' page with " "the JavaScript-built one in hg/js/hgSession.js (a save bar plus a " "searchable, sortable session table with inline share/rename/" "overwrite/delete). Off while the new page is being tested. Read " "through cartUsualBoolean, so this is only the default: the page's " "banner offers a per-user opt-in/opt-out that sets sessionNewPage " "in the cart. Sibling of blatNewForm; comes out once the new page " "replaces the old one."), h("sessionNewPageBanner", "flag", "hg/hgSession/hgSession.c", default="follows sessionNewPage", role="gate", ticket="38157", note="Whether the classic Sessions page carries the banner advertising " "the new one. Defaults to whatever sessionNewPage is set to: where " "the new page is disabled there is nothing to advertise. Set " "explicitly to override either way. Does not gate the link the " "other way: the new page always offers its way back to the classic " "page, since the cart variable that got the user there sticks. " "Sibling of blatNewFormBanner."), h("quickLiftClipToChains", "flag", "hg/lib/quickLift.c", default="TRUE", role="gate", verified=True, ticket="38042", note="Whether an item too big for the chains quickLift loaded is " "pulled in to what they cover rather than dropped. Read once, " "in quickLiftIntervalsToBedClip, which is the display path; " "the details page calls quickLiftIntervalsToBed and always " "sees the item's true extent. Born TRUE, so it never gated a " "release: like alwaysItemRgb, the off position is the way the " "browser behaved before the change rather than anything about " "the machine, so it is a gate and should go once nobody has " "asked for the old behaviour."), ], } # --------------------------------------------------------------------------- # mirror knobs: boolean flags that are meant to live forever # --------------------------------------------------------------------------- MIRROR_KNOBS = { "what": "Settings that are legitimate, permanent deployment switches. " "Nearly all are boolean flags, and those are listed explicitly so " "the sunset report does not nag about them; a few are strings that " "choose between behaviours rather than turning one off, and those " "carry no gate/knob role because only booleans have one.", "vars": [ h("isGbib", "flag", "hg/lib/hdb.c", default="FALSE", role="knob", public=True, verified=True, note="This is the Genome Browser in a Box. Changes paths and " "disables features that make no sense on a VM."), h("isGbic", "flag", "hg/lib/hdb.c", default="FALSE", role="knob", public=True, verified=True, note="This is a Genome Browser in the Cloud install."), h("allowNib", "flag", "hg/lib/hdb.c", default="TRUE", role="knob", public=True, verified=True, note="Permit nib sequence files. In hDbDbNibPath: on, the sequence " "directory comes from dbDb.nibPath; off, it is /gbdb/ " "through hReplaceGbdbSeqDir. Ancient, but an old mirror may " "still hold nib assemblies, so it stays. See forceTwoBit, " "which decides the same question one level up."), h("forceTwoBit", "flag", "hg/lib/hdb.c", default="TRUE", role="knob", public=True, verified=True, note="Where hNibForChrom looks for a chromosome's sequence: on, " "always /gbdb//.2bit through hReplaceGbdb; off, fall " "back to chromInfo.fileName and then to a nib under " "dbDb.nibPath. Filed as a gate until this was settled " "against the code, on the strength of its TRUE default, but " "the off position describes where a machine keeps its " "sequence, not a browser feature waiting to ship, which is " "the same thing allowNib says. Deleting it " "would delete the chromInfo and nib fallbacks with it, and " "nobody has proposed that."), h("freeType", "flag", "hg/cgilib/trackLayout.c", default="TRUE", role="knob", public=True, verified=True, note="FreeType font rendering in track images rather than the built " "in bitmap fonts. Off selects the Helvetica bitmap path in " "trackLayoutInit. Filed as a gate until this was settled " "against the code, because it shipped at v412, but a mirror " "without the URW fonts installed has to be able to turn it " "off, and the bitmap path is also " "measurably faster to draw, so both positions have a " "constituency. Its companions freeTypeDir and freeTypeFont " "are plain values and stay."), h("trustTrackDb", "flag", "hg/lib/hdb.c", default="FALSE", role="knob", verified=True, note="Skip the per-track check that a trackDb row's table or file " "actually exists, in addTrackIfDataAccessible. Whether a " "machine's trackDb can be trusted is a property of that " "machine: on a mirror carrying only tracks it loaded itself " "the check is pure cost, and on hgwdev it catches real " "breakage. Filed as a gate until this was settled against " "the code, where its FALSE default put it in the stalled list " "for 43 releases; it was never a feature " "waiting to ship. The flip to TRUE at v458 and back is a " "record of somebody trying it on a machine, not of a release."), h("sleepOn429", "flag", "hg/lib/botDelay.c", default="TRUE", role="knob", verified=True, note="After emitting the 429 page, hold the process for ten seconds " "before exiting, which slows a robot that ignores the status " "code. Filed as a gate until this was settled against the " "code, but it was born TRUE at v481 and so never held a " "feature back; the off position is for " "a machine that would rather not tie up an Apache child, which " "is a deployment call."), h("browser.dumpStack", "flag", "hg/lib/hCommon.c", default="FALSE", role="knob", public=True, verified=True, note="Dump a stack trace to the error log on a crash. A debugging " "switch an operator turns on when needed."), h("showEarlyErrors", "flag", "hg/lib/hgConfig.c", default="FALSE", role="knob", public=True, verified=True, note="Show errors that happen before the HTML header is written. " "Off in production because it leaks internals; on when " "debugging a CGI that dies immediately."), h("suppressVeryEarlyErrors", "flag", "hg/lib/hgConfig.c", default="FALSE", role="knob", verified=True, note="The opposite switch, for hiding a broken hg.conf from users."), h("hgta.disableAllTables", "flag", "hg/lib/hCommon.c", default="FALSE", role="knob", public=True, verified=True, note="Remove the all-tables option from the Table Browser. A load " "control a mirror is entitled to set."), h("hgta.disableSendOutput", "flag", "hg/hgTables/mainPage.c", default="FALSE", role="knob", public=True, verified=True, note="Remove the send-output-to-Galaxy destinations."), h("udc.useLocalDiskCache", "flag", "hg/lib/hui.c", default="TRUE", role="knob", public=True, verified=True, note="Use the local UDC cache. A mirror on a read-only filesystem " "turns this off."), h("db.neverLocal", "flag", "hg/lib/jksql.c", default="0", role="knob", verified=True, note="Never treat the database as local, so no local file " "shortcuts. Deployment topology, not a feature."), h("traceGbdb", "flag", "hg/lib/hdb.c", default="FALSE", role="knob", verified=True, note="Log every /gbdb file the CGI opens. A diagnostic."), h("drawDot", "flag", "hg/hgc/hgc.c", default="FALSE", role="knob", verified=True, note="Emit graphviz dot output from the details page instead of a " "rendered image. A developer diagnostic."), h("login.https", "flag", "hg/lib/wikiLink.c", default="TRUE", role="knob", public=True, verified=True, note="Require https for login. A mirror without a certificate has " "to be able to turn this off."), h("login.basicAuth", "flag", "hg/lib/wikiLink.c", default="FALSE", role="knob", public=True, verified=True, note="Take identity from HTTP basic auth rather than the login " "system."), h("login.relativeLink", "flag", "hg/lib/hdb.c", default="FALSE", role="knob", public=True, verified=True, note="Relative rather than absolute login links."), h("login.acceptAnyId", "flag", "hg/lib/wikiLink.c", default="FALSE", role="knob", verified=True, note="Accept any identity token. Development only, and dangerous " "on a public machine."), h("login.acceptIdx", "flag", "hg/lib/wikiLink.c", default="FALSE", role="knob", verified=True, note="Companion to login.acceptAnyId."), h("login.pwdEyeIcon", "flag", "hg/hgLogin/hgLogin.c", default="TRUE", role="knob", verified=True, note="Show-password eye icon on the login form."), h("login.emailLink", "flag", "hg/hgLogin/hgLogin.c", default="FALSE", role="knob", public=True, verified=True, ticket="37929", note="Passwordless sign-in: the user is emailed a one-time link " "instead of typing a password. The same switch shows the " "change-email page, since that page has no password check " "either. It needs working outbound mail, so a mirror without " "it leaves this off permanently, and that is what settles it as " "a knob rather than a gate: a machine that cannot send mail can " "never turn it on, so there is no release at which the flag " "could be deleted. If the RR's default ever flips TRUE the " "flag still has to stay for everyone else."), h("login.recovEmailChange", "flag", "hg/hgLogin/hgLogin.c", default="FALSE", role="knob", public=True, verified=True, ticket="38197", note="Offer the page where a signed-in user sets or changes the " "recovery email address on their account. A knob for the same " "reason login.emailLink is one: the new address is confirmed by " "mail before it takes effect, so a mirror that cannot send mail " "leaves this off permanently and there is no release at which " "the flag could be deleted. It also needs login.cookieSalt to " "sign the confirmation link and the gbMembers.recovEmailVerified " "column, and hgLogin hides the page where any of the three is " "missing.", debatable="A gate if the intent is really to ship the page dark and " "then flip it: it was added defaulting FALSE so the page " "could sit on master through QA, which is what a gate is " "for. Filed as a knob because the mail requirement it " "shares with login.emailLink outlives that: even after the " "RR turns it on, a mirror without outbound mail needs the " "switch, so there is no release at which it could be " "deleted."), h("analytics.trackClicks", "flag", "hg/lib/googleAnalytics.c", default="TRUE", role="knob", verified=True, note="Report link clicks to analytics. A mirror with its own " "privacy policy turns this off."), h("analytics.trackButtons", "flag", "hg/lib/googleAnalytics.c", default="TRUE", role="knob", verified=True, note="Report button presses to analytics."), h("wikiTrack.readOnly", "flag", "hg/lib/wikiTrack.c", default="FALSE", role="knob", verified=True, deprecated=True, note="Make the wiki annotation track read-only. The wiki track " "itself is effectively retired."), h("cdw.siteIsPublic", "flag", "hg/cirm/cdw/cdwGetFile/cdwGetFile.c", default="FALSE", role="knob", verified=True, deprecated=True, note="CIRM data warehouse is public. Belongs to the cirm CGIs, " "which are not part of the browser release."), h("multiRegionButtonTop", "flag", "hg/hgTracks/config.c", default="TRUE", role="knob", public=True, verified=True, note="Where the multi-region button lives, which is a layout " "preference a mirror is entitled to keep, so a knob. Two " "reads use it through MULTI_REGION_CFG_BUTTON_TOP: the top-bar " "button at hgTracks.c:9166 and the \"Show all\" checkbox in the " "multi-region dialog at config.c:1024. They disagreed about " "the compiled-in default for four years, because e1f7896a08f " "turned the flag on by default in 2022 and changed only the " "hgTracks.c read, so on a machine that did not set the flag the " "button was in the top bar while the checkbox was hidden. " "config.c now defaults TRUE as well."), h("ignoreDefaultKnown", "flag", "hg/lib/hdb.c", default="FALSE", role="knob", verified=True, note="In hdbDefaultKnownDb, ignore the defaultKnown table and treat " "the requested db as its own known-genes db. A property of a " "machine whose gene tables are not laid out the way the RR's " "are, so a knob; it gates no feature and there is nothing to " "flip."), h("blatOldTracks", "internal", "hg/hgc/hgc.c", default="keep", verified=True, note="What happens to the custom tracks left behind by a user's " "earlier BLAT searches when a new search makes another one. " "Three values: keep (the default) leaves every one of them " "alone, hide leaves them in the session but sets them to hide, " "delete drops them from the session and lets their trash files " "age out. Only tracks tagged blatResult=on are touched, and " "the track the current search just made is always left alone. " "A string rather than a flag, so it carries no gate/knob role, " "but it is a knob in spirit: which of the three a site wants " "depends on whether its users run many searches in a session " "and get confused about which results are current, and that " "does not stop being a question after a release. When set to " "delete, the BLAT search form shows a Keep results checkbox so " "a user can opt out for one search (cart variable " "blatKeepResults, read back in hg/hgc/hgc.c)."), h("blatOnlyLatestCheckbox", "internal", "hg/hgBlat/hgBlat.c", default="off", verified=True, ticket="36292", note="Whether the BLAT search form offers a \"Keep only last " "search\" checkbox, the opt-in inverse of blatOldTracks above: " "results accumulate by default and a user asks for the earlier " "ones to be removed. Two call sites that have to agree, " "hgBlat.c:2815 which shows the box and hgc.c:27616 which acts " "on the cart variable blatOnlyLatest it sets, so a stale cart " "value cannot delete tracks on a machine where the feature is " "off. A knob in spirit for the same reason as blatOldTracks, " "and a string rather than a flag, so it carries no gate/knob " "role. Being a string has one consequence worth knowing: it " "is compared with sameString against \"on\", so only that " "exact value turns it on and true, 1 and yes do not, unlike " "every flag read through cfgOptionBooleanDefault."), ], } # --------------------------------------------------------------------------- # database connections and profiles # --------------------------------------------------------------------------- DATABASE = { "what": "Where the CGIs find MySQL. A profile is a prefix; see the " "profile suffix family for the settings legal under any of them.", "vars": [ h("db.host", "profile", "hg/qaPushQ/qaPushQ.c", public=True, verified=True, env="HGDB_HOST", family="db", note="The main assembly database server."), h("db.user", "profile", "hg/hgc/hgc.c", public=True, verified=True, env="HGDB_USER", family="db"), h("db.password", "credential", "hg/hgc/hgc.c", public=True, verified=True, env="HGDB_PASSWORD", family="db"), h("db.trackDb", "table", "hg/lib/hdb.c", public=True, verified=True, env="HGDB_TRACKDB", family="db", note="Comma-separated list of trackDb tables, searched in order. " "This is how a developer layers a personal trackDb over " "production, and why a sandbox hg.conf can be much slower than " "the CGI's own."), h("db.metaDb", "table", "hg/lib/mdb.c", verified=True, family="db"), h("db.grp", "table", "hg/lib/hdb.c", default="grp", public=True, verified=True, family="db", note="Comma-separated list of grp tables holding the track groups. " "The name reaches cfgOption as a parameter of loadGrps(), so " "the harvester can only see it as {confName} and the reconcile " "would otherwise call it dead documentation."), h("db.relatedTrack", "table", "hg/lib/hui.c", default='"relatedTrack"', verified=True, family="db"), h("central.host", "profile", "hg/qaPushQ/qaPushQ.c", public=True, verified=True, family="central", note="hgcentral, which holds sessions, users, hub status and the " "assembly list."), h("central.user", "profile", "hg/qaPushQ/qaPushQ.c", public=True, verified=True, family="central"), h("central.password", "credential", "hg/qaPushQ/qaPushQ.c", public=True, verified=True, family="central"), h("central.db", "profile", "hg/hgc/lowelab.c", public=True, verified=True, family="central"), h("central.domain", "internal", "hg/hubApi/apiUtils.c", public=True, verified=True, required=True, family="central", note="Cookie domain for the central cookie. Read with cfgVal in " "hubApi, so that CGI will not start without it."), h("central.cookie", "internal", "hg/lib/hui.c", default='"hguid"', public=True, verified=True, family="central", note="Name of the user-identity cookie. Changing it logs every " "user out."), h("cart.host", "profile", "hg/lib/hdb.c", public=True, verified=True, family="cart", note="Optional separate server for cart traffic, which is the " "heaviest write load in the browser."), h("cart.user", "profile", "hg/lib/hdb.c", public=True, verified=True, family="cart"), h("cart.password", "credential", "hg/lib/hdb.c", public=True, verified=True, family="cart"), h("cart.db", "profile", "hg/lib/hdb.c", public=True, verified=True, family="cart"), h("customTracks.host", "profile", "product/ex.hg.conf", public=True, family="customTracks", note="Read through the profile mechanism, so it has no literal call " "site in the tree."), h("customTracks.user", "profile", "product/ex.hg.conf", public=True, family="customTracks"), h("customTracks.password", "credential", "product/ex.hg.conf", public=True, family="customTracks"), h("customTracks.tmpdir", "path", "hg/lib/customAdjacency.c", default='"/data/tmp"', public=True, verified=True, family="customTracks"), h("customTracks.maxBytes", "limit", "hg/lib/customFactory.c", public=True, verified=True, family="customTracks"), h("customTracks.useAll", "flag", "hg/lib/customTrack.c", default="NULL", public=True, verified=True, family="customTracks"), h("customTracks.botCheckMult", "limit", "hg/lib/customTrack.c", default='"1"', verified=True, family="customTracks", note="Multiplier on the bot-delay penalty for custom track loads."), h("showTableCache", "table", "hg/lib/jksql.c", default='"tableList"', public=True, verified=True, required=True, note="Table holding a cached list of table names, which avoids a " "slow SHOW TABLES. One of the four settings read with cfgVal."), ], } # --------------------------------------------------------------------------- # hgcentral table names # --------------------------------------------------------------------------- CENTRAL_TABLES = { "what": "Names of the hgcentral tables. Nearly all are overridable from " "the environment as well, which is how a test instance points at " "its own copies without editing hg.conf.", "vars": [ h("dbDbTableName", "table", "hg/lib/hdb.c", public=True, verified=True, env="HGDB_DBDBTABLE", default="dbDb", note="The assembly list."), h("defaultDbTableName", "table", "hg/lib/hdb.c", public=True, verified=True, env="HGDB_DEFAULTDBTABLE", default="defaultDb"), h("cladeTableName", "table", "hg/lib/hdb.c", public=True, verified=True, env="HGDB_CLADETABLE", default="clade"), h("genomeCladeTableName", "table", "hg/lib/hdb.c", public=True, verified=True, env="HGDB_GENOMECLADETABLE", default="genomeClade"), h("userDbName", "table", "hg/lib/cartDb.c", public=True, verified=True, env="HGDB_USERDBTABLE", default="userDb", note="Per-user cart storage."), h("sessionDbName", "table", "hg/lib/cartDb.c", public=True, verified=True, env="HGDB_SESSIONDBTABLE", default="sessionDb"), h("defaultCartName", "table", "hg/lib/cartDb.c", public=True, verified=True, env="HGDB_DEFAULTCARTTABLE"), h("namedSessionDbName", "table", "hg/lib/cart.c", verified=True, env="HGDB_NAMED_SESSION_DB", default="namedSessionDb", note="Saved sessions."), h("hub.publicTableName", "table", "hg/hgHubConnect/hgHubConnect.c", public=True, verified=True, env="HGDB_HUB_PUBLIC_TABLE", family="hub"), h("hub.statusTableName", "table", "hg/hgHubConnect/hgHubConnect.c", public=True, verified=True, env="HGDB_HUB_STATUS_TABLE", family="hub"), h("hub.genArkTableName", "table", "hg/lib/genark.c", verified=True, env="HGDB_GENARK_STATUS_TABLE", family="hub"), h("hub.assemblyListTableName", "table", "hg/lib/assemblyList.c", verified=True, env="HGDB_ASSEMBLYLIST_STATUS_TABLE", family="hub"), h("liftOverChainName", "table", "hg/lib/liftOver.c", verified=True, env="LIFTOVERCHAINNAME", default="liftOverChain"), h("quickLiftChainName", "table", "hg/lib/quickLift.c", verified=True, env="QUICKLIFTCHAINNAME", default="quickLiftChain", ticket="37788"), h("blatServersTbl", "table", "hg/hgPcr/hgPcr.c", default='"blatServers"', verified=True), h("hubSearchTextTable", "table", "hg/hgGateway/hgGateway.c", default='"hubSearchText"', verified=True), h("authTableName", "table", "hg/lib/hubSpaceKeys.c", verified=True, note="Hub space API keys."), h("ottoTable", "table", "hg/hubApi/apiUtils.c", verified=True, note="Table the hub API reads to report otto track update times."), h("genbankDb", "profile", "hg/hgVai/hgVai.c", public=True, verified=True, env="GENBANKDB"), h("cart.trace", "debug", "hg/lib/cart.c", verified=True, note="Log every cart read and write. Very noisy; a debugging aid " "for session problems."), h("cartVersion", "internal", "hg/cgilib/cartRewrite.c", default='"on"', verified=True, note="Run the cart rewrite steps that migrate old sessions " "forward. Turning it off strands old sessions."), h("browser.sessionKey", "internal", "hg/lib/cartDb.c", verified=True, note="Read through cfgOption2 with a literal prefix, so it is one " "of the few two-part names that does appear as a literal."), ], } # --------------------------------------------------------------------------- # paths and caches # --------------------------------------------------------------------------- PATHS = { "what": "Where things live on disk. These are the settings a mirror is " "most likely to have to change.", "vars": [ h("gbdbLoc1", "path", "hg/lib/hdb.c", public=True, verified=True, note="Primary /gbdb location. Any C code opening a /gbdb path is " "supposed to run it through hReplaceGbdb() so this takes " "effect."), h("gbdbLoc2", "path", "hg/lib/hdb.c", public=True, verified=True, note="Fallback /gbdb location, tried when a file is missing from " "gbdbLoc1. This is how a mirror keeps part of /gbdb local and " "the rest remote."), h("udc.cacheDir", "path", "hg/lib/hui.c", default="udcDefaultDir()", public=True, verified=True, note="UDC cache for remote bigData files. Wants real disk: this is " "where every hub file lands."), h("udc.localDir", "path", "hg/lib/customFactory.c", public=True, verified=True), h("udcLog", "debug", "hg/hgTracks/hgTracks.c", verified=True, note="Log UDC fetches, which is the first thing to turn on when a " "hub is slow."), h("cacheTrackDbDir", "path", "hg/lib/trackDbCache.c", default='"/dev/shm/trackDbCache"', verified=True, note="Shared-memory cache of parsed trackDb. Setting it empty " "forces a fresh read from MySQL every request, which is the " "right way to test trackDb changes; deleting the directory is " "not, since it is shared between users."), h("sessionDataDir", "path", "hg/lib/sessionData.c", verified=True, note="Where session-scoped data (custom tracks belonging to a saved " "session) is kept so it survives cleanup. Four call sites. " "sessionData.c is what gives the setting its meaning: " "sessionDirFromNames builds the hashed directory layout under " "it and errAborts unless the path is absolute. The row used to " "cite hgPcr.c, which stopped reading it at 554396e4474 and now " "asks trashDir.c's isTrashOrSessionDataPath instead."), h("sessionDataDirOld", "path", "hg/lib/trashDir.c", verified=True, note="Previous location, still read so old sessions " "keep working."), h("sessionDataDbPrefix", "internal", "hg/lib/sessionData.c", verified=True), h("sessionThumbnail.imgDir", "path", "hg/cgilib/sessionThumbnail.c", public=True, verified=True, family="sessionThumbnail"), h("sessionThumbnail.webPath", "url", "hg/cgilib/sessionThumbnail.c", public=True, verified=True, family="sessionThumbnail"), h("sessionThumbnail.convertPath", "path", "hg/hgSession/hgSession.c", public=True, verified=True, family="sessionThumbnail"), h("sessionThumbnail.suppress", "flag", "hg/hgSession/hgSession.c", public=True, verified=True, family="sessionThumbnail"), h("freeTypeDir", "path", "hg/hgTracks/config.c", default='"../htdocs/urw-fonts"', verified=True), h("freeTypeFont", "internal", "hg/cgilib/trackLayout.c", default='"Bitmap"', verified=True), h("fonts.extra", "path", "hg/cgilib/trackLayout.c", default="NULL", public=True, verified=True), h("textSize", "internal", "hg/cgilib/trackLayout.c", default='"small"', verified=True), h("tusdDataDir", "path", "hg/lib/userdata.c", verified=True, family="hubSpace", note="Hub space upload staging."), h("tusdMountPoint", "path", "hg/lib/userdata.c", verified=True, family="hubSpace"), h("hubSpaceUrl", "url", "hg/lib/userdata.c", verified=True, family="hubSpace"), h("hubSpaceTusdEndpoint", "url", "hg/hgHubConnect/trackHubWizard.c", default="NULL", verified=True, family="hubSpace"), h("myVariantsDataDir", "path", "hg/lib/myVariants.c", verified=True, note="Goes with the doMyVariants gate."), h("hgPhyloPlaceServerDir", "path", "hg/hgPhyloPlace/runUsher.c", verified=True), h("browser.documentRoot", "path", "hg/lib/hui.c", default="DOCUMENT_ROOT", public=True, verified=True), h("browser.cgiRoot", "path", "hg/lib/hui.c", default="defaultDir", public=True, verified=True), h("browser.javaScriptDir", "path", "hg/lib/web.c", default='"js"', public=True, verified=True), h("browser.styleDir", "path", "hg/lib/web.c", default='"style"', public=True, verified=True), h("browser.styleImagesDir", "path", "hg/lib/web.c", default='"style/images"', public=True, verified=True), h("browser.trixPath", "path", "hg/cgilib/search.c", default='"/gbdb/$db/trackDb.ix"', public=True, verified=True, note="Track search index. The $db is substituted at runtime."), h("downloads.server", "url", "hg/lib/hui.c", default='"hgdownload.soe.ucsc.edu"', public=True, verified=True), h("cramRef", "path", "hg/hgTables/bam.c", public=True, verified=True, note="Reference sequence cache for CRAM."), h("grepIndex.default", "path", "hg/lib/hgFind.c", public=True, verified=True, family="grepIndex"), h("grepIndex.genbank", "path", "hg/lib/hgFind.c", public=True, verified=True, family="grepIndex"), ], } # --------------------------------------------------------------------------- # limits and load control # --------------------------------------------------------------------------- LIMITS = { "what": "Caps on what one request may consume. These are the settings " "that decide whether a heavy request is answered slowly or " "refused.", "vars": [ h("maxMem", "limit", "hg/lib/hgConfig.c", public=True, verified=True, note="Address-space cap applied by cfgSetMaxMem() at CGI startup. " "Exceeding it is what produces the hogExit entries in the " "error log."), h("warnSeconds", "limit", "hg/hgTracks/hgTracks.c", verified=True, note="Log a warning for any hgTracks render slower " "than this."), h("hubSpaceLockTimeout", "limit", "hg/lib/userdata.c", default='"300"', verified=True, ticket="37964", note="Seconds an upload will wait for another upload to the same " "hub to finish before giving up. lockHubDir takes an flock on " "hubDir/.hub.lock to serialise the hub.txt " "read-modify-write across parallel pre-finish hooks, and polls " "with LOCK_NB every 100ms rather than blocking, so one wedged " "upload cannot hold a hub indefinitely; on expiry the user is " "told to try again. The call site clamps it to 1..3600 and " "falls back to the compiled-in 300 outside that, so a " "mistyped hg.conf value cannot produce a negative or " "unbounded wait. Not marked public: hubSpace is a " "UCSC-hosted feature and its other settings are not " "documented for mirrors either."), h("maxItemsPossible", "limit", "hg/hgTracks/simpleTracks.c", default='"100000"', public=True, verified=True), h("BAMMaxItems", "limit", "hg/hgTracks/bamTrack.c", default='"10000"', verified=True), h("bigBedMaxItems", "limit", "hg/hgTracks/bigBedTrack.c", default='"10000"', public=True, verified=True), h("vcfMaxItems", "limit", "hg/hgTracks/vcfTrack.c", default='"10000"', public=True, verified=True), h("maxTrackImageHeightPx", "limit", "hg/hgTracks/hgTracks.c", default='"32000"', verified=True, note="Hard ceiling on image height, which is what stops a dense " "track from producing a PNG no browser will render."), h("maxDisplayPixelWidth", "limit", "hg/cgilib/trackLayout.c", default="NULL", public=True, verified=True), h("barbMergePixels", "limit", "hg/hgTracks/simpleTracks.c", default='"3"', public=True, verified=True), h("quickLift.lengthLimit", "limit", "hg/lib/quickLift.c", default='"10000"', verified=True, ticket="37788"), h("liftDailyLimit", "limit", "hg/hubApi/apiUtils.c", verified=True, note="Per-day liftOver cap for the hub API."), h("hgBlat.maxSequenceCount", "limit", "hg/hgBlat/hgBlat.c", default="NULL", public=True, verified=True), h("parallelFetch.threads", "limit", "hg/hgBlat/hgBlat.c", default='"20"', public=True, verified=True), h("parallelFetch.timeout", "limit", "hg/hgBlat/hgBlat.c", default='"90"', public=True, verified=True), h("logCgiVarMaxLen", "limit", "hg/lib/hgConfig.c", default='"0"', public=True, verified=True, note="Truncate logged CGI variables at this length. Zero disables " "the logging that cfgSetLogCgiVars() would otherwise do."), h("mallocTopPad", "limit", "hg/lib/hgConfig.c", verified=True, ticket="38225", note="Bytes of heap the C library takes from the kernel at a time, " "passed to mallopt(M_TOP_PAD) by cfgSetMallocTopPad() at CGI " "startup. Glibc's own step is 128 kB, and hgTracks draws its " "tracks in parallel threads that each grow their own pool, so " "a heavy render spends over a hundred thousand system calls on " "growing the heap. A bigger step buys that time back and " "costs resident memory on the same renders. Left out, or set " "to anything that is not a positive number, and glibc's " "behaviour is untouched; the whole read sits inside " "#if defined(__GLIBC__), so it does nothing on a machine with " "another C library. A limit rather than a flag, so it carries " "no gate/knob role, but it is a knob in spirit: the right " "value depends on how much memory the machine can spend, and " "that question does not end at a release. The measurements " "are on #38225 and the value for the RR is not settled."), h("snapshot.ttlDays", "limit", "hg/utils/snapshotCleaner/snapshotCleaner.c", verified=True, note="Days an anonymous Share-a-link snapshot session is kept " "before snapshotCleaner deletes it. An anonymous snapshot is " "saved under the reserved user \"l\" with a server-generated " "name, so no account owns it and nothing else would ever " "remove it. The compiled-in default is " "snapshotDefaultTtlDays, 4 * 365 in hg/inc/snapshotSession.h, " "and the -ttlDays command line option wins over hg.conf. The " "utility aborts on a value below 1, so a mistyped setting " "deletes nothing. Read by that utility rather than by a CGI, " "and nothing in the tree schedules it: a machine that keeps " "anonymous snapshots has to run the cleaner itself."), ], } # --------------------------------------------------------------------------- # abuse control # --------------------------------------------------------------------------- ABUSE = { "what": "Bot delay, rate limiting and captchas. Mostly read from " "hg/lib/botDelay.c.", "vars": [ h("bottleneck.host", "url", "hg/lib/botDelay.c", public=True, verified=True, family="bottleneck", note="The bottleneck server that tracks per-IP request rates."), h("bottleneck.port", "internal", "hg/lib/botDelay.c", public=True, verified=True, family="bottleneck"), h("bottleneck.except", "internal", "hg/lib/botDelay.c", verified=True, family="bottleneck", note="IPs exempt from delay."), h("hguidIpTracking.maxIps", "limit", "hg/lib/botDelay.c", default='"10"', verified=True, family="hguidIpTracking"), h("hguidIpTracking.windowSeconds", "limit", "hg/lib/botDelay.c", default='"600"', verified=True, family="hguidIpTracking"), h("hguidIpTracking.table", "table", "hg/lib/botDelay.c", default='"hguidIpAccess"', verified=True, family="hguidIpTracking"), h("cloudFlareSiteKey", "credential", "hg/lib/cart.c", verified=True, required=True, note="Turnstile captcha site key. Read with cfgVal, so a machine " "that enables the captcha path must set it."), h("cloudFlareSecretKey", "credential", "hg/lib/cart.c", verified=True, required=True), h("captchaDebug", "flag", "hg/lib/cart.c", default="FALSE", role="knob", verified=True, note="Log the session id to stderr each time printCaptcha() shows " "the captcha, as \"CAPTCHA_PRINT \". A permanent " "debugging aid for tracing why a given session is being " "captcha-gated, not a feature being shipped dark."), h("noCaptchaAgent.", "internal", "hg/lib/botDelay.c", verified=True, note="A prefix family rather than one setting: every " "noCaptchaAgent.* value is a user-agent string exempt from the " "captcha. Enumerated at runtime with cfgNamesWithPrefix()."), ], } # --------------------------------------------------------------------------- # logging and diagnostics # --------------------------------------------------------------------------- LOGGING = { "what": "Diagnostics. Several of these are safe to leave on and a few " "are expensive, so they are worth telling apart.", "vars": [ h("browser.cgiTime", "debug", "hg/lib/cart.c", default='"yes"', public=True, verified=True, note="Log per-CGI timing. On by default and cheap; this is what " "the log analysis relies on."), h("trackLog", "debug", "hg/hgTracks/hgTracks.c", default='"off"', verified=True, note="Log which tracks were drawn per request."), h("noSqlInj.level", "internal", "hg/lib/cart.c", default='"abort"', verified=True, family="noSqlInj", note="What to do when the SQL injection guard fires: abort, warn or " "ignore. Production wants abort."), h("noSqlInj.dumpStack", "debug", "hg/lib/cart.c", verified=True, family="noSqlInj"), h("signalsHandler", "internal", "hg/lib/cart.c", public=True, verified=True, note="Install handlers that turn a segfault into a " "logged error rather than a blank page."), h("httpsCertCheck", "internal", "hg/lib/cart.c", public=True, verified=True, family="httpsCertCheck", note="How strictly to verify certificates on outbound https, which " "matters because hubs are fetched over it."), h("httpsCertCheckVerbose", "debug", "hg/lib/cart.c", verified=True, family="httpsCertCheck"), h("httpsCertCheckDepth", "internal", "hg/lib/cart.c", verified=True, family="httpsCertCheck"), h("httpsCertCheckDomainExceptions", "internal", "hg/lib/cart.c", public=True, verified=True, family="httpsCertCheck"), h("httpProxy", "url", "hg/lib/cart.c", public=True, verified=True, family="proxy"), h("httpsProxy", "url", "hg/lib/cart.c", public=True, verified=True, family="proxy"), h("ftpProxy", "url", "hg/lib/cart.c", public=True, verified=True, family="proxy"), h("noProxy", "internal", "hg/lib/cart.c", public=True, verified=True, family="proxy"), h("logProxy", "debug", "hg/lib/cart.c", verified=True, family="proxy"), h("pngTimingSampleRate", "debug", "hg/hgTracks/hgTracks.c", default="0", verified=True, ticket="38109", note="One page load in this many measures how long the track image " "took to reach the reader. hgTracks writes the rate into the " "page as a JavaScript variable and reportPngTiming() in " "hg/js/hgTracks.js does the rest: it reads the browser's own " "timing record for the image, then puts the bytes and the " "download time on the query string of a 43 byte image, so one " "apache log line is the whole measurement and no process has " "to start. This is the part of the wait no server-side timing " "can see, because apache stops timing once the kernel has the " "bytes. Zero, or the setting left out, and nothing is written " "into the page, so the beacon is off. It samples rather than " "reporting every load because each report is another request."), ], } # --------------------------------------------------------------------------- # branding and site text # --------------------------------------------------------------------------- BRANDING = { "what": "Text, styling and links that differ between UCSC and a mirror.", "vars": [ h("browser.style", "internal", "hg/lib/cart.c", public=True, verified=True, note="Stylesheet override."), h("browser.theme.", "internal", "hg/hgTracks/config.c", public=True, verified=True, note="A prefix family: each browser.theme.N.Name value defines a " "selectable theme. Enumerated with cfgNamesWithPrefix(), " "which is why the individual names never appear in the source."), h("addJs", "internal", "hg/lib/web.c", verified=True, note="Extra JavaScript file to include on every page."), h("help.html", "path", "hg/lib/hui.c", verified=True), h("hgTracksNoteHtml", "internal", "hg/hgTracks/hgTracks.c", public=True, verified=True, note="A banner on the browser page. This is where a mirror puts " "its own notice."), h("survey", "url", "hg/hgGateway/hgGateway.c", public=True, verified=True, env="HGDB_SURVEY", note="Survey link on the gateway. Set to 'off' to hide it."), h("surveyLabel", "internal", "hg/hgGateway/hgGateway.c", default='"Please take our survey"', public=True, verified=True, env="HGDB_SURVEY_LABEL"), h("surveyLabelImage", "url", "hg/hgGateway/hgGateway.c", verified=True), h("hubSurvey", "url", "hg/hgHubConnect/hgHubConnect.c", verified=True, env="HGDB_HUB_SURVEY"), h("hubSurveyLabel", "internal", "hg/hgHubConnect/hgHubConnect.c", verified=True, env="HGDB_HUB_SURVEY_LABEL"), h("searchHelpUrl", "url", "hg/hgTracks/hgTracks.c", default='"../goldenPath/help/query.html"', verified=True), h("searchHelpLabel", "internal", "hg/hgTracks/hgTracks.c", default='"Examples"', verified=True), h("analyticsKey", "credential", "hg/lib/googleAnalytics.c", public=True, verified=True), h("mouseOverEnabled", "internal", "hg/hgTracks/hgTracks.c", default='"on"', verified=True, note="Not the same thing as the showMouseovers gate: this one is on " "by default and controls the existing tooltip behaviour. The " "near-identical names are a trap worth fixing."), h("bigWarn", "internal", "hg/hgTracks/bigWarn.c", default='"on"', public=True, verified=True), h("defaultGenome", "internal", "hg/lib/hdb.c", default="DEFAULT_GENOME", public=True, verified=True), h("browser.popularGenomes", "internal", "hg/hgIntegrator/hgIntegrator.c", default='"hg38,hg19,mm39,mm10..."', verified=True), h("geneTracks", "internal", "hg/lib/hgFind.c", verified=True), h("browser.recTrackSets", "internal", "hg/hgTracks/recTrackSets.c", verified=True, family="recTrackSets", note="Recommended track sets. Goes with the mergeRecommended " "gate."), h("browser.recTrackSetsDetectChange", "internal", "hg/hgTracks/recTrackSets.c", verified=True, family="recTrackSets", note="Drives the 'session changed' banner. The mergeRecommended " "gate belongs to the same feature."), h("browser.exportedDataHubs", "internal", "hg/lib/exportedDataHubs.c", verified=True), h("browser.cgiExpireMinutes", "limit", "hg/hgHubConnect/hgHubConnect.c", default='"20"', verified=True), h("curatedHubPrefix", "internal", "hg/lib/hubConnect.c", verified=True, note="Which curated hubs this machine shows."), h("genarkHubPrefix", "internal", "hg/hgGateway/hgGateway.c", verified=True), h("test.preview", "flag", "hg/lib/hdb.c", verified=True, note="Marks a preview machine, which changes some banners and " "links. Part of the release plumbing, but a permanent part."), h("restoreMapFind", "internal", "hg/hgTracks/imageV2.c", verified=True), h("blatNewFormSwitchDate", "branding", "hg/hgBlat/hgBlat.c", verified=True, ticket="37996", note="The day the classic BLAT form's banner names as the day the " "new form takes over, written the way it is shown, e.g. " "\"October 21\". Set it and the banner says when the change " "happens and links a news announcement; leave it out and the " "banner keeps its original wording, which only invites the " "reader to try the new page. It is a setting so that a " "slipped date is an hg.conf edit rather than a CGI build " "patch. A string, so no gate/knob role, but it is as " "temporary as blatNewFormBanner: it has nothing left to say " "once the new form is the only form, and goes away with it."), h("blatNewFormNewsUrl", "url", "hg/hgBlat/hgBlat.c", default="../goldenPath/newsarch.html", verified=True, ticket="37996", note="Where the \"news announcement\" link in that banner points. " "The default is the news archive, so the link goes somewhere " "sensible while the announcement is still being written, and " "hg.conf can aim it at the item's own anchor later without a " "rebuild. Read only on the path blatNewFormSwitchDate opens, " "so on a machine that sets no date it does nothing."), ], } # --------------------------------------------------------------------------- # geographic mirroring # --------------------------------------------------------------------------- GEO = { "what": "Settings for the geographically distributed mirrors, which route " "users to a nearby machine.", "vars": [ h("browser.node", "internal", "hg/lib/geoMirror.c", public=True, verified=True, note="Which node this machine is. Drives the geo redirect."), h("browser.geoSuffix", "internal", "hg/geoIpToCountry/geoIpToCountry.c", default='""', verified=True, note="Suffix appended to central table names so each node can have " "its own copies."), ], } # --------------------------------------------------------------------------- # login and wiki # --------------------------------------------------------------------------- LOGIN = { "what": "The login system, and the older wiki-based identity it replaced. " "Most of these are read through macros, which is why the harvester " "cannot date them.", "vars": [ h("login.systemName", "internal", "hg/lib/wikiLink.c", public=True, verified=True, family="login"), h("login.browserName", "internal", "hg/hgLogin/hgLogin.c", public=True, verified=True, family="login"), h("login.browserAddr", "url", "hg/hgLogin/hgLogin.c", public=True, verified=True, family="login"), h("login.mailSignature", "internal", "hg/hgLogin/hgLogin.c", public=True, verified=True, family="login"), h("login.mailReturnAddr", "email", "hg/hgLogin/hgLogin.c", public=True, verified=True, family="login"), h("login.approvedReturn", "url", "hg/lib/wikiLink.c", default="NULL", verified=True, family="login", ticket="36485", note="Comma-separated hosts hgLogin will accept in a returnto " "argument, on top of the login host itself. Absent means no " "host is checked, which is the older behaviour. Read through " "the CFG_APPROVED_HOSTS macro, so a search for the name finds " "only comments. It moved out of hgLogin.c into the library at " "75593a4e317 so that a CGI building a returnto can apply the " "same rule before it writes the link, rather than finding out " "from an error page."), h("login.cookieSalt", "credential", "hg/hgPhyloPlace/hgPhyloPlace.c", public=True, verified=True, family="login", note="Salt for the login cookie. A secret."), h("login.oauth.providers", "internal", "hg/hgLogin/oauthLogin.c", public=True, verified=True, family="login", ticket="37984", note="Comma-separated list of social sign-in providers to offer. " "Each name listed here is then configured through its own " "login.oauth..* settings, none of which appear as " "literals in the tree; see {key}. google, orcid and github " "are picked up even when unlisted, if they carry a clientId."), h("wiki.host", "url", "hg/lib/wikiLink.c", public=True, verified=True, family="wiki", deprecated=True), h("wiki.userNameCookie", "internal", "hg/lib/wikiLink.c", default='"hgLoginUserName"', public=True, verified=True, family="wiki"), h("wiki.loggedInCookie", "internal", "hg/lib/wikiLink.c", default='"hgLoginIdKey"', public=True, verified=True, family="wiki"), h("wiki.sessionCookie", "internal", "hg/lib/wikiTrack.c", public=True, verified=True, family="wiki", deprecated=True), h("wikiTrack.URL", "url", "hg/hgGene/wikiTrack.c", default="NULL", public=True, verified=True, family="wikiTrack", deprecated=True), h("wikiTrack.browser", "internal", "hg/hgGene/wikiTrack.c", default="DEFAULT_BROWSER", public=True, verified=True, family="wikiTrack", deprecated=True), h("wikiTrack.dbList", "internal", "hg/lib/wikiTrack.c", public=True, verified=True, family="wikiTrack", deprecated=True), h("wikiTrack.editors", "internal", "hg/hgc/variomeClick.c", default="NULL", public=True, verified=True, family="wikiTrack", deprecated=True), ], } # --------------------------------------------------------------------------- # external services # --------------------------------------------------------------------------- EXTERNAL = { "what": "Hosts, URLs and helper programs outside the browser. Each is a " "dependency that can fail independently of us.", "vars": [ h("galaxyUrl", "url", "hg/hgTables/galaxy.c", public=True, verified=True), h("rnaPlotPath", "path", "hg/hgGene/rnaStructure.c", default='"../cgi-bin/RNAplot"', public=True, verified=True), h("hgc.psxyPath", "path", "hg/hgc/hgdpClick.c", public=True, verified=True, family="hgc"), h("hgc.ps2rasterPath", "path", "hg/hgc/hgdpClick.c", public=True, verified=True, family="hgc"), h("hgc.ghostscriptPath", "path", "hg/hgc/hgdpClick.c", public=True, verified=True, family="hgc"), h("nextstrainHost", "url", "hg/hgPhyloPlace/phyloPlace.c", verified=True), h("microbeTraceHost", "url", "hg/hgPhyloPlace/phyloPlace.c", verified=True), h("hgPhyloPlaceEnabled", "flag", "hg/hgPhyloPlace/phyloPlace.c", verified=True, note="Read with cfgOption rather than the boolean accessor, so " "absent means off. It gates a whole CGI, which is why it is " "not in the gate list: hgPhyloPlace is optional by design, not " "pending release."), h("resolvProts", "internal", "hg/lib/hui.c", public=True, verified=True, family="resolv"), h("resolvPrefix", "internal", "hg/lib/hui.c", public=True, verified=True, family="resolv"), h("resolvCmd", "path", "hg/lib/hui.c", public=True, verified=True, family="resolv"), h("hubApi.allowHtml", "flag", "hg/hubApi/hubApi.c", default='"off"', verified=True, family="hubApi"), h("hubApi.showActive0", "flag", "hg/hubApi/apiUtils.c", default='"off"', verified=True, family="hubApi"), h("hubApi.blatDelayFraction", "limit", "hg/hubApi/blat.c", default="NULL", verified=True, family="hubApi"), h("hubApi.relaySecret", "credential", "hg/hubApi/apiUtils.c", verified=True, family="hubApi"), h("apiFromEmail", "email", "hg/hubApi/liftOver.c", verified=True), h("chainFileRequestEmail", "email", "hg/hubApi/liftOver.c", verified=True), h("newCustomTrackValidate", "flag", "hg/lib/customFactory.c", verified=True, note="Use the newer custom track validator. Reads like a gate but " "is set per machine on purpose while the two validators " "coexist; worth revisiting."), h("suggest.mailToAddr", "email", "hg/hgUserSuggestion/hgUserSuggestion.c", public=True, verified=True, family="suggest"), h("suggest.mailFromAddr", "email", "hg/hgUserSuggestion/hgUserSuggestion.c", public=True, verified=True, family="suggest"), h("suggest.filterKeyword", "internal", "hg/hgUserSuggestion/hgUserSuggestion.c", public=True, verified=True, family="suggest"), h("suggest.mailSignature", "internal", "hg/hgUserSuggestion/hgUserSuggestion.c", public=True, verified=True, family="suggest"), h("suggest.mailReturnAddr", "email", "hg/hgUserSuggestion/hgUserSuggestion.c", public=True, verified=True, family="suggest"), h("suggest.browserName", "internal", "hg/hgUserSuggestion/hgUserSuggestion.c", public=True, verified=True, family="suggest"), h("suggest.siteKey", "credential", "hg/hgUserSuggestion/hgUserSuggestion.c", verified=True, family="suggest"), h("suggest.secretKey", "credential", "hg/hgUserSuggestion/hgUserSuggestion.c", verified=True, family="suggest"), h("suggest.humanThreshold", "limit", "hg/hgUserSuggestion/hgUserSuggestion.c", default='"-0.1"', verified=True, family="suggest"), h("hgGateway.dbDbTaxonomy", "internal", "hg/hgGateway/hgGateway.c", default="defaultDbDbTree", verified=True), h("hgEncodeVocabDocBaseUrl", "url", "hg/encode/hgEncodeVocab/hgEncodeVocab.c", default='""', verified=True, deprecated=True), h("namedSessionAlt.", "internal", "hg/lib/cart.c", verified=True, note="A prefix family enumerated at runtime: alternative " "namedSessionDb locations to search when resolving a shared " "session."), ], } # --------------------------------------------------------------------------- # retired: settings whose feature is gone or was never part of the browser # --------------------------------------------------------------------------- RETIRED = { "what": "Settings for features that are gone, or for CGIs that are not " "part of the browser release. Kept in the catalog because the " "reads are still in the tree and a mirror's hg.conf may still " "carry them. Deleting the code is a separate job from " "sunsetting a release gate, and these are candidates for it.", "vars": [ h("paypalServer", "url", "hg/gsid/gsidMember/gsidMember.c", deprecated=True, verified=True, family="gsid", note="The GSID member site took payments. That CGI is not built " "for the browser."), h("paypalIpnServer", "url", "hg/gsid/gsidMember/gsidMember.c", deprecated=True, verified=True, family="gsid"), h("paypalCommercialFee", "internal", "hg/gsid/gsidMember/gsidMember.c", deprecated=True, verified=True, family="gsid"), h("paypalAcademicFee", "internal", "hg/gsid/gsidMember/gsidMember.c", deprecated=True, verified=True, family="gsid"), h("paypalEmail", "email", "hg/gsid/gsidMember/gsidMember.c", deprecated=True, verified=True, family="gsid"), h("paypalCert", "credential", "hg/gsid/gsidMember/gsidMember.c", deprecated=True, verified=True, family="gsid"), h("gsidCertId", "credential", "hg/gsid/gsidMember/gsidMember.c", deprecated=True, verified=True, family="gsid"), h("gisaid.structDir", "path", "hg/hgc/virusClick.c", deprecated=True, verified=True, family="gisaid"), h("gisaid.structUrl", "url", "hg/hgc/virusClick.c", deprecated=True, verified=True, family="gisaid"), h("genomeSpace.{variable}", "internal", "hg/hgTables/genomeSpace.c", deprecated=True, verified=True, note="GenomeSpace is shut down. Read with cfgOption2 and a runtime " "suffix, so the harvester reports it half-resolved."), h("rtdb.server", "profile", "hg/rtdbWebUpdate/rtdbWebUpdate.c", deprecated=True, verified=True, family="rtdb"), h("rtdb.port", "profile", "hg/rtdbWebUpdate/rtdbWebUpdate.c", deprecated=True, verified=True, family="rtdb"), h("rtdb.databases", "profile", "hg/rtdbWebUpdate/rtdbWebUpdate.c", deprecated=True, verified=True, family="rtdb"), h("pq.host", "profile", "hg/qaPushQ/qaPushQ.c", verified=True, family="pq", note="qaPushQ, a QA tool rather than a browser CGI."), h("pq.user", "profile", "hg/qaPushQ/qaPushQ.c", verified=True, family="pq"), h("pq.password", "credential", "hg/qaPushQ/qaPushQ.c", verified=True, family="pq"), h("pq.db", "profile", "hg/qaPushQ/qaPushQ.c", verified=True, family="pq"), h("pq.crossHost", "internal", "hg/qaPushQ/qaPushQ.c", verified=True, family="pq"), h("rrcentral.host", "profile", "hg/qaPushQ/qaPushQ.c", verified=True, family="rrcentral"), h("rrcentral.user", "profile", "hg/qaPushQ/qaPushQ.c", verified=True, family="rrcentral"), h("rrcentral.password", "credential", "hg/qaPushQ/qaPushQ.c", verified=True, family="rrcentral"), h("rrcentral.db", "profile", "hg/qaPushQ/qaPushQ.c", verified=True, family="rrcentral"), h("encodeDataWarehouse.dataRoot", "path", "hg/encode3/encodeDataWarehouse/edwWebXSendFile/edwWebXSendFile.c", deprecated=True, verified=True, family="encodeDataWarehouse"), h("encodeDataWarehouse.key", "credential", "hg/encode3/encodeDataWarehouse/edwWebXSendFile/edwWebXSendFile.c", deprecated=True, verified=True, family="encodeDataWarehouse"), ], } # --------------------------------------------------------------------------- # names the scan could not resolve # --------------------------------------------------------------------------- RUNTIME_NAMES = { "what": "Reads whose setting name is built at run time, so there is no " "fixed name to document. Each is a small family rather than a " "single setting, and each is a place where a typo in hg.conf is " "silently ignored.", "vars": [ h("{themeKey}", "internal", "hg/lib/cart.c", verified=True, note="browser.theme., resolved from the theme the user " "picked."), h("{cfgName}", "internal", "hg/hgTracks/hgTracks.c", verified=True), h("{confName}", "internal", "hg/lib/hdb.c", verified=True, note="Resolves to db.grp, passed in by loadGrps()'s only caller at " "hdb.c:5479. Catalogued under its real name as well."), h("{confVariable}", "internal", "hg/hgTracks/quickLift.c", verified=True, ticket="37788", note="quickLift colour settings, named per use."), h("{overlapKey}", "internal", "hg/hgc/myVariantsClick.c", verified=True, note="Goes with the doMyVariants gate."), h("{key}", "internal", "hg/hgLogin/oauthLogin.c", verified=True, ticket="37984", note="login.oauth.., built with safef from the " "provider names in login.oauth.providers, so the whole family " "is invisible to any scan. The fields are clientId, " "clientSecret, label, type, issuer, authUrl, tokenUrl, " "userinfoUrl and scopes. A second read tries the older " "login.. spelling, which is why a mirror can " "have credentials under either prefix."), h("{temp}", "internal", "hg/hgcentralTidy/hgcentralTidy.c", verified=True), h("{cdwSetting}", "internal", "hg/cirm/cdw/cdwWebBrowse/cdwWebBrowse.c", verified=True, deprecated=True), ], } AWAITING_REVIEW = { "what": "Settings the tree reads that --auto-register wrote down without " "a person's help, so no read goes unrecorded while it waits to be " "classified. Everything in a row here is a fact copied off the " "call: name, call site, and the compiled-in default when it is a " "literal. Nothing here is a judgement. A row leaves this " "section by hand, once somebody reads the call site and can say " "what the setting is for, whether a boolean is a gate or a knob, " "and whether a mirror operator would want it; then it moves to " "the section it belongs in with verified=True. Rows are not " "expected to stay here, and --reconcile counts every one of them " "as needing attention.", "vars": [ # --auto-register inserts new rows directly below this line. Leave the # marker in place; it is how the writer finds its way in. # AUTO-REGISTER INSERTION POINT ], } # --------------------------------------------------------------------------- # assembly # --------------------------------------------------------------------------- SECTIONS = [ ("Release gates", RELEASE_GATES), ("Mirror knobs", MIRROR_KNOBS), ("Database connections", DATABASE), ("hgcentral tables", CENTRAL_TABLES), ("Paths and caches", PATHS), ("Limits and load control", LIMITS), ("Abuse control", ABUSE), ("Logging and diagnostics", LOGGING), ("Branding and site text", BRANDING), ("Geographic mirroring", GEO), ("Login and wiki", LOGIN), ("External services", EXTERNAL), ("Retired", RETIRED), ("Runtime-built names", RUNTIME_NAMES), ("Awaiting review", AWAITING_REVIEW), ] AWAITING_TITLE = "Awaiting review" def build(): """The catalog as one structure.""" return {"boundary": BOUNDARY, "accessors": ACCESSORS, "profileSuffixes": PROFILE_SUFFIXES, "policy": {"keepAfterFlip": KEEP_AFTER_FLIP, "qaGrace": QA_GRACE}, "sections": [{"title": t, "what": s["what"], "vars": s["vars"]} for t, s in SECTIONS]} def all_vars(cat): for sec in cat["sections"]: for v in sec["vars"]: yield v def by_name(cat): out = {} for v in all_vars(cat): out.setdefault(v["name"], v) return out def gates(cat): return [v for v in all_vars(cat) if v.get("role") == "gate"] def knobs(cat): return [v for v in all_vars(cat) if v.get("role") == "knob"] def counts(cat): vs = list(all_vars(cat)) return { "sections": len(cat["sections"]), "vars": len(vs), "distinctNames": len({v["name"] for v in vs}), "gates": len(gates(cat)), "knobs": len(knobs(cat)), "public": len([v for v in vs if v["public"]]), "deprecated": len([v for v in vs if v.get("deprecated")]), "verified": len([v for v in vs if v["verified"]]), "unverified": len([v for v in vs if not v["verified"]]), "required": len([v for v in vs if v.get("required")]), "envOverridable": len([v for v in vs if v.get("env")]), } # --------------------------------------------------------------------------- # the harvester, for --reconcile and --sunset # --------------------------------------------------------------------------- def load_harvester(): """Import harvestHgConf from next door.""" sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) try: import harvestHgConf except ImportError: return None return harvestHgConf # --------------------------------------------------------------------------- # sunset report # --------------------------------------------------------------------------- def gate_lifecycle(cat, ages): """Join each gate with the version history. Returns a record per gate carrying what the tree knows (added, flipped, current default) and what the catalog decided (sunset). Everything the report says follows from this join, so a gate cannot be described as healthy just because nobody updated its entry. """ first = ages.get("first", {}) first_true = ages.get("firstTrue", {}) cur = ages.get("current") hh = load_harvester() out = [] for v in gates(cat): name = v["name"] added = (first.get(name) or {}).get("version") flipped = (first_true.get(name) or {}).get("version") tickets, ticket_from = ([], None) if hh and hasattr(hh, "ticket_for"): tickets, ticket_from = hh.ticket_for(name, ages) shipped = v.get("default") == "TRUE" # A flip date with a FALSE default now means the flip was reverted, so # the flag is back to gating and the flip date must not drive a # deadline. reverted = bool(flipped) and not shipped sunset = v.get("sunset") if sunset is None and shipped and flipped: sunset = flipped + KEEP_AFTER_FLIP out.append({ "name": name, "src": v["src"], "default": v.get("default"), "note": v.get("note"), "added": added, "flipped": flipped, "shipped": shipped, "reverted": reverted, "sunset": sunset, "current": cur, "age": (cur - added) if (cur and added) else None, "tickets": tickets, "ticketFrom": ticket_from, }) return out def gate_states(life, cur): """name -> "overdue" or "stalled", for the gates a release can act on. The two states the backlog file tracks. A gate in neither is scheduled, inside its QA grace period, or undatable, and none of those is news. The precedence matches sunset_report: an overdue gate is reported as overdue even if it would also qualify as stalled. """ states = {} for g in life: if g["sunset"] and cur and g["sunset"] <= cur: states[g["name"]] = "overdue" elif (not g["shipped"] and g["age"] is not None and g["age"] > QA_GRACE): states[g["name"]] = "stalled" return states def read_backlog(path=None): """The overdue and stalled gates already accounted for. Returns {name: (state, firstVersion)}. A missing file means nothing has been accepted yet, so every flagged gate reads as new; that is the right behaviour the first time this runs, and --update-baseline is how it stops. The default resolves at call time rather than at import, so a test can point BACKLOG_FILE somewhere else. """ path = path or BACKLOG_FILE known = {} try: with open(path) as f: for line in f: line = line.split("#", 1)[0].strip() if not line: continue f3 = line.split() if len(f3) < 2: continue ver = None if len(f3) > 2 and f3[2].lstrip("v").isdigit(): ver = int(f3[2].lstrip("v")) known[f3[0]] = (f3[1], ver) except OSError: pass return known def write_backlog(states, cur, life=None, path=None): """Write the backlog, keeping each gate's original first-reported version. A gate that changes state keeps the version it was first flagged at, since the useful number is how long it has been somebody's problem, not when it last changed shape. """ path = path or BACKLOG_FILE was = read_backlog(path) notes = {g["name"]: g for g in (life or [])} with open(path, "w") as f: f.write("""\ # hgConfGateBacklog.txt - the hg.conf release gates already known to be overdue # for deletion or stalled in QA. Refs #37925. # # hgConfCatalog.py --sunset-new reports only the gates that are not in here, so # this is what lets the weekly build's wrap-up say "two gates went overdue this # release" instead of reprinting the whole standing list every three weeks. # Regenerate with --update-baseline and read the diff before committing: a name # arriving is a new gate nobody sunset, and a name leaving means somebody # actually deleted a flag, which is the outcome the whole exercise is for. # # The first version of this file was accepted wholesale, as a snapshot of the # backlog on the day the report learned to tell new from standing. So a name # being in here is not evidence that anybody reviewed it; only the ones added # since, which arrive a few at a time in a reviewable diff, carry that weight. # # Fields: name, state (overdue or stalled), and the release it was first # reported at. hgConfCatalog.py --sunset prints the full list with dates. """) for name in sorted(states): first = was.get(name, (None, None))[1] if first is None: first = cur g = notes.get(name, {}) note = "" if g.get("flipped") and states[name] == "overdue": note = " # flipped v%s" % g["flipped"] elif g.get("added"): note = " # added v%s" % g["added"] f.write("%-26s %-8s %s%s\n" % (name, states[name], "v%s" % first if first else "?", note)) def sunset_delta(cat, ages, sites=None, out=sys.stdout): """Only what changed since the backlog was last accepted. The mode meant for the weekly build's wrap-up, and the reason the backlog file exists. --sunset is an 18-line standing list that reads as a worklist the first week and as wallpaper by the third; what a release can actually act on is the handful of gates that crossed a deadline while it was being built. Prints a one-line summary on every run, so silence means the step did not run rather than that nothing changed. Exits nonzero when there is something new, the same contract as --reconcile, and also when the age cache is stale, because a stale cache is exactly the condition under which a newly overdue gate would go unreported. """ cur = ages.get("current") life = gate_lifecycle(cat, ages) now = gate_states(life, cur) was = read_backlog() byname = {g["name"]: g for g in life} fresh = sorted(n for n in now if n not in was) moved = sorted(n for n in now if n in was and was[n][0] != now[n]) gone = sorted(n for n in was if n not in now) print("hg.conf release gates at v%s: %d newly overdue or stalled, " "%d resolved, %d in the standing backlog (%s)" % (cur, len(fresh) + len(moved), len(gone), len(now), os.path.basename(BACKLOG_FILE)), file=out) def describe(name): g = byname.get(name, {}) bits = [] if g.get("added"): bits.append("added v%d" % g["added"]) if g.get("flipped") and not g.get("reverted"): bits.append("flipped v%d" % g["flipped"]) if g.get("sunset"): bits.append("sunset v%d" % g["sunset"]) n = len((sites or {}).get(name, [])) or 0 if n: bits.append("%d call site%s" % (n, "" if n == 1 else "s")) tik = ", ".join("#%d" % t for t in g.get("tickets") or []) if tik: bits.append(tik) return " %-26s %s" % (name, ", ".join(bits)) if fresh: print("\nNEWLY FLAGGED (was fine at the last release, is not now):", file=out) for name in sorted(fresh, key=lambda n: now[n]): print(describe(name) + " [%s]" % now[name], file=out) if moved: print("\nCHANGED STATE:", file=out) for name in moved: print(describe(name) + " [%s -> %s]" % (was[name][0], now[name]), file=out) if gone: print("\nRESOLVED since the backlog was accepted:", file=out) for name in gone: n = len((sites or {}).get(name, [])) or 0 if n: # Still read, so it left the list by being reclassified, by # shipping, or by being given a later deadline, not by being # deleted. Worth naming either way: the flag is still there. print(" %-26s no longer flagged, but still read at %d call " "site%s" % (name, n, "" if n == 1 else "s"), file=out) else: print(" %-26s reads are gone from the tree; drop its catalog " "row too" % name, file=out) if fresh or moved or gone: print("\nFull list: hgConfCatalog.py --sunset\n" "Accept the new state: hgConfCatalog.py --update-baseline, then " "commit %s" % os.path.basename(BACKLOG_FILE), file=out) if ages.get("stale"): print("\nWARNING: the age cache was built at v%s and the tree is now " "at v%s, so a gate\nadded since v%s has no date and cannot be " "reported here at all. Rebuild with\nharvestHgConf.py --age " "--refresh and commit hgConfAges.json." % (ages.get("cachedAt"), cur, ages.get("cachedAt")), file=out) return 1 return 1 if (fresh or moved) else 0 def sunset_report(cat, ages, sites=None, out=sys.stdout): """What should be deleted, what has no deadline, what is stuck in QA.""" cur = ages.get("current") life = gate_lifecycle(cat, ages) known = read_backlog() print("current tree version: v%s" % cur, file=out) if ages.get("stale"): print("\nWARNING: the age cache was built at v%s and the tree is now " "at v%s.\nDeadlines below are still correct, but any flag added " "since v%s has no date\nand will show as 'age unknown' rather " "than being reported. Rebuild with\nharvestHgConf.py --age " "--refresh.\n" % (ages.get("cachedAt"), cur, ages.get("cachedAt")), file=out) undated = [g for g in life if g["added"] is None] if undated: print("\n%d gate(s) could not be dated from history, so no deadline " "applies to them:\n %s\n" % (len(undated), ", ".join(sorted(g["name"] for g in undated))), file=out) print("policy: keep a flag %d releases after its default flips TRUE; " "a gate\nstill defaulting FALSE after %d releases is stalled.\n" % (KEEP_AFTER_FLIP, QA_GRACE), file=out) noticket = [g["name"] for g in life if not g["tickets"]] if noticket: print("The ticket column is the ticket cited by the commit that added " "the flag, or\nby the commit that turned it on, marked (flip). " "%d of %d gates have neither\nand are blank: %s\n" % (len(noticket), len(life), ", ".join(sorted(noticket))), file=out) def line(g): bits = [] if g["added"]: bits.append("added v%d" % g["added"]) if g["flipped"] and not g["reverted"]: bits.append("flipped v%d" % g["flipped"]) if g["reverted"]: bits.append("flip v%d reverted" % g["flipped"]) if g["sunset"]: bits.append("sunset v%d" % g["sunset"]) n = len((sites or {}).get(g["name"], [])) or None tail = "%d call site%s" % (n, "" if n == 1 else "s") if n else "" tik = ", ".join("#%d" % t for t in g["tickets"]) if tik and g["ticketFrom"] == "flip": tik += " (flip)" # A gate in the backlog file has been reported before, so say since # when. Anything unmarked in the overdue or stalled lists crossed its # line during this release and is what --sunset-new would have printed. first = known.get(g["name"], (None, None))[1] if first: tail = (tail + " backlog since v%d" % first).strip() return " %-26s %-46s %-16s %s" % (g["name"], ", ".join(bits), tik, tail) overdue = sorted([g for g in life if g["sunset"] and cur and g["sunset"] <= cur], key=lambda g: g["sunset"]) print("OVERDUE (delete the flag and every branch that reads it): %d" % len(overdue), file=out) for g in overdue: print(line(g), file=out) # Deleting the flag is only half of it. A gate that shipped was turned on # by hand on some machine first, and that line has to go too, or the next # person to read the conf file cannot tell a live switch from litter. # Counted over the gates listed above only, so the number belongs where it # is printed; --redundant covers every shipped gate. names = {g["name"] for g in overdue} stale = [r for r in redundant_settings(cat, ages)["redundant"] if r["name"] in names] if stale: print("\n %d of the gates above are still set by hand in an hg.conf " "readable from here\n (%d line%s). hgConfCatalog.py " "--redundant names them." % (len({r["name"] for r in stale}), len(stale), "" if len(stale) == 1 else "s"), file=out) due = sorted([g for g in life if g["sunset"] and cur and g["sunset"] > cur], key=lambda g: g["sunset"]) print("\nSCHEDULED (shipped, deadline not yet reached): %d" % len(due), file=out) for g in due: print(line(g), file=out) stalled = sorted([g for g in life if not g["shipped"] and g["age"] is not None and g["age"] > QA_GRACE], key=lambda g: -g["age"]) print("\nSTALLED IN QA (default still FALSE %d+ releases after it landed): " "%d" % (QA_GRACE, len(stalled)), file=out) print(" Either turn these on or delete the feature. A flag nobody " "flips is not\n gating a release, it is hiding dead code.", file=out) for g in stalled: print(line(g) + " %d releases old" % g["age"], file=out) fresh = [g for g in life if not g["shipped"] and (g["age"] is None or g["age"] <= QA_GRACE)] print("\nIN QA (recent, inside the grace period): %d" % len(fresh), file=out) for g in sorted(fresh, key=lambda g: -(g["age"] or 0)): print(line(g) + (" %d releases old" % g["age"] if g["age"] is not None else " age unknown"), file=out) nodate = [g for g in life if g["sunset"] is None and g["shipped"]] if nodate: print("\nNO DEADLINE (shipped but the flip version is unknown, so no " "deadline\ncould be computed; needs a sunset= in the catalog): " "%d" % len(nodate), file=out) for g in nodate: print(line(g), file=out) return len(overdue) # --------------------------------------------------------------------------- # settings a machine sets by hand and no longer needs to # --------------------------------------------------------------------------- # A gate is turned on per machine by hand while the feature is in QA, by # appending one line to that machine's hg.conf. Nothing goes back to remove # the line when the default in the tree flips TRUE, so the line outlives the # reason for it, and while it is there the flag reads on locally whatever the # tree says. That is the mechanism by which a wrong default survives: the # people most likely to notice are on the one machine that cannot see it. # # These are the conf files readable from hgwdev. hgwbeta, the RR and every # mirror keep their own, none of which can be read from here, so the report # covers what it can see and says what it cannot. Those are QA's to clean up. MACHINE_CONFS = [ ("/usr/local/apache/cgi-bin/hg.conf", "shared dev conf, genome-test reads it"), ("/usr/local/apache/cgi-bin-beta/hg.conf", "the beta CGI install on this host"), ] # The six words cfgOptionBooleanDefault accepts, lowercase only, from # hg/lib/hgConfig.c. Anything else there is not a falsy value, it is an # errAbort in every CGI that reads the setting, which is worth its own line in # the report rather than being quietly skipped. TRUE_WORDS = ("yes", "on", "true") FALSE_WORDS = ("no", "off", "false") def machine_confs(): """The conf files to read, as (path, label), skipping what is not there. HGCONF_MACHINE_CONFS overrides the list, colon separated, which is how a test points this at a fixture rather than at the live configuration. The invoking user's own sandbox conf comes last: a stale line there costs nobody but them, and it is the one file they can fix without asking. """ env = os.environ.get("HGCONF_MACHINE_CONFS") if env: pairs = [(p, "from HGCONF_MACHINE_CONFS") for p in env.split(":") if p] else: pairs = list(MACHINE_CONFS) user = os.environ.get("USER") if user: pairs.append(("/usr/local/apache/cgi-bin-%s/hg.conf" % user, "your own sandbox")) return [(p, why) for p, why in pairs if os.access(p, os.R_OK)] def parse_conf(path): """What one hg.conf sets itself, as a list of (line number, name, value). Follows the same rules as parseConfigFile in hg/lib/hgConfig.c and no others. A "#" comments a line only at the start of it, so a value may contain one and splitting on it would silently truncate a password. `include` and `delete` lines are not settings. It deliberately does not follow `include`: this report tells somebody to delete a line out of a named file, so what the file it includes sets is not this file's business. A name set on two lines is kept twice, because both lines are equally deletable and hiding one would send the reader back. """ out = [] try: with open(path) as f: for n, line in enumerate(f, 1): line = line.strip() if not line or line.startswith("#"): continue if re.match(r"(include|delete)\s", line): continue if "=" not in line: continue name, val = line.split("=", 1) out.append((n, name.strip(), val.strip())) except (IOError, OSError): return [] return out def redundant_settings(cat, ages=None): """Gate settings on this machine that the tree's own default now covers. Three populations, and the difference between them is the whole point: redundant the catalog says the default is TRUE and the conf turns it on. The line does nothing and can go. optout the conf turns it off. That is a live decision, not litter, and telling somebody to delete it would turn a feature on behind their back. unparsable the value is not one of the six words hg.conf accepts, so every CGI reading that setting aborts on this machine. Only gates are considered. A knob is a switch a machine is entitled to set forever, so setting one to its default value is redundant in the letter and not in the spirit, and nagging about it would be the crying wolf the gate/knob split exists to prevent. """ flipped = (ages or {}).get("firstTrue", {}) by = {v["name"]: v for v in gates(cat)} found = {"redundant": [], "optout": [], "unparsable": [], "files": []} for path, why in machine_confs(): found["files"].append((path, why)) for lineno, name, val in parse_conf(path): v = by.get(name) if v is None or v.get("default") != "TRUE": continue rec = {"name": name, "path": path, "line": lineno, "value": val, "flipped": (flipped.get(name) or {}).get("version")} if val in TRUE_WORDS: found["redundant"].append(rec) elif val in FALSE_WORDS: found["optout"].append(rec) else: # Includes the right word in the wrong case: hgConfig.c # compares with sameString, so "On" is not "on" and the CGI # aborts on it like any other unrecognized value. found["unparsable"].append(rec) return found def redundant_report(cat, ages=None, verbose=False, out=sys.stdout): """Print what --redundant found. Returns the number of lines to delete. Silent and 0 when there is nothing to delete, so a cron can run it beside --reconcile and mail only what needs a person. --verbose prints the whole picture, including the files read and the opt-outs, which is what somebody working the list wants and what a nightly mail does not. """ found = redundant_settings(cat, ages) if verbose: if not found["files"]: print("no hg.conf readable on this machine, so nothing to check. " "Set\nHGCONF_MACHINE_CONFS to point somewhere else.", file=out) return 0 print("hg.conf files read here: %d" % len(found["files"]), file=out) for path, why in found["files"]: print(" %-46s %s" % (path, why), file=out) print("", file=out) order = {path: i for i, (path, _) in enumerate(found["files"])} def where_key(r): return (order.get(r["path"], len(order)), r["line"]) def line(r): where = "%s:%d" % (r["path"], r["line"]) when = ("default TRUE since v%d" % r["flipped"] if r["flipped"] else "flip version unknown, refresh the age cache") return " %-26s %-8s %-52s %s" % (r["name"], r["value"], where, when) n = len(found["redundant"]) if n: print("REDUNDANT (the tree defaults these on; the line can go): %d" % n, file=out) for r in sorted(found["redundant"], key=where_key): print(line(r), file=out) print("\n Delete a line only after the release that flipped the " "default is installed\n on the machine that reads the file. " "Until then the line is what is turning\n the feature on there, " "and removing it turns the feature off.\n" "\n hgwbeta, the RR and the mirrors keep their own hg.conf and " "none of them can\n be read from hgwdev, so a line may be " "waiting there too. Those are QA's.", file=out) if found["unparsable"]: print("\nNOT A BOOLEAN (%d): hg.conf takes yes, no, on, off, true or " "false, lowercase.\nEvery CGI that reads one of these aborts on " "that machine:" % len(found["unparsable"]), file=out) for r in sorted(found["unparsable"], key=where_key): print(line(r), file=out) if verbose and not n and not found["unparsable"]: print("nothing to delete: no gate defaulting TRUE is set by hand in " "the conf files\nreadable here.", file=out) if verbose and found["optout"]: print("\nDELIBERATE (%d): the default is on and this machine turns it " "off. Leave these\nalone; the flag exists so a machine can do " "exactly this:" % len(found["optout"]), file=out) for r in sorted(found["optout"], key=where_key): print(line(r), file=out) return n + len(found["unparsable"]) # --------------------------------------------------------------------------- # checks # --------------------------------------------------------------------------- def check(cat, out=sys.stderr): """Internal consistency. Returns the number of problems found.""" problems = 0 c = counts(cat) print("=== counts ===", file=out) for k in sorted(c): print("%-16s %s" % (k, c[k]), file=out) print("\n=== consistency ===", file=out) seen = {} for sec in cat["sections"]: for v in sec["vars"]: seen.setdefault(v["name"], []).append(sec["title"]) dupes = {n: s for n, s in seen.items() if len(s) > 1} if dupes: print("names in more than one section (%d):" % len(dupes), file=out) for n, s in sorted(dupes.items()): print(" %-34s %s" % (n, ", ".join(s)), file=out) for v in all_vars(cat): if v.get("role") == "gate" and v.get("kind") != "flag": print("gate not of kind flag: %s" % v["name"], file=out) problems += 1 if v.get("sunset") and not v.get("role") == "gate": print("sunset on a non-gate: %s" % v["name"], file=out) problems += 1 unver = [v["name"] for v in all_vars(cat) if not v["verified"]] if unver: print("\nunverified rows (%d): classification not confirmed against " "the code" % len(unver), file=out) for n in sorted(unver): print(" %s" % n, file=out) arguable = [v for v in all_vars(cat) if v.get("debatable")] if arguable: print("\n=== gate or knob: the calls worth arguing about (%d) ===" % len(arguable), file=out) print("Everything else in the split is either obvious or was confirmed " "at its\ncall site. These are the ones a second opinion should " "settle, because\nfiling a gate as a knob hides it from the " "sunset report forever.\n", file=out) for v in sorted(arguable, key=lambda x: x["name"].lower()): print(" %s (currently: %s, default %s)" % (v["name"], v.get("role"), v.get("default")), file=out) for line in wrap_text(v["debatable"], 72): print(" %s" % line, file=out) print(file=out) print("problems: %d" % problems, file=out) return problems def wrap_text(s, width): """Wrap without pulling in textwrap for one caller.""" words = s.split() lines, cur = [], "" for w in words: if cur and len(cur) + 1 + len(w) > width: lines.append(cur) cur = w else: cur = (cur + " " + w).strip() if cur: lines.append(cur) return lines def cited_at(name, row, sites=None): """The row's src, with a line number resolved from the tree if there is one. src names a file and not a line on purpose: a line number is derived data, and storing derived data by hand means every edit above a read turns into a citation to repair. That cost was real. Between 2026-08-07 and 2026-08-28 the nightly register pushed sixteen commits to master and thirteen of them changed nothing but line numbers, because one insertion near the top of hgTracks.c moves fifteen of them at once. A reader still wants the exact line, so it is computed here at print time from the harvester, which finds every read independently anyway. With no harvester result, or a read the scan cannot see (a name reached through a helper, a prefix family), the file alone is what there is to say. """ src = row.get("src") or "" if not sites or ":" in src or name.startswith("{"): return src here = sorted(s for s in sites.get(name, ()) if s.rpartition(":")[0] == src) if not here: return src if len(here) == 1: return here[0] # Several reads in the one file. Name the first and say how many, rather # than pick one and imply it is the read that matters. return "%s (%d reads)" % (here[0], len(here)) def moved_citations(cat, sites): """Rows whose src file no longer contains any read of that setting. src is the one field here that is a pointer rather than a claim, and it is what --json, --html and the sunset report print next to a gate, so a reader who follows it lands on whatever code happens to occupy that place now. Since src carries no line number, drift within a file is not a finding and never needs a commit. What is left is the case that always needed a person: the read left the file, and somebody has to decide which of its call sites the row should point at. Names the harvester has no site for at all are not judged: a read reached through a helper (db.grp goes through loadGrps) or a prefix family has no single literal line to check against. Nor is a row whose src is not source at all: the three customTracks profile members cite product/ex.hg.conf, because documentation is the only place they appear. """ moved = [] for name, row in sorted(by_name(cat).items()): src = row.get("src") or "" if not src.endswith((".c", ".h")) or name.startswith("{"): continue seen = sites.get(name) if not seen: continue if any(s.rpartition(":")[0] == src for s in seen): continue moved.append((name, src, sorted(seen))) return moved def awaiting_review(cat): """Rows the writer put in the holding pen, still unclassified.""" for sec in cat["sections"]: if sec["title"] == AWAITING_TITLE: return list(sec["vars"]) return [] def introducing_commit(hh, name, path): """The oldest commit whose diff touched this setting name in this file. Attribution is worth having automatically because it is the one thing a reviewer wants first and the slowest to dig up by hand: which change brought the setting in, and under which ticket. The age cache carries this for every name it has seen, but a setting added since the last --refresh is missing from it, and a name added yesterday is exactly the case this writer exists to serve. One pickaxe walk over a single file is seconds, against the four minutes a full refresh costs. """ out = hh.git("log", "--reverse", "--format=%h\t%s", "-S", name, "--", path) for line in out.splitlines(): if "\t" not in line: continue commit, _, subject = line.partition("\t") tickets = re.findall(r"#(\d{4,6})", subject) return commit.strip(), subject.strip(), tickets return None, None, [] def wrap_note(note, width): """Wrap a note the way the hand-written ones are spaced. wrap_text would do, except that it rejoins on single spaces and every note in this file puts two after a full stop. A machine-written row sitting next to hand-written ones should not be spottable by its spacing. """ lines, cur = [], "" for w in note.split(): sep = "" if not cur else (" " if cur.endswith(".") else " ") if cur and len(cur) + len(sep) + len(w) > width: lines.append(cur) cur = w else: cur = cur + sep + w if cur: lines.append(cur) return lines def pyq(s): """Escape text for splicing into a double-quoted Python string literal. render_row writes source into this very file, and --auto-register runs it unattended from cron over text nobody vetted: a git commit subject that quotes a setting name would otherwise land a bare " in a generated row and break every import of the catalog, with no human in the loop to notice. Newlines and tabs go too, since a row has to stay on the lines it was wrapped onto. """ s = re.sub(r"[\x00-\x1f\x7f]", " ", str(s)) return s.replace("\\", "\\\\").replace('"', '\\"') def render_row(name, kind, src, default=None, ticket=None, note=None): """One h(...) call as source text, wrapped the way the file is written.""" head = ' h("%s", "%s", "%s"' % (pyq(name), pyq(kind), pyq(src)) args = [] if default is not None: args.append('default="%s"' % pyq(default)) if ticket: args.append('ticket="%s"' % pyq(ticket)) lines = [] cur = head for a in args: if len(cur) + 2 + len(a) > 78: lines.append(cur + ",") cur = " " + a else: cur = cur + ", " + a if note: # Wrap on the text as read, escape each piece after: the escapes are # not what a reader of the file sees, so they should not push a note # onto an extra line. wrapped = [pyq(w) for w in wrap_note(note, 61)] if len(cur) + 2 > 78: lines.append(cur + ",") cur = " " else: cur = cur + ",\n " cur += 'note="%s "' % wrapped[0] if len(wrapped) > 1 \ else 'note="%s"' % wrapped[0] lines.append(cur) for i, w in enumerate(wrapped[1:], start=1): last = i == len(wrapped) - 1 lines.append(' "%s%s"' % (w, "" if last else " ")) lines[-1] += ")," else: lines.append(cur + "),") return "\n".join(lines) + "\n" def auto_register(cat, hh, found, out=sys.stdout): """Write a row for every setting the tree reads and the catalog lacks. The developer who adds an hg.conf read should not have to know this catalog exists, and asking them to has not worked: hubSpaceLockTimeout arrived as one supporting line inside an unrelated bugfix and nothing caught it until somebody ran --reconcile by hand a day later. So the facts get recorded here instead of being requested from them. What goes in is only what the call itself says: the name, the site, the compiled-in default when it is a literal, and the commit and ticket that introduced it. What stays out is every judgement. role= in particular is left unset on purpose, and that is the whole design: guessing "knob" would exempt a real release gate from the sunset report forever, which is the exact failure this catalog exists to prevent, and guessing "gate" would feed fiction into the overdue arithmetic. verified= stays false because no person has read the call site. So this buys silence for the developer, not for us: every row it writes is counted by --reconcile as needing attention until somebody classifies it and moves it out of the holding pen. """ tree = hh.by_name(found) cataloged = by_name(cat) pending = {v["name"] for v in awaiting_review(cat)} new = sorted(n for n in tree if n not in cataloged and n not in pending) if not new: print("no unregistered setting to write down", file=out) return 0 # The default is only worth copying when the tree states it outright. An # identifier (hubSpaceLockTimeout's is HUB_LOCK_TIMEOUT_DEFAULT) says # nothing at this level, and reconcile's default check compares literals # only, so recording the identifier would just look like a fact. literal = re.compile(r'^(TRUE|FALSE|-?\d+|".*")$') reads = {} for rec in hh.all_reads(found): reads.setdefault(rec["name"], []).append(rec) path = os.path.abspath(__file__) with open(path) as f: txt = f.read() marker = " # AUTO-REGISTER INSERTION POINT\n" if marker not in txt: print("the AUTO-REGISTER INSERTION POINT marker is gone from %s; " "cannot write rows" % os.path.basename(path), file=out) return 0 block = "" for name in new: recs = reads.get(name, []) # The row cites the file only. A line number written here would be # stale by the next edit above the read, and repairing it would put # this cron back to committing line numbers every night. site = sorted(tree[name]["sites"])[0] src = site.rpartition(":")[0] boolean = tree[name].get("boolean") kind = "flag" if boolean else "unreviewed" default = None raw = recs[0].get("default") if recs else None if raw and literal.match(str(raw)): default = str(raw).strip('"') commit, subject, tickets = introducing_commit(hh, name, src) note = ("Written down by --auto-register, not yet reviewed by a " "person. Read with %s in %s." % (recs[0]["func"] if recs else "an accessor", src)) if len(recs) > 1: note += " %d call sites." % len(recs) if raw and default is None: note += (" The compiled-in default is %s, an identifier the scan " "cannot resolve." % raw) if commit: note += " Came in at %s, %s." % (commit, subject.rstrip(".")) # For a boolean the kind is not in question: the call says it is a # flag. What is missing is the gate-or-knob call. For anything else # the kind is the placeholder above and has to be chosen too. note += (" Needs a description and a gate or knob call." if boolean else " Needs a kind, a description, and a section to live in.") block += render_row(name, kind, src, default=default, ticket=tickets[0] if tickets else None, note=note) print(" %-40s %s%s" % (name, site, " (%s)" % tickets[0] if tickets else ""), file=out) txt = txt.replace(marker, marker + block, 1) with open(path, "w") as f: f.write(txt) print("wrote %d row%s to the '%s' section of %s; classify %s and move %s " "out" % (len(new), "" if len(new) == 1 else "s", AWAITING_TITLE, os.path.basename(path), "it" if len(new) == 1 else "them", "it" if len(new) == 1 else "them"), file=out) return len(new) def reconcile(cat, out=sys.stdout, verbose=False): """Diff the catalog against what the tree actually reads. Five questions: 1. does the catalog list something the tree no longer reads 2. does the tree read something the catalog has not classified 3. is every boolean flag in the tree classified gate or knob 4. does each flag's hand-written default still match the tree's 5. does each row's src still name a file that reads it Questions 3 and 4 are the ones that keep the sunset report honest: an unclassified flag is one nobody has decided the fate of, and a wrong default puts a shipped gate in the stalled list or the reverse. Only 2, 3, 4 and 5 count as problems, because only those mean somebody changed a setting and did not write it down, which is the one thing a person has to act on. The rest is drift that has been there for years (documentation for a feature that was deleted, a read that moved into a helper), and printing it on every run is what would turn a nightly cron into mail nobody reads. So it goes out only under --verbose. Silent and 0 means nothing new. 5 fires only when a read leaves its file entirely, which is a judgement about which call site to cite. Drift within a file is not a question at all, because src carries no line number to drift. """ hh = load_harvester() if hh is None: print("harvestHgConf.py not importable; cannot reconcile", file=out) return 1 found, _ = hh.harvest() tree = hh.by_name(found) sites = {n: d["sites"] for n, d in tree.items()} cataloged = by_name(cat) problems = 0 # A scan that finds almost nothing is a broken scan, not a clean tree, and # the difference matters: pointed at the wrong KENT_SRC or an empty clone, # everything below would come up empty and report all clear forever. The # real number is in the hundreds. if len(tree) < MIN_TREE_NAMES: print("only %d settings found under %s: expected at least %d, so the " "scan is\nbroken rather than the tree being clean. Check " "KENT_SRC." % (len(tree), hh.ROOT, MIN_TREE_NAMES), file=out) return 1 # Three kinds of name legitimately have no literal read to point at, and # all three have to be excused or the report is nothing but false alarms: # profile members (read through cfgOption2 with a runtime prefix), prefix # families (enumerated with cfgNamesWithPrefix), and the members of such a # family as spelled out in the example configs. suffixes = set(PROFILE_SUFFIXES["suffixes"]) prefix_scans = set(found["prefixScans"]) def is_profile_member(name): return "." in name and name.rsplit(".", 1)[1] in suffixes def is_prefix_family(name): if name in prefix_scans or name.endswith("."): return True return any(name.startswith(p) for p in prefix_scans) if verbose: print("=== catalog vs tree ===", file=out) missing = sorted(n for n in tree if n not in cataloged and not n.startswith("{")) # {ident} names are carried in the catalog with the braces, so match those # separately. missing += sorted(n for n in tree if n.startswith("{") and n not in cataloged) if missing: problems += len(missing) print("\nread by the tree, not in the catalog (%d):" % len(missing), file=out) for n in missing: print(" %-40s %s" % (n, sorted(tree[n]["sites"])[0]), file=out) stale = sorted(n for n in cataloged if n not in tree and not is_profile_member(n) and not is_prefix_family(n)) if stale and verbose: print("\nin the catalog, no literal read found (%d):" % len(stale), file=out) print(" (a prefix family or a profile member is expected here; " "anything else\n is a catalog row whose read has gone away)", file=out) for n in stale: print(" %-40s %s" % (n, cited_at(n, cataloged[n], sites)), file=out) if verbose: print("\n=== boolean flags: every one must be a gate or a knob ===", file=out) tree_flags = {n for n, d in tree.items() if d["boolean"]} classified = {v["name"] for v in gates(cat)} | {v["name"] for v in knobs(cat)} pending = {v["name"] for v in awaiting_review(cat)} # A flag the writer put in the holding pen is unclassified too, but it is # already written down and reported under its own heading below, so saying # it twice under two different instructions would just be confusing. unclassified = sorted(tree_flags - classified - pending) if unclassified: problems += len(unclassified) print("\nboolean flag classified neither gate nor knob (%d): nobody " "has decided\nwhether these are temporary:" % len(unclassified), file=out) for n in unclassified: print(" %-40s %s" % (n, sorted(tree[n]["sites"])[0]), file=out) elif verbose: print("all %d boolean flags in the tree are classified" % len(tree_flags), file=out) # Does the catalog's default= still match the tree's? # # This field is hand-written while everything else about a gate's lifecycle # is read out of git, and until this check nothing compared the two. It # matters more than it looks: gate_lifecycle decides a gate has shipped # from this field alone, so a flag whose compiled-in default was flipped # TRUE without the row being updated sits in the sunset report's stalled # list forever, under a heading telling somebody to turn on a flag that is # already on. A stale age cache is announced; this was not. # # Only reads whose default is the literal TRUE or FALSE can be compared. # Where the default is an identifier (autoBlatBigPsl's is the file-scope # autoBigPsl) the tree does not say what it is at this level, so the row's # default stands unchallenged and the note has to carry the argument. bool_defaults = {} for rec in hh.all_reads(found): if rec.get("boolean") and rec.get("default") in ("TRUE", "FALSE"): bool_defaults.setdefault(rec["name"], {}).setdefault( rec["default"], []).append(rec["src"]) mismatched = [] split = [] for n in sorted(bool_defaults): seen = bool_defaults[n] if len(seen) > 1: split.append(n) continue row = cataloged.get(n) if row is None or row.get("default") not in ("TRUE", "FALSE"): continue treeval = next(iter(seen)) if row["default"] != treeval: mismatched.append((n, row["default"], treeval, seen[treeval][0])) if mismatched: problems += len(mismatched) print("\nthe catalog's default disagrees with the tree (%d): the " "sunset report reads\nthis field to decide whether a gate has " "shipped, so it has to be right:" % len(mismatched), file=out) for n, catval, treeval, site in mismatched: print(" %-40s catalog %s, tree %s at %s" % (n, catval, treeval, site), file=out) if split and verbose: # Two reads of one flag with opposite compiled-in defaults. Not a # catalog error, so not a problem here: on a machine that does not set # the flag the two halves of the feature disagree, and only whoever # owns that code can say which default was meant. print("\nread with both TRUE and FALSE as the compiled-in default " "(%d):" % len(split), file=out) for n in split: print(" %s" % n, file=out) for val in sorted(bool_defaults[n]): for site in sorted(bool_defaults[n][val]): print(" %-8s %s" % (val, site), file=out) # The holding pen. These are counted as problems on purpose: the writer # buys the developer silence, not us, and if nothing ever pressed for the # classification we would have traded a loud failure for a quiet pile. waiting = awaiting_review(cat) if waiting: problems += len(waiting) print("\nwritten down but not classified (%d): --auto-register " "recorded the read;\nsomebody has to read the call site, say " "what it is for, and move the row\ninto the section it belongs " "in with verified=True:" % len(waiting), file=out) for v in waiting: print(" %-40s %s" % (v["name"], cited_at(v["name"], v, sites)), file=out) # Does each row's src still name a file that reads it? if verbose: print("\n=== source citations ===", file=out) moved = moved_citations(cat, sites) if moved: problems += len(moved) print("\nthe read named in src has left that file (%d): pick which " "call site\nthe row should cite:" % len(moved), file=out) for name, src, sites in moved: print(" %-40s %s, now read at %s" % (name, src, ", ".join(sites[:3])), file=out) elif verbose: print("every src names a file that reads its setting", file=out) phantom = sorted(classified - tree_flags) if phantom and verbose: print("\nclassified as a flag but not read with " "cfgOptionBooleanDefault (%d):" % len(phantom), file=out) for n in phantom: print(" %-40s %s" % (n, cited_at(n, cataloged[n], sites)), file=out) if verbose: print("\n=== product/ex.hg.conf ===", file=out) docs = hh.parse_doc_files() pub = {v["name"] for v in all_vars(cat) if v["public"]} # A prefix family counts as documented if any member of it is. def documented(name): if name in docs: return True return name.endswith(".") and any(d.startswith(name) for d in docs) undocumented = sorted(n for n in pub if not documented(n)) if undocumented and verbose: print("marked public in the catalog, absent from the example configs " "(%d):" % len(undocumented), file=out) print(" (these are settings a mirror operator would want and has no " "way to\n discover)", file=out) for n in undocumented: print(" %-40s %s" % (n, cited_at(n, cataloged[n], sites)), file=out) only_docs = sorted(n for n in docs if n not in cataloged and not is_profile_member(n) and not is_prefix_family(n)) if only_docs and verbose: print("\nin the example configs, nothing in the tree reads them (%d):" % len(only_docs), file=out) print(" (either the feature was deleted and the documentation was " "not, or the\n documented spelling is wrong, in which case a " "mirror that sets it is\n silently ignored)", file=out) for n in only_docs: print(" %-40s %s" % (n, docs[n]["sites"][0]), file=out) if problems and problems == len(waiting): # Nothing here is the developer's to do: the reads are all written # down, and what is left is the judgement --auto-register refuses to # fake. Telling somebody to add a row in the same commit as the code # would be the wrong instruction, and would be aimed at the wrong # person besides. print("\nproblems: %d, all of them rows waiting to be classified. " "Read each call site,\nfill in the kind, the note and (for a " "boolean) role=gate or role=knob, set\nverified=True, and move " "the row into the section it belongs in." % problems, file=out) elif problems: print("\nproblems: %d. Add or correct the row in hgConfCatalog.py for " "each. A name the\ntree reads and this catalog does not is " "normally written down by nightlyRegister.sh\nwithout anybody " "asking, so one showing up here means either the run has not " "come\nround yet or the name is invisible to the harvester and " "has to be added by hand." % problems, file=out) elif verbose: print("\nproblems: 0", file=out) return problems # --------------------------------------------------------------------------- # HTML # --------------------------------------------------------------------------- CSS = """ body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif; margin: 0 auto; max-width: 1180px; padding: 1em 2em; color: #222; line-height: 1.45; } h1 { font-size: 1.6em; margin-bottom: 0.2em; } h2 { font-size: 1.2em; margin-top: 1.8em; border-bottom: 2px solid #4b6c9e; padding-bottom: 0.2em; color: #1a3a6b; } h3 { font-size: 1.05em; margin-top: 1.4em; color: #1a3a6b; } p.what { color: #444; margin: 0.4em 0 1em 0; } table { border-collapse: collapse; width: 100%; font-size: 0.87em; margin-bottom: 1.2em; } th { background: #eaf0f8; text-align: left; padding: 5px 7px; border-bottom: 2px solid #4b6c9e; font-weight: 600; } td { padding: 5px 7px; border-bottom: 1px solid #dde3ec; vertical-align: top; } tr:hover td { background: #f6f9fd; } code { font-family: Menlo, Consolas, monospace; font-size: 0.94em; } td.name code { font-weight: 600; } td.src { color: #666; font-size: 0.9em; white-space: nowrap; } span.kind { display: inline-block; padding: 1px 6px; border-radius: 3px; font-size: 0.82em; background: #e8e8e8; color: #444; } span.gate { background: #fde9c8; color: #7a4a00; } span.knob { background: #d9ead9; color: #24541f; } span.overdue { background: #f8d3d3; color: #8a1f1f; font-weight: 600; } span.stalled { background: #fdf0c0; color: #6b5300; font-weight: 600; } span.dep { background: #eee; color: #777; } span.req { background: #dce6f8; color: #1a3a6b; } div.note { color: #555; margin-top: 3px; font-size: 0.95em; } div.arguable { color: #6b4a00; background: #fdf6e3; border-left: 3px solid #d9a441; padding: 3px 7px; margin-top: 4px; font-size: 0.93em; } span.arguable { background: #fdf0c0; color: #6b5300; } div.box { background: #f6f8fb; border-left: 4px solid #4b6c9e; padding: 0.7em 1em; margin: 1em 0; } div.policy { background: #fff8ec; border-left: 4px solid #d9a441; padding: 0.7em 1em; margin: 1em 0; } ul.toc { columns: 3; list-style: none; padding-left: 0; font-size: 0.92em; } """ def esc(s): return html.escape(str(s), quote=False) def ticket_map(cat, ages): """name -> (tickets, kind) for every setting git can attribute. kind is "introduced" when the commit that added the read cited the ticket and "flip" when only the commit that turned a flag on did. A setting missing from this map has no ticket in either commit, which for anything added before about v270 means the tree predates Redmine. """ hh = load_harvester() if not (ages and hh and hasattr(hh, "ticket_for")): return {} out = {} for v in all_vars(cat): tickets, kind = hh.ticket_for(v["name"], ages) if tickets: out[v["name"]] = (tickets, kind) return out def ticket_links(rec): tickets, kind = rec links = " ".join('#%d' % (REDMINE % t, t) for t in tickets) if kind == "flip": return "turned on by %s" % links return "introduced by %s" % links def var_rows(vs, life_by_name=None, tickets=None, sites=None): rows = [] for v in sorted(vs, key=lambda x: x["name"].lower()): tags = ['%s' % esc(v["kind"])] role = v.get("role") if role: tags.append('%s' % (role, role)) if v.get("required"): tags.append('required') if v.get("deprecated"): tags.append('retired') life = (life_by_name or {}).get(v["name"]) if life: cur = life.get("current") if life.get("sunset") and cur and life["sunset"] <= cur: tags.append('overdue v%d' % life["sunset"]) elif life.get("sunset"): tags.append('sunset v%d' % life["sunset"]) if (not life.get("shipped") and life.get("age") is not None and life["age"] > QA_GRACE): tags.append('stalled %d' % life["age"]) extra = "" if life and life.get("added"): extra = "added v%d" % life["added"] if life.get("flipped"): extra += ", flipped v%d" % life["flipped"] tik = (tickets or {}).get(v["name"]) if tik: extra += (", " if extra else "") + ticket_links(tik) note = "" if v.get("note"): note = '
%s
' % esc(v["note"]) if v.get("debatable"): tags.append('gate or knob?') note += ('
Arguable: %s
' % esc(v["debatable"])) env = "" if v.get("env"): env = '
environment: %s
' \ % esc(v["env"]) default = esc(v.get("default") or "") rows.append( "%s%s%s" "%s%s" "%s%s" % (esc(v["name"]), note, env, " ".join(tags), default, esc(cited_at(v["name"], v, sites)), ("
%s
" % extra) if extra else "")) return "\n".join(rows) def table_of(vs, life_by_name=None, tickets=None, sites=None): return ("" "\n%s\n
settingkinddefaultread at
" % var_rows(vs, life_by_name, tickets, sites)) def render_html(cat, ages=None, sites=None): life_by_name = {} sunset_html = "" tickets = ticket_map(cat, ages) if ages: life = gate_lifecycle(cat, ages) life_by_name = {g["name"]: g for g in life} cur = ages.get("current") overdue = [g for g in life if g["sunset"] and cur and g["sunset"] <= cur] stalled = [g for g in life if not g["shipped"] and g["age"] is not None and g["age"] > QA_GRACE] sunset_html = ( '
Sunset status at v%s. ' '%d shipped gates are past their removal deadline and %d have been ' 'sitting at a FALSE default for more than %d releases. ' 'Policy: keep a flag %d releases after its default flips TRUE, ' 'then delete it and every branch that reads it. ' 'hgConfCatalog.py --sunset prints the working list.' '
' % (cur, len(overdue), len(stalled), QA_GRACE, KEEP_AFTER_FLIP)) c = counts(cat) parts = ["

Genome Browser hg.conf variables

", "

%d settings the CGIs read from hg.conf, " "generated from hg/utils/hgConfCatalog/. " "%d are release gates and %d are permanent deployment knobs." "

" % (c["distinctNames"], c["gates"], c["knobs"]), '
%s
' % esc(cat["boundary"]), sunset_html] if tickets: intro = len([1 for _, k in tickets.values() if k == "introduced"]) parts.append( '
Where a setting came from. Each entry ' 'below carries the Redmine ticket cited by the commit that added ' 'the read, or for a flag the commit that turned its default on, ' 'marked there as "turned on by". %d of %d settings are attributed ' 'that way. The rest are blank on purpose: no commit in the chain ' 'names a ticket, and for anything added before about v270 the tree ' 'predates our use of Redmine, so there is nothing to find. Nothing ' 'here is inferred from a later commit that merely edited the line, ' 'which would have credited half the file to whatever refactor ' 'touched it last. harvestHgConf.py --tickets lists ' 'the unattributed settings, separating the ones old enough to be ' 'hopeless from the ones somebody could still fill in.
' % (intro, c["distinctNames"])) parts.append("

Contents

    ") for sec in cat["sections"]: parts.append("
  • %s
    • " % (esc(sec["title"].replace(" ", "-")), esc(sec["title"]))) parts.append("
") parts.append("

How a setting is read

") parts.append("") for fn, what in sorted(ACCESSORS.items()): parts.append("" % (esc(fn), esc(what))) parts.append("
accessorbehaviour
%s%s
") ps = cat["profileSuffixes"] parts.append("

Database profile suffixes

") parts.append("

%s

" % esc(ps["what"])) parts.append("

Suffixes: %s

" % ", ".join("%s" % esc(s) for s in ps["suffixes"])) parts.append("

Profiles in use: %s

" % ", ".join("%s." % esc(s) for s in ps["knownProfiles"])) for sec in cat["sections"]: parts.append("

%s

" % (esc(sec["title"].replace(" ", "-")), esc(sec["title"]))) parts.append("

%s

" % esc(sec["what"])) parts.append(table_of(sec["vars"], life_by_name, tickets, sites)) return ("\n" "hg.conf variables" "\n%s\n\n" % (CSS, "\n".join(parts))) # --------------------------------------------------------------------------- # main # --------------------------------------------------------------------------- def main(): ap = argparse.ArgumentParser( description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("--json") ap.add_argument("--html") ap.add_argument("--check", action="store_true") ap.add_argument("--reconcile", action="store_true") ap.add_argument("--sunset", action="store_true") ap.add_argument("--sunset-new", dest="sunsetNew", action="store_true", help="report only the gates that went overdue or stalled " "since %s was accepted, and exit 1 if any did" % os.path.basename(BACKLOG_FILE)) ap.add_argument("--redundant", action="store_true", help="name the hg.conf lines on this machine that set a " "gate the tree now defaults on, and exit 1 if any " "exist") ap.add_argument("--update-baseline", dest="updateBaseline", action="store_true", help="rewrite %s from the current tree; read the diff " "before committing it" % os.path.basename(BACKLOG_FILE)) ap.add_argument("--auto-register", dest="autoRegister", action="store_true", help="write a row for every setting the tree reads that " "the catalog lacks, into the '%s' section, with the " "facts filled in and every judgement left out; read " "the diff before committing it" % AWAITING_TITLE) ap.add_argument("--verbose", action="store_true", help="with --reconcile, also print the standing drift " "that needs no action; with --redundant, the files " "read and the deliberate opt-outs") ap.add_argument("--cache", help="read and write the age cache here instead of " "hgConfAges.json next to the harvester. Also " "settable as HGCONF_AGE_CACHE") ap.add_argument("--refresh", action="store_true", help="rebuild the age cache before reporting (walks " "history, a few minutes). Needed after a release " "bumps CGI_VERSION, or a flag added since the cache " "was built has no date at all") args = ap.parse_args() cat = build() ages = None sites = None hh = None found = None if (args.sunset or args.sunsetNew or args.updateBaseline or args.html or args.json or args.refresh or args.autoRegister or args.redundant): hh = load_harvester() if hh is None: print("harvestHgConf.py not importable", file=sys.stderr) return 1 if args.cache: hh.CACHE = args.cache found, _ = hh.harvest() ages = hh.harvest_ages(names=sorted(hh.by_name(found)), refresh=args.refresh) sites = {n: d["sites"] for n, d in hh.by_name(found).items()} if args.refresh: print("rebuilt the age cache at %s: v%s, %d dated names, %d " "recorded flips" % (hh.CACHE, ages.get("current"), len([1 for e in ages.get("first", {}).values() if e.get("version")]), len(ages.get("firstTrue", {})))) rc = 0 if args.check: rc |= 1 if check(cat) else 0 if args.autoRegister: # The section dicts are module-level literals, so the rows just written # are not in this process's `cat`. Say so rather than let a combined # --auto-register --reconcile run report the state from before the write. if auto_register(cat, hh, found) and args.reconcile: print("re-run --reconcile to see the catalog as it now stands") if args.reconcile: rc |= 1 if reconcile(cat, verbose=args.verbose) else 0 if args.sunset: sunset_report(cat, ages, sites) if args.sunsetNew: rc |= 1 if sunset_delta(cat, ages, sites) else 0 if args.redundant: rc |= 1 if redundant_report(cat, ages, verbose=args.verbose) else 0 if args.updateBaseline: cur = ages.get("current") life = gate_lifecycle(cat, ages) states = gate_states(life, cur) was = read_backlog() write_backlog(states, cur, life) print("wrote %s: %d gates, %d added, %d dropped" % (BACKLOG_FILE, len(states), len(set(states) - set(was)), len(set(was) - set(states)))) if args.json: # Attribution rides along with each setting rather than in a table of # its own, so a consumer reading one entry sees where it came from. tmap = ticket_map(cat, ages) for v in all_vars(cat): if v["name"] in tmap: v["tickets"], v["ticketFrom"] = tmap[v["name"]] with open(args.json, "w") as f: json.dump(cat, f, indent=1) print("wrote %s" % args.json) if args.html: with open(args.html, "w") as f: f.write(render_html(cat, ages, sites)) print("wrote %s" % args.html) if not any([args.check, args.reconcile, args.sunset, args.sunsetNew, args.updateBaseline, args.json, args.html, args.refresh, args.autoRegister, args.redundant]): for k, v in sorted(counts(cat).items()): print("%-16s %s" % (k, v)) return rc if __name__ == "__main__": sys.exit(main())