70ac948e82b41ef316523635c04e5e2c4a89c417 mspeir Fri Sep 4 16:57:54 2026 -0700 trackLists: give the variant frequency projects their own table, move the page to goldenPath/help/mirrorTracks.html, add the otto cron line, refs #37781 Half the restricted list was national sequencing cohorts sitting under varFreqs and phasedVars, and in one alphabetical table they buried the tracks people actually write in about, OMIM, HGMD and DECIPHER. Those cohorts now get a table of their own below the rest. The split is read off the trackDb parent chain, so the next cohort added under varFreqs lands in the right table with no edit here. The page moves off the htdocs root to goldenPath/help/mirrorTracks.html, beside mirror.html, which now links to it. That link goes in src/product/README.txt, the pandoc source mirror.html is generated from. The licensing page link follows the move, and the page title changes with the file name. Also adds the weekly otto line, placed above the HGDB_CONF that would otherwise apply to it, and keeps the job from listing itself as a self-updating track. Co-Authored-By: Claude Opus 5 (1M context) diff --git src/hg/utils/otto/trackLists/README.txt src/hg/utils/otto/trackLists/README.txt index e73e96178fb..c78b93e8c8c 100644 --- src/hg/utils/otto/trackLists/README.txt +++ src/hg/utils/otto/trackLists/README.txt @@ -1,70 +1,73 @@ trackLists - RM #37781 Builds one page answering the three questions mirror sites keep asking: which tracks we cannot pass on, which tracks update themselves, and which tracks were contributed by someone outside UCSC. collect.py gathers all three lists -> collected.json mkPage.py renders collected.json -> HTML trackLists.sh what cron runs Why list 1 uses more than one query ----------------------------------- No single trackDb setting marks every restricted track: * tableBrowser off is the usual marker, but OMIM does not use it. OMIM sets "tableBrowser noGenome ..." with a noGenomeReason naming OMIM's distribution terms, so a query for "off" alone silently misses it. * Not all noGenome is about licensing. CRISPR and JASPAR set it because a genome-wide query times out. The reason text is what separates them. * The convention of putting restricted files under an underscore directory (/gbdb/hg38/varFreqs/_topmed/ and friends) is real but partial: decipher, mexbb, spliceAI, cosmicRegions and hgmd are restricted and are not under one. So collect.py runs every test it can and unions the results, recording on each row which tests fired. It also checks the download server both directions: a MySQL track that exists here but is missing from hgdownload is almost certainly restricted, and a file we call restricted that hgdownload still serves is a bug worth mailing about. Careful with the public page ---------------------------- The hgdownload cross-check names restricted files that are currently reachable. That must never appear on a page anyone can read, so mkPage.py omits it unless --internal is passed. trackLists.sh writes the public variant to htdocs and keeps the internal one in this directory. Speed ----- The GenArk crawl walks /gbdb/genark and takes more than ten minutes, so it is cached in cache/contrib.txt and re-run only when the cache is over a week old (--refresh-contrib forces it). The crawl writes to a temp file and renames, because a crawl cut short mid-write leaves a shorter list that still looks plausible. hgdownload directory listings are cached for a day. A run that hits warm caches takes a couple of minutes; a cold run with the crawl takes fifteen or so. Publishing ---------- -The script writes /usr/local/apache/htdocs/trackLists.html on hgwdev. -Pushing that to the RR needs a /root/AutoPush script added to -/etc/crontab by cluster-admin; ask them for it, and use the existing lines as -the template (tipsAutoPush, thumbNailAutoPush, asmAliasAutoPush). +The script writes goldenPath/help/mirrorTracks.html under htdocs on hgwdev, +next to mirror.html, which is the page that links to it. Pushing that to the RR +needs a /root/AutoPush script added to /etc/crontab by cluster-admin; ask +them for it, and use the existing lines as the template (tipsAutoPush, +thumbNailAutoPush, asmAliasAutoPush). + +The otto job is still called trackLists while the page it writes is called +mirrorTracks.html. The page was renamed after review; the job was not, since it +is referred to by path in otto.crontab. The page must be mode 775 in htdocs. Apache runs the SSI includes on a .html file only when its execute bit is set (XBitHack); without it the page is served verbatim and the reader sees the bare content with no menu bar and no stylesheets. allTips.html is in exactly that state on the RR today, so this is an easy mistake to repeat. trackLists.sh chmods the page after copying it. The generated page is deliberately not in the kent tree. That matches the other generated pages: allTips.html and thumbNailLinks.html live only in htdocs and are not tracked in git. Only the generator is committed. (assemblyRequest.html looks like a precedent but is not one; it is now just a redirect stub.) -Where the page should finally live is not settled. The ticket says only that it -is "autoPushed out on a cycle and linked to the mirror site", naming no path, so -the top-level htdocs location here is a choice, not a requirement, and it can be -moved. The natural reading of "the mirror site" is a link from -goldenPath/help/mirror.html, but that is an inference from Lou's wording rather -than something the ticket states. +"The mirror site" in the ticket meant goldenPath/help/mirror.html, confirmed by +Lou on 2026-09-04, and the page is linked from there. It also keeps its link from +the licensing page, src/hg/htdocs/license/index.html, which is where a reader who +wants the exact list of what we cannot hand on is most likely to start.