dcb1fa309fa0157eb7076263b1a0a36b0c7babec
mspeir
  Thu Sep 3 18:28:29 2026 -0700
Add trackLists otto generator: tracks we cannot redistribute, tracks that
update themselves, and contributed tracks, refs #37781

Builds one page answering the three questions mirror sites keep asking.

List 1 unions several tests rather than relying on one, because no single
trackDb setting marks every restricted track: tableBrowser off is the usual
marker but OMIM instead sets tableBrowser noGenome with a noGenomeReason
naming its distribution terms, so a query for "off" alone misses it. Not all
noGenome is about licensing either -- CRISPR and JASPAR set it because a
genome-wide query times out -- so the reason text is what separates them. The
convention of putting restricted files under an underscore directory is real
but partial: decipher, mexbb, spliceAI, cosmicRegions and hgmd are restricted
and are not under one.

It also checks the download server both directions. A trackDb track whose
MySQL table exists here but is missing from hgdownload is almost certainly
restricted, and a file we call restricted that hgdownload still serves is a
bug the script prints so cron mails it.

mkPage.py omits that second cross-check unless --internal is given, since
naming reachable restricted files on a world-readable page would defeat the
point. The generated page is not committed, matching allTips.html and
thumbNailLinks.html, which live only in htdocs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/hg/utils/otto/trackLists/trackLists.sh src/hg/utils/otto/trackLists/trackLists.sh
new file mode 100755
index 00000000000..4963cceea3b
--- /dev/null
+++ src/hg/utils/otto/trackLists/trackLists.sh
@@ -0,0 +1,39 @@
+#!/bin/bash
+#
+# trackLists.sh - build the mirror-facing page listing tracks we cannot
+# redistribute, tracks that update themselves, and contributed tracks. RM #37781
+#
+# Cron (otto): once a week is plenty; none of these lists move daily.
+#     32 6 * * 4 /hive/data/outside/otto/trackLists/trackLists.sh
+#
+# Writes the public page to htdocs on hgwdev. Getting it to the RR needs a
+# /root/<name>AutoPush entry in /etc/crontab, which only cluster-admin can add
+# (same pattern as the daily tips and the session thumbnails, which are likewise
+# generated into htdocs and are not tracked in git).
+#
+# Output behaviour: quiet on success, except that a restricted file found to be
+# reachable on hgdownload always prints, so cron mails it.
+
+set -o errexit -o pipefail
+umask 002
+
+DIR=${RTDIR:-/hive/data/outside/otto/trackLists}
+HTDOCS=${HTDOCS:-/usr/local/apache/htdocs}   # override for testing
+PAGE=trackLists.html
+
+cd "$DIR"
+
+# --refresh-contrib is implicit: the crawl re-runs itself when the cache ages out
+./collect.py --cache "$DIR/cache" -o "$DIR/collected.json"
+
+# public page: no list of reachable restricted files
+./mkPage.py -i "$DIR/collected.json" -o "$DIR/$PAGE"
+
+# internal copy, keeps the hgdownload cross-check, stays on hgwdev
+./mkPage.py -i "$DIR/collected.json" -o "$DIR/internal.html" --internal
+
+# only replace the live page if it actually changed
+if ! cmp -s "$DIR/$PAGE" "$HTDOCS/$PAGE"; then
+    cp -p "$DIR/$PAGE" "$HTDOCS/$PAGE"
+    echo "trackLists: updated $HTDOCS/$PAGE"
+fi