70ac948e82b41ef316523635c04e5e2c4a89c417 mspeir Fri Sep 4 16:57:54 2026 -0700 trackLists: give the variant frequency projects their own table, move the page to goldenPath/help/mirrorTracks.html, add the otto cron line, refs #37781 Half the restricted list was national sequencing cohorts sitting under varFreqs and phasedVars, and in one alphabetical table they buried the tracks people actually write in about, OMIM, HGMD and DECIPHER. Those cohorts now get a table of their own below the rest. The split is read off the trackDb parent chain, so the next cohort added under varFreqs lands in the right table with no edit here. The page moves off the htdocs root to goldenPath/help/mirrorTracks.html, beside mirror.html, which now links to it. That link goes in src/product/README.txt, the pandoc source mirror.html is generated from. The licensing page link follows the move, and the page title changes with the file name. Also adds the weekly otto line, placed above the HGDB_CONF that would otherwise apply to it, and keeps the job from listing itself as a self-updating track. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> diff --git src/hg/utils/otto/otto.crontab src/hg/utils/otto/otto.crontab index 68549e48fd0..7a9a636567c 100644 --- src/hg/utils/otto/otto.crontab +++ src/hg/utils/otto/otto.crontab @@ -1,205 +1,212 @@ # DO NOT EDIT THE OTTO CRONTAB FILE DIRECTLY! # This file is in the kent git repository: ~/kent/src/hg/utils/otto/otto.crontab # Please edit and commit changes to the git repository, # and then ssh otto@hgwdev and run crontab on the edited and checked in file, # for example crontab ~lrnassar/kent/src/hg/utils/otto/otto.crontab # To save the current crontab back into git: # crontab -l > ~max/kent/src/hg/utils/otto/otto.crontab # logout # cd ~max/kent/src/hg/utils/otto/ # git commit otto.crontab -m 'updating crontab' # Note that you can see the list of auto-pushes of files from hgwdev to the RR by running: # cat /etc/crontab # on Hgwdev. TMPDIR=/data/tmp ottomeister=otto-group@ucsc.edu #Format: Minute - Hour - DayofMonth month Day-of-week MAILTO=otto-group@ucsc.edu # Daily sandbox update 41 00 * * 1-7 /cluster/home/otto/bin/readOnlyKentMirror # save shell history and last log into log files in ~/lastLog/ # last day of the month to catch them just before log rotation 38 23 30 4,6,9,11 * ~/lastLog/lastLog.sh 38 23 31 1,3,5,7,8,10,12 * ~/lastLog/lastLog.sh 38 23 28 2 * ~/lastLog/lastLog.sh 38 23 29 2 * ~/lastLog/lastLog.sh ####################################################### ###################### OTTO ########################### ####################################################### SHELL=/bin/sh PATH=/cluster/bin/scripts:/cluster/bin/x86_64:/usr/local/bin:/bin:/usr/bin MAILTO=otto-group@ucsc.edu #md5sum checker script - makes sure otto scripts in hive match git tree 0 1 * * 1-5 ~/kent/src/hg/utils/otto/ottoCompareGitVsHiveFiles.py # PanelApp MAILTO=max@soe.ucsc.edu,otto-group@ucsc.edu 10 10 * * 2 /hive/data/outside/otto/panelApp/doPanelApp.py #DECIPHER - Reactivated 5/13/2022 by Lou 11 4 * * * /hive/data/outside/otto/decipher/decipherWrapper.sh #GWAS Catalog 41 4 * * 3 /hive/data/outside/otto/gwas/gwasWrapper.sh #GeneReviews 00 8 * * 2 /hive/data/outside/otto/geneReviews/geneReviewsWrapper.sh #dbVar 13 8 2 * * /hive/data/outside/otto/dbVar/dbVarWrapper.sh #Orphanet 10 7 10 * * /hive/data/outside/otto/orphanet/orphanetWrapper.sh #ClinVar MAILTO=otto-group@ucsc.edu 08 00 8 * * umask 002; /hive/data/outside/otto/clinvar/doUpdate.sh #MANE 11 5 * * 1 python3 /hive/data/outside/otto/mane/doMane.py #GenCC 8 16 * * 2 python3 /hive/data/outside/otto/genCC/doGenCC.py #G2P (Gene2Phenotype) - monthly on the 5th; G2P publishes near month-end, so an #early-month run catches the previous month's update (no-ops if unchanged) 22 6 5 * * /hive/data/outside/otto/g2p/g2pWrapper.sh #OMIM 17 4 * * * /hive/data/outside/otto/omim/omimWrapper.sh 2>&1 #LOVD 14 13 * * mon /hive/data/outside/otto/lovd/doUpdate.sh #MitoMap 55 8 * * 3 /hive/data/outside/otto/mitoMap/checkMitoMapUpdate.sh #RefSeq Historical update notifier (RM #35766) - silent unless NCBI has a new release 40 8 * * 3 /hive/data/outside/otto/refSeqHistorical/checkRefSeqHistoricalUpdate.sh # InSiGHT VCEP ClinVar curated variants weekly update 08 03 * * 2 umask 002; /hive/data/outside/otto/insight/doUpdate.sh # VCEP hub spec version notifier (RM #37795) - monthly, silent unless one of our # VCEP hub description pages is behind the ClinGen CSpec registry MAILTO=otto-group@ucsc.edu 40 06 15 * * /hive/data/outside/otto/vcepVersions/checkVcepVersions.py # UniProt massive 110+ database update, takes 3-4 days. #Format: Minute - Hour - DayofMonth month Day-of-week MAILTO=max@soe.ucsc.edu,otto-group@ucsc.edu 00 07 26 * * /hive/data/outside/otto/uniprot/doUpdate.sh run #wuhCor1 uniprot otto cron by Max, May 2021 00 4 * * * /hive/data/outside/otto/uniprot/covidCheck.sh #ClinGen 00 9 * * * /hive/data/outside/otto/clinGen/clinGenWrapper.sh #ClinGen CSpec 11 11 * * 3 /hive/data/outside/otto/clinGen/clinGenCspec/makeClinGenCspec.sh #VarChat 22 22 * * 4 /hive/data/outside/otto/varChat/varChatOtto.sh #vista 0 9 1 * * /hive/data/outside/otto/vista/vistaOtto.sh # CIViC update. Releases are typically on the 1st of the month 12 12 2 * * /hive/data/outside/otto/civic/doCivicUpdate.sh #GRC Incident MAILTO=hiram@soe.ucsc.edu,otto-group@ucsc.edu 33 06 * * * /hive/data/outside/otto/grcIncidentDb/runUpdate.sh makeItSo # liftOver request watch - sends email to configured address in hg.conf # e.g.: chainFileRequestEmail=... - running every 7 minutes 1,8,15,22,29,36,43,50,57 * * * * /hive/data/outside/otto/liftRequest/devAndRR.sh # GenArk update hgcentral.genark table, once a week on Tuesday # the script itself sends email for notices about activity 37 10 * * 2 /hive/data/inside/GenArk/updateHgcentral.sh makeItSo ##### this script is making a listing that is used by the pushRR.sh script. ##### this takes a significant amount of time to run, 2 or 3 hours. ##### Needs to be done before the pushRR.sh script below is running at 01:03 58 18 * * * /hive/data/inside/GenArk/pushRR/devList.sh ##### see also, in the source tree: src/hg/utils/otto/genArk/pushRR/README.txt # push out the /gbdb/hubs/GC[AF]/ hierarchy to: # hgwbeta hgw0 hgw1 hgw2 euroNode # There is an equivalent 'pull' script running on the asiaNode # as of 2025-07-24 this script now protects itself from overrunning itself # with a lock file. It will email when it is doing that. 03 01 * * * /hive/data/inside/GenArk/pushRR/pushRR.sh ### verify liftOverChain and quickLiftChain tables have ### identical fromDb,toDb pairs: 11 3 * * * /hive/data/inside/GenArk/checkChainTables/toFromCounts.sh #NCBI RefSeq 23 8 * * 3 /hive/data/outside/otto/ncbiRefSeq/ottoNcbiRefSeq.sh #Index page public session thumbnails update MAILTO=otto-group@ucsc.edu 11 1 * * 4 /cluster/software/bin/python /cluster/bin/scripts/buildPublicSessionThumbnailsIndexPage.py /usr/local/apache/htdocs/ #Add a new daily tip to the home page 5 5 * * 1-5 /cluster/software/bin/python /cluster/bin/scripts/generateTipOfDay.py -s ~/kent/src/hg/htdocs/allTipsRaw.html +# Mirror-facing track lists (RM #37781): tracks we cannot redistribute, tracks +# that update themselves, tracks contributed from outside UCSC. Writes +# goldenPath/help/mirrorTracks.html into htdocs; reaching the RR needs a /root +# autoPush entry from cluster-admin. Weekly is plenty, none of the three lists +# moves daily. Silent unless a restricted file is found reachable on hgdownload. +32 6 * * 4 /hive/data/outside/otto/trackLists/trackLists.sh + #OMIM Upload HGDB_CONF=/cluster/home/otto/.hg.conf.beta 15 05 * * * /hive/data/outside/otto/omim/omimUploadWrapper.sh MAILTO=cells@ucsc.edu # Cell Browser news sec update by Matt/Brittney Sept 2022 0 0 * * 1-5 umask 002; ~bwick/miniconda3/bin/python3.8 ~bwick/cellBrowser/ucsc/updateNewsSec -r # Update Cell Browser .json facets files using tabulate_facets.pl 0 0 * * 1-5 cd /usr/local/apache/htdocs-cells-submit/facets && ~bwick/cellBrowser/ucsc/tabulate_facets.pl /usr/local/apache/htdocs-cells-beta/dataset.json 0 0 * * 1-5 cd /usr/local/apache/htdocs-cells-submit/facets && /usr/local/apache/htdocs-cells-submit/facets/tsv_to_json.py # Hidden-dataset reminders (RM #38246). Once a month, queue one draft email per # hidden dataset whose submitter has not been asked in a year, and mail the # report to cells@. Nothing reaches an author until a person reviews the queue # and runs "cbPingHidden send". Silent when nothing is due. 0 8 1 * * umask 002; ~mspeir/cellBrowser/ucsc/cbPingHidden scan -q --mail-report cells@ucsc.edu # cbAnnotServer: Cell Browser login/annotation API (RM #37492). Keeps gunicorn # (127.0.0.1:5051, Apache proxies /api to it) alive; restarts if down / on reboot. * * * * * umask 002; CBANNOT_ENV_FILE=/hive/data/inside/cells/cbAnnotServer/cbAnnot.env ~mspeir/cellBrowser/src/cbAnnotServer/deploy/watchdog.sh >> /hive/data/inside/cells/cbAnnotServer/logs/watchdog.log 2>&1 @reboot umask 002; sleep 180; CBANNOT_ENV_FILE=/hive/data/inside/cells/cbAnnotServer/cbAnnot.env ~mspeir/cellBrowser/src/cbAnnotServer/deploy/watchdog.sh >> /hive/data/inside/cells/cbAnnotServer/logs/watchdog.log 2>&1 # DE worker: Cell Browser differential expression (RM #24912). Keeps deWorker.py # alive (reads the /hive job queue); restarts if down / on reboot. # # MOVED (2026-08-22): the DE worker now runs from the otto crontab ON # hgcompute-08, not here. The compute host reads exprMatrix.bin from a /hive # cache it fetches over HTTP (deCache.py), so it no longer needs the web host's # local docroot, and DE compute is off hgwdev. These two lines are left commented # for history; the live copy is in otto's crontab on hgcompute-08. #* * * * * umask 002; DE_ENV_FILE=/hive/data/inside/cells/cbAnnotServer/de.env ~mspeir/cellBrowser/src/cbAnnotServer/de/deploy/deWorkerWatchdog.sh >> /hive/data/inside/cells/cbAnnotServer/logs/deWorker-watchdog.log 2>&1 #@reboot umask 002; DE_ENV_FILE=/hive/data/inside/cells/cbAnnotServer/de.env ~mspeir/cellBrowser/src/cbAnnotServer/de/deploy/deWorkerWatchdog.sh >> /hive/data/inside/cells/cbAnnotServer/logs/deWorker-watchdog.log 2>&1 # Malacards update. Not done monthly, but running monthly anyways, as they don't have a fixed schedule #Format: Minute - Hour - DayofMonth month Day-of-week MAILTO=mhaeussl@ucsc.edu 04 04 20 * * /hive/data/outside/otto/malacards/update.sh MAILTO=jbirgmei@ucsc.edu # PubTator3 dbSNP track update 28 08 * * * /bin/zsh -c 'cd /hive/data/outside/otto/pubtatorDbSnp && ./doUpdate.sh' #MAILTO=cells@ucsc.edu #@reboot umask 0000; sleep 240; /cluster/home/max/htdocs-cells-submit/tusd/tusd_linux_amd64/tusd -upload-dir /hive/data/inside/cells/upload/ -port 8443 -max-size 100000000000