69f4ef65bd1f1d15304c3fe0a47e701d58d6e688 mspeir Fri Sep 18 13:05:49 2026 -0700 trackLists: take the otto assembly counts from the jobs, and quiet the weekly mail. refs #38295 Code review feedback from Lou on the mirror track lists page. The Assemblies column counted every assembly with a table matching the job's keyword, not the assemblies the job rebuilds, so NCBI RefSeq read 84 where ottoNcbiRefSeq.sh runs four and GRC Incident read 61 where runUpdate.sh works through ten. Each job now carries the list it actually builds, read out of its script; UniProt keeps the trackDb count because doUniprot chooses its assemblies at run time. A run reports either way the list and trackDb disagree. An otto job the keyword table did not recognize vanished from the page without a word, which is how STRchive went missing from it. STRchive is described now, and anything else unrecognized is named on the page and in the cron mail. The weekly mail was the whole progress log with the reachable restricted files buried in it, because every note() went to stderr and the "updated" line fired every run. Progress now goes to stdout and on to lastRun.log; stderr carries only what needs a person, so mail from this cron line means something is wrong. The "every file marked as restricted is correctly blocked" line could print on a run where no fetch got an answer, and covered only the 39 of 58 rows that name a file. It now says how many were tried and stays quiet about blocking when the check could not finish. Four rows reading "Mutation: A" through "Mutation: T" sat under M in the restricted table, nowhere near AlphaGenome. A subtrack of a listed track now sorts directly under that track, keeping its own row, since each of these is a separate file a mirror cannot have. A track named after its container is labelled with the container as well, so varFreqsBackground reads "SNV Frequencies: Population reference" rather than "Population reference". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> diff --git src/hg/utils/otto/otto.crontab src/hg/utils/otto/otto.crontab index f0ab527e0e9..4aa9c3ca027 100644 --- src/hg/utils/otto/otto.crontab +++ src/hg/utils/otto/otto.crontab @@ -1,217 +1,219 @@ # DO NOT EDIT THE OTTO CRONTAB FILE DIRECTLY! # This file is in the kent git repository: ~/kent/src/hg/utils/otto/otto.crontab # Please edit and commit changes to the git repository, # and then ssh otto@hgwdev and run crontab on the edited and checked in file, # for example crontab ~lrnassar/kent/src/hg/utils/otto/otto.crontab # To save the current crontab back into git: # crontab -l > ~max/kent/src/hg/utils/otto/otto.crontab # logout # cd ~max/kent/src/hg/utils/otto/ # git commit otto.crontab -m 'updating crontab' # Note that you can see the list of auto-pushes of files from hgwdev to the RR by running: # cat /etc/crontab # on Hgwdev. TMPDIR=/data/tmp ottomeister=otto-group@ucsc.edu #Format: Minute - Hour - DayofMonth month Day-of-week MAILTO=otto-group@ucsc.edu # Daily sandbox update 41 00 * * 1-7 /cluster/home/otto/bin/readOnlyKentMirror # save shell history and last log into log files in ~/lastLog/ # last day of the month to catch them just before log rotation 38 23 30 4,6,9,11 * ~/lastLog/lastLog.sh 38 23 31 1,3,5,7,8,10,12 * ~/lastLog/lastLog.sh 38 23 28 2 * ~/lastLog/lastLog.sh 38 23 29 2 * ~/lastLog/lastLog.sh ####################################################### ###################### OTTO ########################### ####################################################### SHELL=/bin/sh PATH=/cluster/bin/scripts:/cluster/bin/x86_64:/usr/local/bin:/bin:/usr/bin MAILTO=otto-group@ucsc.edu #md5sum checker script - makes sure otto scripts in hive match git tree 0 1 * * 1-5 ~/kent/src/hg/utils/otto/ottoCompareGitVsHiveFiles.py # PanelApp MAILTO=max@soe.ucsc.edu,otto-group@ucsc.edu 10 10 * * 2 /hive/data/outside/otto/panelApp/doPanelApp.py #DECIPHER - Reactivated 5/13/2022 by Lou 11 4 * * * /hive/data/outside/otto/decipher/decipherWrapper.sh #GWAS Catalog 41 4 * * 3 /hive/data/outside/otto/gwas/gwasWrapper.sh #GeneReviews 00 8 * * 2 /hive/data/outside/otto/geneReviews/geneReviewsWrapper.sh #dbVar 13 8 2 * * /hive/data/outside/otto/dbVar/dbVarWrapper.sh #Orphanet 10 7 10 * * /hive/data/outside/otto/orphanet/orphanetWrapper.sh #ClinVar MAILTO=otto-group@ucsc.edu 08 00 8 * * umask 002; /hive/data/outside/otto/clinvar/doUpdate.sh #MANE 11 5 * * 1 python3 /hive/data/outside/otto/mane/doMane.py #GenCC 8 16 * * 2 python3 /hive/data/outside/otto/genCC/doGenCC.py #G2P (Gene2Phenotype) - monthly on the 5th; G2P publishes near month-end, so an #early-month run catches the previous month's update (no-ops if unchanged) 22 6 5 * * /hive/data/outside/otto/g2p/g2pWrapper.sh #OMIM 17 4 * * * /hive/data/outside/otto/omim/omimWrapper.sh 2>&1 #LOVD 14 13 * * mon /hive/data/outside/otto/lovd/doUpdate.sh #MitoMap 55 8 * * 3 /hive/data/outside/otto/mitoMap/checkMitoMapUpdate.sh #RefSeq Historical update notifier (RM #35766) - silent unless NCBI has a new release 40 8 * * 3 /hive/data/outside/otto/refSeqHistorical/checkRefSeqHistoricalUpdate.sh #STRchive - hg38 disease-associated tandem repeats, dashnowlab/STRchive#333. #Weekly check for a new GitHub release, silent unless STRchive published one. #STRchive releases one to two times a month. 45 7 * * 1 umask 002; /hive/data/outside/otto/strchive/strchiveOtto.py # InSiGHT VCEP ClinVar curated variants weekly update 08 03 * * 2 umask 002; /hive/data/outside/otto/insight/doUpdate.sh # VCEP hub spec version notifier (RM #37795) - monthly, silent unless one of our # VCEP hub description pages is behind the ClinGen CSpec registry MAILTO=otto-group@ucsc.edu 40 06 15 * * /hive/data/outside/otto/vcepVersions/checkVcepVersions.py # UniProt massive 110+ database update, takes 3-4 days. #Format: Minute - Hour - DayofMonth month Day-of-week MAILTO=max@soe.ucsc.edu,otto-group@ucsc.edu 00 07 26 * * /hive/data/outside/otto/uniprot/doUpdate.sh run #wuhCor1 uniprot otto cron by Max, May 2021 00 4 * * * /hive/data/outside/otto/uniprot/covidCheck.sh #ClinGen 00 9 * * * /hive/data/outside/otto/clinGen/clinGenWrapper.sh #ClinGen CSpec 11 11 * * 3 /hive/data/outside/otto/clinGen/clinGenCspec/makeClinGenCspec.sh #VarChat 22 22 * * 4 /hive/data/outside/otto/varChat/varChatOtto.sh #vista 0 9 1 * * /hive/data/outside/otto/vista/vistaOtto.sh # CIViC update. Releases are typically on the 1st of the month 12 12 2 * * /hive/data/outside/otto/civic/doCivicUpdate.sh #GRC Incident MAILTO=hiram@soe.ucsc.edu,otto-group@ucsc.edu 33 06 * * * /hive/data/outside/otto/grcIncidentDb/runUpdate.sh makeItSo # liftOver request watch - sends email to configured address in hg.conf # e.g.: chainFileRequestEmail=... - running every 7 minutes 1,8,15,22,29,36,43,50,57 * * * * /hive/data/outside/otto/liftRequest/devAndRR.sh # GenArk update hgcentral.genark table, once a week on Tuesday # the script itself sends email for notices about activity 37 10 * * 2 /hive/data/inside/GenArk/updateHgcentral.sh makeItSo ##### this script is making a listing that is used by the pushRR.sh script. ##### this takes a significant amount of time to run, 2 or 3 hours. ##### Needs to be done before the pushRR.sh script below is running at 01:03 58 18 * * * /hive/data/inside/GenArk/pushRR/devList.sh ##### see also, in the source tree: src/hg/utils/otto/genArk/pushRR/README.txt # push out the /gbdb/hubs/GC[AF]/ hierarchy to: # hgwbeta hgw0 hgw1 hgw2 euroNode # There is an equivalent 'pull' script running on the asiaNode # as of 2025-07-24 this script now protects itself from overrunning itself # with a lock file. It will email when it is doing that. 03 01 * * * /hive/data/inside/GenArk/pushRR/pushRR.sh ### verify liftOverChain and quickLiftChain tables have ### identical fromDb,toDb pairs: 11 3 * * * /hive/data/inside/GenArk/checkChainTables/toFromCounts.sh #NCBI RefSeq 23 8 * * 3 /hive/data/outside/otto/ncbiRefSeq/ottoNcbiRefSeq.sh #Index page public session thumbnails update MAILTO=otto-group@ucsc.edu 11 1 * * 4 /cluster/software/bin/python /cluster/bin/scripts/buildPublicSessionThumbnailsIndexPage.py /usr/local/apache/htdocs/ #Add a new daily tip to the home page 5 5 * * 1-5 /cluster/software/bin/python /cluster/bin/scripts/generateTipOfDay.py -s ~/kent/src/hg/htdocs/allTipsRaw.html # Mirror-facing track lists (RM #37781): tracks we cannot redistribute, tracks # that update themselves, tracks contributed from outside UCSC. Writes # goldenPath/help/mirrorTracks.html into htdocs; reaching the RR needs a /root # autoPush entry from cluster-admin. Weekly is plenty, none of the three lists -# moves daily. Silent unless a restricted file is found reachable on hgdownload. +# moves daily. Progress goes to trackLists/lastRun.log; mail from this line means +# something needs a person (a restricted file reachable on hgdownload, an otto job +# whose assembly list disagrees with trackDb, an otto job with no description). 32 6 * * 4 /hive/data/outside/otto/trackLists/trackLists.sh #OMIM Upload HGDB_CONF=/cluster/home/otto/.hg.conf.beta 15 05 * * * /hive/data/outside/otto/omim/omimUploadWrapper.sh MAILTO=cells@ucsc.edu # Cell Browser news sec update by Matt/Brittney Sept 2022 0 0 * * 1-5 umask 002; ~bwick/miniconda3/bin/python3.8 ~bwick/cellBrowser/ucsc/updateNewsSec -r # Update Cell Browser .json facets files using tabulate_facets.pl 0 0 * * 1-5 cd /usr/local/apache/htdocs-cells-submit/facets && ~bwick/cellBrowser/ucsc/tabulate_facets.pl /usr/local/apache/htdocs-cells-beta/dataset.json 0 0 * * 1-5 cd /usr/local/apache/htdocs-cells-submit/facets && /usr/local/apache/htdocs-cells-submit/facets/tsv_to_json.py # Hidden-dataset reminders (RM #38246). Once a month, queue one draft email per # hidden dataset whose submitter has not been asked in a year, and mail the # report to cells@. Nothing reaches an author until a person reviews the queue # and runs "cbPingHidden send". Silent when nothing is due. 0 8 1 * * umask 002; ~mspeir/cellBrowser/ucsc/cbPingHidden scan -q --mail-report cells@ucsc.edu # cbAnnotServer: Cell Browser login/annotation API (RM #37492). Keeps gunicorn # (127.0.0.1:5051, Apache proxies /api to it) alive; restarts if down / on reboot. * * * * * umask 002; CBANNOT_ENV_FILE=/hive/data/inside/cells/cbAnnotServer/cbAnnot.env ~mspeir/cellBrowser/src/cbAnnotServer/deploy/watchdog.sh >> /hive/data/inside/cells/cbAnnotServer/logs/watchdog.log 2>&1 @reboot umask 002; sleep 180; CBANNOT_ENV_FILE=/hive/data/inside/cells/cbAnnotServer/cbAnnot.env ~mspeir/cellBrowser/src/cbAnnotServer/deploy/watchdog.sh >> /hive/data/inside/cells/cbAnnotServer/logs/watchdog.log 2>&1 # DE worker: Cell Browser differential expression (RM #24912). Keeps deWorker.py # alive (reads the /hive job queue); restarts if down / on reboot. # # MOVED (2026-08-22): the DE worker now runs from the otto crontab ON # hgcompute-08, not here. The compute host reads exprMatrix.bin from a /hive # cache it fetches over HTTP (deCache.py), so it no longer needs the web host's # local docroot, and DE compute is off hgwdev. These two lines are left commented # for history; the live copy is in otto's crontab on hgcompute-08. #* * * * * umask 002; DE_ENV_FILE=/hive/data/inside/cells/cbAnnotServer/de.env ~mspeir/cellBrowser/src/cbAnnotServer/de/deploy/deWorkerWatchdog.sh >> /hive/data/inside/cells/cbAnnotServer/logs/deWorker-watchdog.log 2>&1 #@reboot umask 002; DE_ENV_FILE=/hive/data/inside/cells/cbAnnotServer/de.env ~mspeir/cellBrowser/src/cbAnnotServer/de/deploy/deWorkerWatchdog.sh >> /hive/data/inside/cells/cbAnnotServer/logs/deWorker-watchdog.log 2>&1 # Malacards update. Not done monthly, but running monthly anyways, as they don't have a fixed schedule #Format: Minute - Hour - DayofMonth month Day-of-week MAILTO=mhaeussl@ucsc.edu 04 04 20 * * /hive/data/outside/otto/malacards/update.sh MAILTO=jbirgmei@ucsc.edu # PubTator3 dbSNP track update 28 08 * * * /bin/zsh -c 'cd /hive/data/outside/otto/pubtatorDbSnp && ./doUpdate.sh' #MAILTO=cells@ucsc.edu #@reboot umask 0000; sleep 240; /cluster/home/max/htdocs-cells-submit/tusd/tusd_linux_amd64/tusd -upload-dir /hive/data/inside/cells/upload/ -port 8443 -max-size 100000000000