52357be3947a4643b735276094b8d01da8e3f8f0 braney Tue Sep 8 10:30:02 2026 -0700 ottoMonitor: check the last closed grace window, and let a job be owned by whoever is on duty Three changes, all from comments on the ticket. Lou: civic has no individual owner, so it belongs to the otto person. The owner column of ottoOwners.tsv now accepts ottoOnDuty, which the monitor resolves from the ottoOnDuty header at run time, so the rotation stays a one-line edit. The ticket body says the job has no individual owner, and the same person is not added as a watcher twice when the owner is also the person on duty. The grace window is now measured back from the last scheduled time whose window has already closed, instead of forward from the latest scheduled time. Written the other way, a daily job scheduled fewer than graceHours before the monitor's own 12:15 run could never be reported late, because every check landed inside a fresh window. Six of the forty jobs were in that hole: clinGen, genArkPushRR, grcIncidentDb, liftRequest, omim and pubtatorDbSnp. Max: a uniprot run can take days, and how long depends on the size of the release. Its stamp is created by a > redirect when the run starts, so the grace does not have to cover the run length, and a fresh stamp does not mean the run worked. That limit is now written down in the stamps table and the README, with the live case: the uniprot run dies after about 37 minutes on a missing lxml and has produced no output since January 2025, while the monitor reads it as on time. refs #38101 diff --git src/hg/utils/otto/ottoMonitor/README src/hg/utils/otto/ottoMonitor/README index dd90899e598..bbc23416771 100644 --- src/hg/utils/otto/ottoMonitor/README +++ src/hg/utils/otto/ottoMonitor/README @@ -67,34 +67,51 @@ THE TWO COPIES Same rule as the rest of otto. Edit the copy in the kent tree at src/hg/utils/otto/ottoMonitor, commit, then copy it out to /hive/data/outside/otto/ottoMonitor, which is what cron runs. Never edit the hive copy directly. TRAPS WORTH KNOWING BEFORE YOU CHANGE A STAMP A file's mtime is not always the run time. hgsqlTableDate calls utime() to set a file's mtime to the database TABLE's date, on purpose, so a later -nt test asks about the data rather than the run. omimUpload's upload/*.date files look like an ideal run stamp and are not; the directory mtime is used instead. +A run stamp says the job started, not that it worked. Most of these globs are +written early in a run on purpose, so a job that dies half way still shows that +it tried. The cost is that a job which fails the same way every time looks on +time forever. uniprot is the live example: its monthly run dies after about 37 +minutes on a missing lxml, it has produced no output since January 2025, and the +browser serves UniProt release 2024_06 while the downloaded source is at 2026_02. +The monitor still reads it as on time, because lastRun.log is fresh every month. +Catching that needs a second kind of check, on the age of the job's OUTPUT +rather than on its schedule, and the monitor does not have one. + The eight directory-mtime jobs work today but nobody designed that signal. Copying a file into one of those directories by hand resets the clock and the monitor will read it as a run. +The grace window is measured from the last scheduled time whose window has +already closed, not from the latest scheduled time. Written the other way, a +daily job scheduled fewer than graceHours before the monitor's own run time can +never be called late, because every check lands inside a fresh window. Six of +the forty were in that hole: clinGen, genArkPushRR, grcIncidentDb, liftRequest, +omim and pubtatorDbSnp. + cron's day rule is an OR, not an AND: when both day-of-month and day-of-week are restricted the job runs when EITHER matches. A job like "14 13 * * mon" would look like it never runs if this were read the other way. A job with more than one crontab line has its schedules joined with ";" in the cron column of ottoOwners.tsv, and the monitor takes whichever fired last. ottoLastLog is the only one so far, because cron cannot say "the last day of the month" in one line. THE EIGHT BLIND JOBS gwas, mane, ncbiRefSeq, ottoGitVsHive, refSeqHistorical, strchive, uniprotWuhCor1, vcepVersions. Two of them are a one-line fix rather than a monitor problem. gwas already