d1444e2ca8e21460228433810b1a6703c6520db1 braney Mon Sep 7 12:02:02 2026 -0700 otto: keep the two surveys beside the monitor that reads them, refs #38101 The monitor's stamp table is the machine-readable half of a survey that lived only in the redmineNotes directory, which is not a repository. The reasoning behind each of the forty globs, and the measured curl shape behind each source URL, were therefore one copy on /hive. Both surveys now sit beside the script. Read ottoFailureSignatures.tsv before changing a stamp glob: it says what each job writes and when, which is the difference between a stamp that tracks every run and one that only moves when the data changes. diff --git src/hg/utils/otto/ottoMonitor/README src/hg/utils/otto/ottoMonitor/README index 1598ff33e54..bd91101cfa3 100644 --- src/hg/utils/otto/ottoMonitor/README +++ src/hg/utils/otto/ottoMonitor/README @@ -11,30 +11,43 @@ WHAT IT ASKS One question per job: did it run when it was supposed to. It answers that from a run stamp, meaning something the job leaves behind whether or not the data changed. Fifteen jobs write a log or a named file on every run. Nine leave only their working directory's mtime, because they write a temp file and delete it. Eight leave nothing at all, and those are reported as blind rather than as passing, so the gap stays visible instead of reading as good news. A late job is not automatically somebody's bug. So a late job with a source URL gets that URL fetched. If the source is down the job is left alone, and only a second run in a row that fails the same way becomes a ticket. If the source answers, the failure was something else and it files the same day. +THE SURVEYS BEHIND IT + + ottoSourceUrls.tsv where each job's data comes from, the URL to probe, + and the curl shape that URL actually answers. Every + one was fetched, not read out of a script. + ottoFailureSignatures.tsv what a run, a change and a failure look like on disk + for each of the 47 jobs. ottoMonitorStamps.tsv is + the machine-readable part of it; this is the reasoning. + +Read the signature survey before changing a stamp glob. It says what each job +writes and when, which is the difference between a stamp that tracks every run +and one that only moves when the data changes. + THE THREE INPUTS ottoOwners.tsv who owns each job, whether to watch it, its source URL, and its schedule. Canonical copy is in genecats at otto/ottoOwners.tsv. The team edits it there. ottoMonitorStamps.tsv where each job's run stamp lives. Kept beside this script, because it is about the monitor and not about ownership. state.json written under /hive/data/outside/otto/ottoMonitor/. Holds the last run seen, the last verdict, the number of runs in a row that failed the same way, and the ticket number when one is open. Without it a second strike cannot be told from a first. RUNNING IT