430bb5b5b7601fa51ed1872e5c7a55def547c450 hiram Wed Apr 29 14:27:34 2026 -0700 refactor the workflowMonitor to get json operations into galaxyState.py and finalizing state diagram refs #31811 diff --git src/hg/utils/otto/userRequests/README.txt src/hg/utils/otto/userRequests/README.txt index b53779b6bc3..abbd1c7db95 100644 --- src/hg/utils/otto/userRequests/README.txt +++ src/hg/utils/otto/userRequests/README.txt @@ -1,87 +1,103 @@ +The 'status' field in the ottoRequest table maintains state: + + 0 == request has been received by API (set by API) + 1 == request has been acknowledged by ottoRequest.py (set by ottoRequest.py) + 2 == galaxy jobs have been started (set by ottoRequestAlign.sh) + 3 == galaxy jobs have completed (set by workflowMonitor.sh) + 4 == download from galaxy has taken place and track files have been created + (set by ottoRequestWatch.sh) + 5 == symlinks for tracks are in place, ready to push files + (set by ottoRequestWatch.sh) + 6 == push of files is complete (set by ottoRequestPush.sh) + 7 == error condition - some error has taken place, set by any script + 8 == final email notification has been sent, galaxy workflow + has been deleted, process is complete (set by ottoRequestWatch.sh) + Method of operation: ############################################################################ ### 1. User makes request via the liftRequest.html page. The submit button causes a call to the API with four arguments: fromGenome toGenome email comment The API does an INSERT operation into the hgcentral.ottoRequest table: "INSERT INTO %s (requestType, fromDb, toDb, email, comment, requestTime, status, buildDir) VALUES ( 'liftOver', '%s','%s','%s','%s',now(), 0, '')", ottoTable, fromGenome, toGenome, email, comment); For example: *************************** 1. row *************************** id: 1 requestType: liftOver fromDb: GCF_000260355.1 toDb: GCF_004115215.2 email: nullmodel@gmail.com comment: testing the galaxy pipeline, from: star-nosed mole (GCF_000260355.1), to: platypus (Pmale09 v4 2020) (GCF_004115215.2) requestTime: 2026-04-23 15:20:19 status: 0 buildDir: completeTime: NULL -The 'status' field is going to keep track: - 0 pending, 1 notified, 2 in progress, 3 galaxy done, 4 tracks complete, - 5 finish notification, 6 complete, 7 problems -and will affect other operations. - ############################################################################ ### 2. ottoRequest.py - otto user cron job running each minute to watch the - - ottoRequest table in hgcentral - when new entries are + - ottoRequest table in hgcentral. Watches for two types of + - entries: liftRequest and assembly - when new entries are - detected (status==0) it marks the table entry as - pending (status=1) and sends - out notification emails, one to the requesting user and - - the other to UCSC via the specification in hg.conf: - chainFileRequestEmail=chain-file-request-group@ucsc.edu - apiFromEmail=genome-www@soe.ucsc.edu - - TBD: verify bounces get back to that apiFromEmail - - this email bounce operation might work much better here - - on hgwdev and it isn't from the 'apache' user + - Bcc to either chain-file-request-group@ucsc.edu for liftRequest + - or Bcc genark-request-group@ucsc.edu for assembly requests. + - The From and Return-To addresses are genome-www@soe.ucsc.edu + - and I believe that will receive bounces from bad user addresses ############################################################################ ### 3. ottoRequestWatch.sh - cron script running in hiram hgwdev account to watch - the ottoRequest table. When new entries show - up (status=1) it will get the galaxy workflow - running by using ottoRequestAlign.sh to construct the - kegAlignLastz.sh script arguments ############################################################################ ### 4. ottoRequestAlign.sh - given an 'id' number in the ottoRequest table, this - will generate the arguments to: 'kegAlignLastz.sh' - to get the alignment started in galaxy. - Uses the hgcentraltest.genark table and the file - dbDb.name.clade.tsv to determine full assembly ID - names and 'clades' for the kegAlignLastz.sh script: - primate - mammal - other - and this will decide which - assembly will be target and query by checking their - respective N50 sizes. Also uses the file: dbDb.name.clade.tsv - to map the UCSC database names into - GenArk 'clades' to make the selection: - primate - mammal - other - can set status to: - 2 == in progress + - 7 == problems ############################################################################ ### 5. kegAlignLastz.sh - script to start the galaxy workflow, typical call: kegAlignLastz.sh GCF_004115215.2_mOrnAna1.pri.v4 GCF_000260355.1_ConCri1.0 mammal mammal + This script works in two directories, the target lastz work directory + and the query swap directory. ############################################################################ ### 6. workflowMonitor.sh - after the galaxy WF has started, this script can - check the status of the job and if it is done, then - the processing of the results will take place to - construct the chain files. - can set status to: - 7 == problems - 3 == galaxy finished - - 4 == tracks complete + +############################################################################ +### 7. +ottoRequestPush.sh - cron job watching status == 5 - will run the pushing + procedure for the relevant assemblies. When complete will set status = 6