75e828960283291546d2c1a27845e2cf3823adcd max Mon Sep 14 05:29:02 2026 -0700 uniprot otto: the miniprot cluster job needs absolute paths GRCz12ab failed with the parasol job crashing four times, return 1, no output. The wrapper I wrote ran miniprot -t 16 --gff protToGenome/GRCz12ab/.../genome.fa fasta/7955.fa > $1 and a parasol job runs with its working directory set to the batch directory, not to the directory the pipeline runs in, so neither input existed from the job's point of view. The BLAST batch next door gets away with relative paths because it cds into its own workdir and its jobList is written relative to that; this batch directory sits a level deeper and its paths were relative to the otto root. Every path in the wrapper, the jobList command and the output check is now absolute. Verified on the cluster against the real 1.48 Gb zebrafish genome: successful batch, a 195 MB GFF with 93518 mRNA records. refs #38300 diff --git src/hg/makeDb/doc/hg38/fiberSeq.txt src/hg/makeDb/doc/hg38/fiberSeq.txt index ce6d30291ec..7abb25d631e 100644 --- src/hg/makeDb/doc/hg38/fiberSeq.txt +++ src/hg/makeDb/doc/hg38/fiberSeq.txt @@ -105,45 +105,57 @@ # That is correct for an interrupted transfer and silently corrupting when the # server has REPLACED the file: curl would have appended the tail of the new # 5.2 GB hap1 file to the 512-byte stub and reported success, and the size check # afterwards would have passed because the total matches. The script now # fetches into <file>.part and moves it into place only when complete, so a # resume can never land on a finished file, and the final mv is atomic, so # hgTracks never reads a half-written file out of /gbdb. # # The peak file was reissued too but still has the fieldCount 3 header, so the # rebuild below is still required. Its peak calls changed substantially, # 429,883 source peaks before and 196,742 now, which is a different call set # rather than a reformatting. Noted on fiberSeqCompendium.html, since figures # made from the first version of the track will not reproduce for this sample. # --------------------------------------------------------------------------- -# Nucleosome density +# Nucleosome density: mirrored, NOT displayed # --------------------------------------------------------------------------- # Sent separately in September 2026, as all.nucleosome.coverage.bw under the # same per-sample trackHub/bw directory. It is NOT in the lab's own hub. # Present for all 41 samples, 141 GB, added to the file list in the download # and check scripts and mirrored with a plain re-run: ~/kent/src/hg/makeDb/scripts/fiberSeq/fiberSeqDownload.sh \ /hive/data/genomes/hg38/bed/fiberSeq 12 > download.nucleosome.log 2>&1 -# It is read depth, not a percentage, which makes it the one wiggle here that -# cannot take fixed viewLimits: the genome-wide mean runs from 25 (PS00971) to -# 142 (PM00001) with sequencing depth, and single loci reach 1.7e5. Drawn with -# autoScale on, so the axis is comparable within a sample and not between them. +# The lab then asked us to hold the display back while they settle internally +# on how they want nucleosomes shown, and to keep the track set on FIRE peaks +# and CpG methylation. So the files are mirrored and checked but no subtrack +# is generated: INCLUDE_NUC in fiberSeqTrackDb.py is False. Setting it True and +# regenerating is the whole of turning it on, no download needed. +# +# What that flag builds, when the lab comes back: it is read depth, not a +# percentage, which makes it the one wiggle here that cannot take fixed +# viewLimits. The genome-wide mean runs from 25 (PS00971) to 142 (PM00001) with +# sequencing depth, and single loci reach 1.7e5, so it is drawn with autoScale +# on and the axis is comparable within a sample but not between samples. +# +# The flag also moves the per-sample priorities, which is why they are assigned +# from a running counter rather than a fixed index per data type. With the flag +# off the generated fiberSeq.ra is byte-identical to the version before the +# nucleosome work, which is the check that the counter renumbers correctly. ~/kent/src/hg/makeDb/scripts/fiberSeq/fiberSeqCheck.sh /hive/data/genomes/hg38/bed/fiberSeq # no lines: all 492 files present, parsing and non-empty # --------------------------------------------------------------------------- # Rebuild the FIRE peak bigBeds # --------------------------------------------------------------------------- # The peak files carry narrowPeak data and even embed the narrowPeak autoSql, # but their bigBed header records a field count of 3: bigBedInfo -as PM00001/fire-peaks.bb | head -25 # fieldCount: 3 # definedFieldCount: 3 # extraFieldCount: 0 # as: table fire_peaks ... 10 columns ... @@ -200,49 +212,50 @@ # after the PM00001 reissue only that one rebuilt and the other 40 said "have". # That also means the run only prints one "got" line, and peakFixReport.txt has # to have PM00001's row replaced rather than being rewritten wholesale. # Chromosome naming is otherwise already UCSC style, no chromAlias needed: bigWigInfo -chroms PM00001/all.percent.accessible.bw | head -12 # chromCount 25, chr1 ... chrY (primary chromosomes only) bigWigInfo -chroms PM00001/cpg.combined.bw | head -12 # chromCount 195, primary plus alts and patches # --------------------------------------------------------------------------- # trackDb, metadata and colors # --------------------------------------------------------------------------- # One script writes the whole track stanza plus the two faceted-composite -# metadata and color files, so the 625 stanzas stay consistent: +# metadata and color files, so the 584 stanzas stay consistent: ~/kent/src/hg/makeDb/scripts/fiberSeq/fiberSeqTrackDb.py # It writes # ~/kent/src/hg/makeDb/trackDb/human/hg38/fiberSeq.ra # /hive/data/genomes/hg38/bed/fiberSeq/fiberSeqCompendium_metadata.tsv # /hive/data/genomes/hg38/bed/fiberSeq/fiberSeqCompendium_colors.json # Structure: # fiberSeq container, group regulation # fiberSeqAcc multiWig overlay of 7 common cell lines, shown by default -# fiberSeqCompendium faceted composite, 41 samples, seven data types: -# acc, peaks, hap, nuc, cpg, cpgHap, cpgDiff +# fiberSeqCompendium faceted composite, 41 samples, six data types: +# acc, peaks, hap, cpg, cpgHap, cpgDiff +# (a seventh, nuc, is built but held back: see below) # # Accessibility and methylation started as two composites, fiberSeqCompendium and # fiberSeqMeth, and were merged into one. They cover the identical 41 samples, # and cartDump.c assigns priority with the data element as the OUTER loop and the -# data type as the inner one, so a single composite keeps a sample's seven subtracks +# data type as the inner one, so a single composite keeps a sample's six subtracks # contiguous in the image. As two composites the display was an accessibility # block followed by a methylation block, so comparing the two assays for one # sample meant reading across every other sample - which is the whole point, # since both come off the same molecules in the same experiment. # # The composite uses the same faceted-composite machinery as Methbase # (methbase2.ra): metaDataUrl for the sample table, colorSettingsUrl for the # facet swatches, primaryKey for the row id, and dataTypes for the per-sample # data kinds. Two things about that machinery drove the naming: # # 1. Subtracks must be named <composite>_<accession>_<dataType>, with the # accession as the ONLY middle component. facetedCompositeUi() in # hg/hgTrackUi/hgTrackUi.c cuts the data element at the first underscore # after the composite name, and cartDump.c rebuilds subtrack names as # <composite>_<element>_<type>. The lab's hub used