a29276ca751e3fa5f0fc03784014b56e74aa840a max Tue Aug 11 05:03:46 2026 -0700 adding makedoc for cactus447 update, refs #37841 diff --git src/hg/makeDb/doc/hg38/cactus447.txt src/hg/makeDb/doc/hg38/cactus447.txt index 4ba80582206..e9547c4f477 100644 --- src/hg/makeDb/doc/hg38/cactus447.txt +++ src/hg/makeDb/doc/hg38/cactus447.txt @@ -1472,57 +1472,82 @@ # not to patch our size integers: the corrected file is a fully regenerated # alignment whose blocks do not correspond 1:1 to the old ones. # corrected published source (single-copy, hg38-referenced, .tai indexed): # https://cgl.gi.ucsc.edu/data/cactus/447-mammalian-2022v1.fix2.single.maf.gz # all work under: WORK=/hive/data/genomes/hg38/bed/cactus447fix2 # Per-chrom slices were pulled from the indexed source by contig byte-range. # Only the reference row is renamed accession->UCSC via chromAlias; the 446 # query rows keep their cactus Species.scaffold names. Then iRows added: # mafAddIRows -nBeds=nBeds <chrom>.maf hg38.2bit <chrom>.iRows.maf # final per-chrom iRows mafs (also the per-chrom download source): # $WORK/iRows/result/chrN.iRows.maf.gz - # bigMaf bb: per-chrom builds with uniform genome-wide chrom numbering and - # zoom schedule, so they stitch without re-sorting the ~1.5 TB of block text - # (bedToBigBed -globalChromIds/-cat/-zoomInitial, added to - # src/utils/bedToBigBed/bedToBigBed.c this cycle - its own commit/ticket). - # per chrom: + # bigMaf bb: built the 2023 way with stock tools - per-chrom mafToBigMaf, + # then concatenate, sort genome-wide, and one bedToBigBed. per chrom: # mafToBigMaf hg38 <chrom>.iRows.maf stdout | sort -k1,1 -k2,2n > <chrom>.bigMaf - # bedToBigBed -globalChromIds -zoomInitial=147 -itemsPerSlot=4 -type=bed3+1 \ - # -as=$HOME/kent/src/hg/lib/bigMaf.as -tab \ - # <chrom>.bigMaf /hive/data/genomes/hg38/chrom.sizes <chrom>.bb - # stitch all 194 per-chrom bb into the genome bb (no re-parse / no re-deflate): - # bedToBigBed -cat=allbb.list $WORK/hg38.cactus447way.bb - # -> 1.48 TB, itemCount 97,277,175, chromCount 711, basesCovered 3,099,750,718 + # merge the (already-sorted) per-chrom files, then build the genome bb: + # sort -m -k1,1 -k2,2n $WORK/*.bigMaf > $WORK/hg38.sorted447way.bigMaf + # bedToBigBed -verbose=2 -itemsPerSlot=4 -type=bed3+1 \ + # -as=$HOME/kent/src/hg/lib/bigMaf.as -tab $WORK/hg38.sorted447way.bigMaf \ + # /hive/data/genomes/hg38/chrom.sizes $WORK/hg38.cactus447way.bb + # -> itemCount 97,277,175, basesCovered 3,099,750,718 + # + # NOTE on chromCount (711 vs ~194): bigBedInfo on the currently-staged + # hg38.cactus447way.bb reports chromCount 711, which is every sequence listed + # in hg38 chrom.sizes, not only the ~194 chroms that actually carry alignment + # blocks. A plain bedToBigBed as documented above indexes only the chroms that + # have data, so rebuilding it gives chromCount ~194. This is a chrom-B-tree + # bookkeeping difference only: the alignment is identical either way (same + # itemCount 97,277,175, same basesCovered), and the empty chroms in the B-tree + # do not affect display or queries. If an exact chromCount match to this doc is + # wanted, rebuild the bb with the command above. # summary bb: replicate the 2023 summary pipeline with the FIXED # hgLoadMafSummary. Build and run it from ~/bin/$MACHTYPE, NOT the bare name - # the PATH resolves to the old production binary. Script: $WORK/summary/summaryAll.sh # per chrom: # hgLoadMafSummary -minSize=30000 -mergeGap=1500 -maxSize=200000 -test \ # hg38 <chrom> <chrom>.maf ; cut -f2- <chrom>.tab | sort -k1,1 -k2,2n # then: # bedToBigBed -type=bed3+4 -as=$HOME/kent/src/hg/lib/mafSummary.as -tab \ # cactus447Summary.bed /hive/data/genomes/hg38/chrom.sizes cactus447waySummary.bb # -> $WORK/summary/cactus447waySummary.bb # 1.03 GB, itemCount 74,427,201, maxDepth 446 (= 447 - hg38 reference) # VERIFICATION (all pass): # old chrY size!=nongap blocks: 94 (232 s-lines) ; new: 0 # summary src column: 446 clean Genus_species names genome-wide (chrY and # chr1 both), 0 with a residual dot. This is the hgLoadMafSummary # name-split fix, refs #37928 (commit 2db6bab8db0, jkweb move 2070aedea88). # genome bb itemCount == exact sum of the 194 per-chrom itemCounts. # STAGING (reversible; genome-test/hgwdev only - NOT pushed to public): # /usr/local/apache/htdocs-hgdownload/goldenPath/hg38/cactus447way/ # {hg38.cactus447way.bb, cactus447waySummary.bb, maf/} symlinked -> fix2 # rollback: $WORK/swap_rollback.sh # trackDb (human/hg38/trackDb.447way.ra, track cactus447way) already points # bigDataUrl/summary at hgdownload.soe.ucsc.edu; the corrected files reach the # live browser only after the build/QA rsync pushes fix2 to the public server. - # frames bb (cactus447wayFrames.bb) not rebuilt (derived; reassess if needed). + # frames bb (cactus447wayFrames.bb, DONE 2026-08-10): the 2023 frames were + # built from the old blocks, so rebuilt for fix2. The per-species gene + # predictions are gene annotations, unchanged by the re-alignment, so reuse the + # 2023 genes/ set (44 gene assemblies). genePredToMafFrames accepts all gene + # sets in one call and spends ~95% of its time reading the MAF, so read each + # chrom once (one call, all gene sets) instead of per (chrom x gene). Ran on + # hgwdev, 8 chroms in parallel (script $WORK/frames/framesAll.sh): + # GENES=/hive/data/genomes/hg38/bed/cactus447/frames/genes + # GENEARGS="" ; for gp in $GENES/*.gp.gz; do g=$(basename $gp .gp.gz); \ + # GENEARGS="$GENEARGS $g $gp"; done + # # per chrom: + # zcat $WORK/iRows/result/<chrom>.iRows.maf.gz \ + # | genePredToMafFrames hg38 stdin stdout $GENEARGS \ + # | gzip > parts/<chrom>.mafFrames.gz + # find parts -name '*.mafFrames.gz' | while read F; do zcat "$F" | awk '11==NF'; done \ + # | sort -k1,1 -k2,2n > cactus447wayFrames.bed + # bedToBigBed -type=bed3+7 -as=$HOME/kent/src/hg/lib/mafFrames.as -tab \ + # cactus447wayFrames.bed /hive/data/genomes/hg38/chrom.sizes cactus447wayFrames.bb + # -> 168 MB, itemCount 16,098,320, basesCovered 66,595,568 (2023: ~16,063,019) #############################################################################