0b36c1276f7fd41e8ba9ad53bbdb20c1d438913d
braney
  Tue Aug 25 12:08:14 2026 -0700
trackDb: build the trix once per database, and run the beta make in parallel, refs #35489

The trackDb make for a final build takes about 18 minutes.  It is not one
slow step.  It is 348 per-database chains of short processes and network
round trips, run strictly one after another.

Two changes.

First, stop building the same trix twice.  The beta recipe called buildTrix
once for hgwbeta and again for hgwdev, with identical inputs and only the
destination differing.  buildTrix now takes -alsoTo=machine:path, so the
index is built once and sent to both places.  It also assembles the files in
a temporary directory under their final names, so each destination takes one
rsync instead of one per file.  That takes the trix step from ten ssh
connections per database down to two, and it no longer leaves .offsets files
behind in the trackDb directory.

Second, run the make in parallel.  The header comment warning against this
was out of date.  hgTrackDb and hgFindSpec put their temporary files in
TMPDIR under process-unique names, and every other file and table a recipe
writes is already named after its database, so the per-database chains do not
collide.  Two makes at once are still unsafe, because they share table
names, and the comment now says that instead.

makeStrictBeta.csh passes -O -j 8.  Set TRACKDB_MAKE_JOBS to change the job
count.  Going much above 8 wants ssh connection sharing first, or hgwbeta's
sshd starts refusing connections.

Measured over all 348 databases, with the output compared line by line
against a serial run each time: 321 seconds today, 194 with the tdbQuery fix
alone, 81 with -j 8 alone, and 39 with both.  The -j 8 figure lands on the
hg38 chain, which is why the tdbQuery fix matters more than its share of the
total suggests.

diff --git src/hg/makeDb/trackDb/buildTrix src/hg/makeDb/trackDb/buildTrix
index 07726a675a7..a4c709e5842 100755
--- src/hg/makeDb/trackDb/buildTrix
+++ src/hg/makeDb/trackDb/buildTrix
@@ -1,58 +1,76 @@
 #!/bin/bash -e
-usage='buildTrix trixName metaDbName path_to_cv.ra outputMachine outputPath ${DBS}'
+usage='buildTrix [options] trixName metaDbName path_to_cv.ra outputMachine outputPath ${DBS}
+
+options:
+   -alsoTo=machine:path -- copy the same files to another machine and path as well.
+                           May be given more than once.  Using this instead of a second
+                           invocation avoids rebuilding the index from scratch.'
+
+dests=()
+while [[ $1 == -* ]] ; do
+    case $1 in
+        -alsoTo=*)
+            dests+=("${1#-alsoTo=}") ;;
+        *)
+            echo "invalid option: $1" >&2
+            exit 1 ;;
+    esac
+    shift
+done
 
 if [ $# -lt 6 ] ; then
     echo "wrong # args: $usage" >&2
     exit 1
 fi
 
 trixName="$1"; shift
 metaDbName="$1" ; shift
 cvRaPath="$1" ; shift
 outMachine="$1" ; shift
 outPath="$1" ; shift
 dbs="$@"
 
+dests=("$outMachine:$outPath" "${dests[@]}")
+
 # check if a database exists, print note and return non-zero if it doesn't
 dbExists() {
     local db="$1"
     local dbChk=$(/cluster/bin/x86_64/hgsql -Ne 'show databases like "'$db'"')
     if [ -z "$dbChk" ] ; then
         echo "Note: database $db does not exist, skipping"
         return 1
     else
         return 0
     fi
 }
 
 buildDbTrix() {
     local db="$1"
     local trixName="$2"
     local metaDbName="$3"
     local cvRaPath="$4"
-    local outPath="$5"
-    local tmpFile=`mktemp`;
-    local tmpFile2=`mktemp`;
-    local tmpFile3=`mktemp`;
-    /cluster/bin/x86_64/makeTrackIndex $db $metaDbName $cvRaPath > $tmpFile;
-    if test -s $tmpFile; then
-        /cluster/bin/x86_64/ixIxx -maxWordLength=64 $tmpFile $tmpFile2 $tmpFile3
+    # Build into a temporary directory using the final file names, so that each
+    # destination takes one rsync instead of one per file.  That is five fewer ssh
+    # connections per destination, which matters when this make runs in parallel.
+    local tmpDir=`mktemp -d`;
+    local base="$tmpDir/${db}_${trixName}"
+    /cluster/bin/x86_64/makeTrackIndex $db $metaDbName $cvRaPath > $base.txt;
+    if test -s $base.txt; then
+        /cluster/bin/x86_64/ixIxx -maxWordLength=64 $base.txt $base.ix $base.ixx
         # trixContextIndex makes the $db_$trixName.offsets and $db_$trixName.offsets.ixx files:
-        /cluster/bin/x86_64/trixContextIndex $tmpFile ${db}_${trixName}
-        chmod 664  $tmpFile2 $tmpFile3 $tmpFile ${db}_${trixName}.offsets ${db}_${trixName}.offsets.ixx
-        rsync -a $tmpFile2 $outMachine:$outPath/${db}_${trixName}.ix
-        rsync -a $tmpFile3 $outMachine:$outPath/${db}_${trixName}.ixx
-        # these three are for snippets on search pages:
-        rsync -a $tmpFile $outMachine:$outPath/${db}_${trixName}.txt
-        rsync -a ${db}_${trixName}.offsets $outMachine:$outPath/${db}_${trixName}.offsets
-        rsync -a ${db}_${trixName}.offsets.ixx $outMachine:$outPath/${db}_${trixName}.offsets.ixx
-        rm -f ${db}_${trixName}.offsets ${db}_${trixName}.offsets.ixx
+        /cluster/bin/x86_64/trixContextIndex $base.txt $base
+        chmod 664 $base.txt $base.ix $base.ixx $base.offsets $base.offsets.ixx
+        # the .txt, .offsets and .offsets.ixx files are for snippets on search pages:
+        local dest
+        for dest in "${dests[@]}"; do
+            rsync -a $tmpDir/ "${dest%%:*}:${dest#*:}/"
+        done
     fi
-    rm -f $tmpFile $tmpFile2 $tmpFile3
+    rm -rf $tmpDir
 }
 
 for db in $dbs ; do
     if dbExists $db ; then
-        buildDbTrix $db $trixName $metaDbName $cvRaPath $outPath
+        buildDbTrix $db $trixName $metaDbName $cvRaPath
     fi
 done