0b36c1276f7fd41e8ba9ad53bbdb20c1d438913d
braney
  Tue Aug 25 12:08:14 2026 -0700
trackDb: build the trix once per database, and run the beta make in parallel, refs #35489

The trackDb make for a final build takes about 18 minutes.  It is not one
slow step.  It is 348 per-database chains of short processes and network
round trips, run strictly one after another.

Two changes.

First, stop building the same trix twice.  The beta recipe called buildTrix
once for hgwbeta and again for hgwdev, with identical inputs and only the
destination differing.  buildTrix now takes -alsoTo=machine:path, so the
index is built once and sent to both places.  It also assembles the files in
a temporary directory under their final names, so each destination takes one
rsync instead of one per file.  That takes the trix step from ten ssh
connections per database down to two, and it no longer leaves .offsets files
behind in the trackDb directory.

Second, run the make in parallel.  The header comment warning against this
was out of date.  hgTrackDb and hgFindSpec put their temporary files in
TMPDIR under process-unique names, and every other file and table a recipe
writes is already named after its database, so the per-database chains do not
collide.  Two makes at once are still unsafe, because they share table
names, and the comment now says that instead.

makeStrictBeta.csh passes -O -j 8.  Set TRACKDB_MAKE_JOBS to change the job
count.  Going much above 8 wants ssh connection sharing first, or hgwbeta's
sshd starts refusing connections.

Measured over all 348 databases, with the output compared line by line
against a serial run each time: 321 seconds today, 194 with the tdbQuery fix
alone, 81 with -j 8 alone, and 39 with both.  The -j 8 figure lands on the
hg38 chain, which is why the tdbQuery fix matters more than its share of the
total suggests.

diff --git src/utils/qa/weeklybld/makeStrictBeta.csh src/utils/qa/weeklybld/makeStrictBeta.csh
index b3655fd0851..9f7d48dd415 100755
--- src/utils/qa/weeklybld/makeStrictBeta.csh
+++ src/utils/qa/weeklybld/makeStrictBeta.csh
@@ -1,28 +1,34 @@
 #!/bin/tcsh
 
 cd $BUILDDIR
 set dir = "v"$BRANCHNN"_branch"
 cd $dir
 
 cd kent
 pwd
 # the makefile now does zoo automatically now when you call it
 echo "trackDb Make strict. [${0}: `date`]"
 cd $BUILDDIR/$dir/kent/src/hg/makeDb/trackDb
-#make -O -j 5 beta >& make.strict.log  # gotta fix the metaDb makefiles in trackDb first
-make beta >& make.strict.log
+# The databases are independent of each other, so run them in parallel.  -O keeps
+# each database's output together in the log.  Set TRACKDB_MAKE_JOBS to change the
+# job count; going much above 8 needs ssh connection sharing first, or hgwbeta's
+# sshd starts refusing connections (MaxStartups).
+set trackDbJobs = 8
+if ( $?TRACKDB_MAKE_JOBS ) set trackDbJobs = $TRACKDB_MAKE_JOBS
+echo "trackDb make with -j $trackDbJobs"
+make -O -j $trackDbJobs beta >& make.strict.log
 /bin/egrep -i "html missing" make.strict.log > warning.txt
 /bin/egrep -iv "html missing" make.strict.log > make.strict.log2
 mv make.strict.log2 make.strict.log
 set res = `/bin/egrep -i "error|warn" make.strict.log | grep -v ignored`
 set wc = `echo "$res" | wc -w` 
 if ( "$wc" != "0" ) then
    echo "trackDb strict errs found:"
    echo "$res"
    tail make.strict.log
    exit 1
 endif
 
 set wc = `cat warning.txt | wc -w`
 if ( "$wc" != "0" ) then
     echo "trackDb strict html non-fatal errs found:"