0b36c1276f7fd41e8ba9ad53bbdb20c1d438913d
braney
  Tue Aug 25 12:08:14 2026 -0700
trackDb: build the trix once per database, and run the beta make in parallel, refs #35489

The trackDb make for a final build takes about 18 minutes.  It is not one
slow step.  It is 348 per-database chains of short processes and network
round trips, run strictly one after another.

Two changes.

First, stop building the same trix twice.  The beta recipe called buildTrix
once for hgwbeta and again for hgwdev, with identical inputs and only the
destination differing.  buildTrix now takes -alsoTo=machine:path, so the
index is built once and sent to both places.  It also assembles the files in
a temporary directory under their final names, so each destination takes one
rsync instead of one per file.  That takes the trix step from ten ssh
connections per database down to two, and it no longer leaves .offsets files
behind in the trackDb directory.

Second, run the make in parallel.  The header comment warning against this
was out of date.  hgTrackDb and hgFindSpec put their temporary files in
TMPDIR under process-unique names, and every other file and table a recipe
writes is already named after its database, so the per-database chains do not
collide.  Two makes at once are still unsafe, because they share table
names, and the comment now says that instead.

makeStrictBeta.csh passes -O -j 8.  Set TRACKDB_MAKE_JOBS to change the job
count.  Going much above 8 wants ssh connection sharing first, or hgwbeta's
sshd starts refusing connections.

Measured over all 348 databases, with the output compared line by line
against a serial run each time: 321 seconds today, 194 with the tdbQuery fix
alone, 81 with -j 8 alone, and 39 with both.  The -j 8 figure lands on the
hg38 chain, which is why the tdbQuery fix matters more than its share of the
total suggests.

diff --git src/hg/makeDb/trackDb/makefile src/hg/makeDb/trackDb/makefile
index 225ee16c300..04c2c66a64c 100644
--- src/hg/makeDb/trackDb/makefile
+++ src/hg/makeDb/trackDb/makefile
@@ -1,28 +1,34 @@
 kentSrc = ../../..
 # Make your private trackDb with 
 #	make update
 # Your private trackDb with additional options, for example:
 #       make EXTRA="-strict -settings" update
 # Make it for genome-test with
 #	make alpha
 # Make for ENCODE reporting (includes release alpha and release beta)
 #	make encodeReport
 
-# DO NOT RUN MULTIPLE INSTANCES OF THIS SCRIPT 
-# Because it creates temporary files in the current directory 
-# before loading them into the database.
+# DO NOT RUN TWO OF THESE MAKES AT THE SAME TIME
+# Two makes using the same table names will fight over the same tables.
+#
+# Running ONE make with -j is fine, and is much faster, since the databases are
+# independent of each other.  Every per-database recipe writes only to files and
+# tables named after that database, and hgTrackDb and hgFindSpec put their
+# temporary files in $$TMPDIR under process-unique names.  Pass -O as well so
+# that the output of each database stays together in the log:
+#	make -O -j 8 beta
 
 # Browser supports multiple trackDb's so that individual developers
 # can change things rapidly without stepping on other people's toes. 
 # Usually when updating it is best to update your own trackDb and
 # test it to make sure it works and that you have git updated all
 # of trackDb/ before doing a make alpha. Note that you
 # must specify which trackDb you are using in your .hg.conf file
 # or in the cgi-bin-$(USER)/hg.conf file. Something like: 
 # db.trackDb=trackDb_YourUserName
 
 # note:  new group ARCHIVED_DBS created for assemblies still needed
 # to support Conservation tracks in other assemblies.  These should
 # have only chromInfo table on hgwbeta and RR.  make will not rebuild
 # trackDb on hgwdev unless the db is specified explicitly on the 
 # command line.
@@ -261,33 +267,33 @@
 	${GIT} pull
 	${MAKE} beta_all
 
 beta_all: ${DBS:%=%_beta}
 
 %_betaTest:
 	# now do loads on hgwbeta
 	HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta ./loadTracks -addVersion ${EXTRA} -strict -remoteLogin=${BETA_MACHINE} -release=beta trackDb_betaTest hgFindSpec_betaTest $*
 
 %_beta:
 	# now do loads on hgwbeta
 	-ssh  ${BETA_MACHINE} find /gbdb/$*  -type f -size +1c  > $*.gbdbList.txt
 	HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta ./loadTracks -addVersion ${EXTRA} -strict -gbdbList=$*.gbdbList.txt -release=beta trackDb hgFindSpec $*
 	rm  -f $*.gbdbList.txt
 	HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta ./checkMetaDb beta metaDb $*
-	HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta HGDB_TRACKDB=trackDb ${BUILD_TRIX} trackDb metaDb cv/beta/cv.ra ${BETA_MACHINE} ${DATA_TRIX} $*
-	# now make files for hgwdev-beta (we share the tables with hgwbeta)
-	HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta HGDB_TRACKDB=trackDb ${BUILD_TRIX} trackDb metaDb cv/beta/cv.ra ${ALPHA_MACHINE} ${HIVE_TRIX} $*
+	# the -alsoTo also makes the files for hgwdev-beta (we share the tables with
+	# hgwbeta), without building the same index a second time
+	HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta HGDB_TRACKDB=trackDb ${BUILD_TRIX} -alsoTo=${ALPHA_MACHINE}:${HIVE_TRIX} trackDb metaDb cv/beta/cv.ra ${BETA_MACHINE} ${DATA_TRIX} $*
 	# now build the relatedTracks table for each db
 	HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta ./buildRelatedTracks relatedTrack $*
 	-ssh  ${BETA_MACHINE} rm -rf /dev/shm/trackDbCache/$*.trackDb
 	HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta HGDB_TRACKDB=trackDb ./maybeBuildHub  beta trackDb $*
 
 publicTest: onbeta clean
 	${MAKE} publicTest_all
 
 publicTest_all: ${DBS:%=%_publicTest}
 
 public: onbeta clean
 	${GIT} pull
 	${MAKE} public_all
 
 public_all: ${DBS:%=%_public}