0b36c1276f7fd41e8ba9ad53bbdb20c1d438913d braney Tue Aug 25 12:08:14 2026 -0700 trackDb: build the trix once per database, and run the beta make in parallel, refs #35489 The trackDb make for a final build takes about 18 minutes. It is not one slow step. It is 348 per-database chains of short processes and network round trips, run strictly one after another. Two changes. First, stop building the same trix twice. The beta recipe called buildTrix once for hgwbeta and again for hgwdev, with identical inputs and only the destination differing. buildTrix now takes -alsoTo=machine:path, so the index is built once and sent to both places. It also assembles the files in a temporary directory under their final names, so each destination takes one rsync instead of one per file. That takes the trix step from ten ssh connections per database down to two, and it no longer leaves .offsets files behind in the trackDb directory. Second, run the make in parallel. The header comment warning against this was out of date. hgTrackDb and hgFindSpec put their temporary files in TMPDIR under process-unique names, and every other file and table a recipe writes is already named after its database, so the per-database chains do not collide. Two makes at once are still unsafe, because they share table names, and the comment now says that instead. makeStrictBeta.csh passes -O -j 8. Set TRACKDB_MAKE_JOBS to change the job count. Going much above 8 wants ssh connection sharing first, or hgwbeta's sshd starts refusing connections. Measured over all 348 databases, with the output compared line by line against a serial run each time: 321 seconds today, 194 with the tdbQuery fix alone, 81 with -j 8 alone, and 39 with both. The -j 8 figure lands on the hg38 chain, which is why the tdbQuery fix matters more than its share of the total suggests. diff --git src/hg/makeDb/trackDb/makefile src/hg/makeDb/trackDb/makefile index 225ee16c300..04c2c66a64c 100644 --- src/hg/makeDb/trackDb/makefile +++ src/hg/makeDb/trackDb/makefile @@ -1,28 +1,34 @@ kentSrc = ../../.. # Make your private trackDb with # make update # Your private trackDb with additional options, for example: # make EXTRA="-strict -settings" update # Make it for genome-test with # make alpha # Make for ENCODE reporting (includes release alpha and release beta) # make encodeReport -# DO NOT RUN MULTIPLE INSTANCES OF THIS SCRIPT -# Because it creates temporary files in the current directory -# before loading them into the database. +# DO NOT RUN TWO OF THESE MAKES AT THE SAME TIME +# Two makes using the same table names will fight over the same tables. +# +# Running ONE make with -j is fine, and is much faster, since the databases are +# independent of each other. Every per-database recipe writes only to files and +# tables named after that database, and hgTrackDb and hgFindSpec put their +# temporary files in $$TMPDIR under process-unique names. Pass -O as well so +# that the output of each database stays together in the log: +# make -O -j 8 beta # Browser supports multiple trackDb's so that individual developers # can change things rapidly without stepping on other people's toes. # Usually when updating it is best to update your own trackDb and # test it to make sure it works and that you have git updated all # of trackDb/ before doing a make alpha. Note that you # must specify which trackDb you are using in your .hg.conf file # or in the cgi-bin-$(USER)/hg.conf file. Something like: # db.trackDb=trackDb_YourUserName # note: new group ARCHIVED_DBS created for assemblies still needed # to support Conservation tracks in other assemblies. These should # have only chromInfo table on hgwbeta and RR. make will not rebuild # trackDb on hgwdev unless the db is specified explicitly on the # command line. @@ -261,33 +267,33 @@ ${GIT} pull ${MAKE} beta_all beta_all: ${DBS:%=%_beta} %_betaTest: # now do loads on hgwbeta HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta ./loadTracks -addVersion ${EXTRA} -strict -remoteLogin=${BETA_MACHINE} -release=beta trackDb_betaTest hgFindSpec_betaTest $* %_beta: # now do loads on hgwbeta -ssh ${BETA_MACHINE} find /gbdb/$* -type f -size +1c > $*.gbdbList.txt HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta ./loadTracks -addVersion ${EXTRA} -strict -gbdbList=$*.gbdbList.txt -release=beta trackDb hgFindSpec $* rm -f $*.gbdbList.txt HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta ./checkMetaDb beta metaDb $* - HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta HGDB_TRACKDB=trackDb ${BUILD_TRIX} trackDb metaDb cv/beta/cv.ra ${BETA_MACHINE} ${DATA_TRIX} $* - # now make files for hgwdev-beta (we share the tables with hgwbeta) - HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta HGDB_TRACKDB=trackDb ${BUILD_TRIX} trackDb metaDb cv/beta/cv.ra ${ALPHA_MACHINE} ${HIVE_TRIX} $* + # the -alsoTo also makes the files for hgwdev-beta (we share the tables with + # hgwbeta), without building the same index a second time + HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta HGDB_TRACKDB=trackDb ${BUILD_TRIX} -alsoTo=${ALPHA_MACHINE}:${HIVE_TRIX} trackDb metaDb cv/beta/cv.ra ${BETA_MACHINE} ${DATA_TRIX} $* # now build the relatedTracks table for each db HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta ./buildRelatedTracks relatedTrack $* -ssh ${BETA_MACHINE} rm -rf /dev/shm/trackDbCache/$*.trackDb HGDB_CONF=/cluster/home/${USER}/.hg.conf.beta HGDB_TRACKDB=trackDb ./maybeBuildHub beta trackDb $* publicTest: onbeta clean ${MAKE} publicTest_all publicTest_all: ${DBS:%=%_publicTest} public: onbeta clean ${GIT} pull ${MAKE} public_all public_all: ${DBS:%=%_public}