adca1531bedb1f883c7f46f37bfcf690858050c2
braney
  Wed Aug 26 08:15:31 2026 -0700
trackDb: rsync the trix files by name, not the directory they sit in, refs #35489

The beta make died on the first database with:

rsync: [generator] failed to set times on "/data/trix/.":
Operation not permitted (1)
rsync error: some files/attrs were not transferred (code 23)

My change yesterday replaced five per-file rsyncs with one rsync of the
temporary directory.  A trailing slash on the source means "the contents of
this directory", but rsync still puts the directory itself in the file list.
So -a gets applied to the destination directory too, and rsync tries to give
/data/trix the temporary directory's timestamp.  The build user does not own
/data/trix, the utime call fails, and rsync exits 23.

strace shows the extra call the file list form never makes:

dir source:  utimensat(AT_FDCWD, ".", [... 2019-09-09 ...]) = 0
file list:   no call on "."

The -alsoTo destination on hgwdev would have failed the same way, since the
build user does not own /hive/data/inside/trix either.

Name the five files as arguments, the way the code did before.  That is still
one rsync and one ssh connection per destination, so the speedup stays.

diff --git src/hg/makeDb/trackDb/buildTrix src/hg/makeDb/trackDb/buildTrix
index a4c709e5842..3fb75ece067 100755
--- src/hg/makeDb/trackDb/buildTrix
+++ src/hg/makeDb/trackDb/buildTrix
@@ -1,76 +1,80 @@
 #!/bin/bash -e
 usage='buildTrix [options] trixName metaDbName path_to_cv.ra outputMachine outputPath ${DBS}
 
 options:
    -alsoTo=machine:path -- copy the same files to another machine and path as well.
                            May be given more than once.  Using this instead of a second
                            invocation avoids rebuilding the index from scratch.'
 
 dests=()
 while [[ $1 == -* ]] ; do
     case $1 in
         -alsoTo=*)
             dests+=("${1#-alsoTo=}") ;;
         *)
             echo "invalid option: $1" >&2
             exit 1 ;;
     esac
     shift
 done
 
 if [ $# -lt 6 ] ; then
     echo "wrong # args: $usage" >&2
     exit 1
 fi
 
 trixName="$1"; shift
 metaDbName="$1" ; shift
 cvRaPath="$1" ; shift
 outMachine="$1" ; shift
 outPath="$1" ; shift
 dbs="$@"
 
 dests=("$outMachine:$outPath" "${dests[@]}")
 
 # check if a database exists, print note and return non-zero if it doesn't
 dbExists() {
     local db="$1"
     local dbChk=$(/cluster/bin/x86_64/hgsql -Ne 'show databases like "'$db'"')
     if [ -z "$dbChk" ] ; then
         echo "Note: database $db does not exist, skipping"
         return 1
     else
         return 0
     fi
 }
 
 buildDbTrix() {
     local db="$1"
     local trixName="$2"
     local metaDbName="$3"
     local cvRaPath="$4"
     # Build into a temporary directory using the final file names, so that each
     # destination takes one rsync instead of one per file.  That is five fewer ssh
     # connections per destination, which matters when this make runs in parallel.
     local tmpDir=`mktemp -d`;
     local base="$tmpDir/${db}_${trixName}"
     /cluster/bin/x86_64/makeTrackIndex $db $metaDbName $cvRaPath > $base.txt;
     if test -s $base.txt; then
         /cluster/bin/x86_64/ixIxx -maxWordLength=64 $base.txt $base.ix $base.ixx
         # trixContextIndex makes the $db_$trixName.offsets and $db_$trixName.offsets.ixx files:
         /cluster/bin/x86_64/trixContextIndex $base.txt $base
         chmod 664 $base.txt $base.ix $base.ixx $base.offsets $base.offsets.ixx
         # the .txt, .offsets and .offsets.ixx files are for snippets on search pages:
+        # Name the files explicitly rather than rsyncing $tmpDir/ itself.  With a
+        # directory as the source, rsync also applies -a to the destination
+        # directory, and setting its times fails unless we own it.
         local dest
         for dest in "${dests[@]}"; do
-            rsync -a $tmpDir/ "${dest%%:*}:${dest#*:}/"
+            rsync -a $base.txt $base.ix $base.ixx $base.offsets $base.offsets.ixx \
+                "${dest%%:*}:${dest#*:}/"
         done
     fi
     rm -rf $tmpDir
 }
 
 for db in $dbs ; do
     if dbExists $db ; then
         buildDbTrix $db $trixName $metaDbName $cvRaPath
     fi
 done