af126b8f0cf9ddc553a78869a3c820a4bd4fdc59
braney
  Mon Aug 17 13:36:13 2026 -0700
build: link the browser against zlib-ng instead of the system zlib, refs #38125

Adds zlib-ng as a second git submodule, alongside htslib, and points ZLIB at it.
It is built with --zlib-compat, so the header and the symbol names are the
ordinary zlib ones and no calling code has to change.

Measured over the eight Recommended Track Set pages, hgTracks uses 29 percent
less processor time and 26 percent less wall clock, and the image it produces is
pixel identical. Writing the PNG is about three times faster and reading a
bigBed data block about twice as fast, which is why the gain is larger than the
image work alone would give.

This also makes the platforms agree. x86_64 was linking a static /lib64/libz.a
only because that file happened to be present, the arm64 Docker build fell
through to a shared -lz, and Darwin was taking MacPorts zlib. All three now use
the submodule.

Verified on x86_64 Linux, aarch64 Linux and Apple silicon: zlib-ng builds, its
own test suite passes, the makefiles pick it up, and bigBedToBed output is byte
identical to before the change.

diff --git src/submodules/submoduleSetup src/submodules/submoduleSetup
index 353802f8111..b3d12f06d6a 100755
--- src/submodules/submoduleSetup
+++ src/submodules/submoduleSetup
@@ -6,38 +6,65 @@
 # having remains of old htslib copy to deal with switch from old version.
 
 if [ -e htslib ] ; then
     echo "*********************************************************************" >&2
     echo "Note: an outdated kent/src/htslib/ exists in the working tree." >&2
     echo "      It is no longer used and can be removed."   >&2
     echo "*********************************************************************" >&2
 fi
 
 if git rev-parse --is-inside-work-tree >/dev/null 2>&1 ; then
     # Sync submodules to the commits pinned by the parent repo on every build,
     # not just on first-time init, so a moved pointer (e.g. after a pull) is
     # always checked out.  --init covers the first-time case, --recursive also
     # updates the nested htscodecs submodule.  git is silent when already in sync.
     git -c protocol.file.allow=always submodule update --init --recursive
-elif [ ! -e submodules/htslib/hts.c -o ! -e submodules/htslib/htscodecs/htscodecs/htscodecs.c ] ; then
+elif [ ! -e submodules/htslib/hts.c -o ! -e submodules/htslib/htscodecs/htscodecs/htscodecs.c \
+       -o ! -e submodules/zlib-ng/deflate.c ] ; then
     # not a git working tree (e.g. a source release); submodule sources must
     # already be present since they can not be checked out without git.
-    echo "Error: submodule sources under submodules/htslib are missing and this" >&2
+    echo "Error: submodule sources under submodules/ are missing and this" >&2
     echo "       is not a git working tree, so they can not be initialized." >&2
     exit 1
 fi
 
+# Configure zlib-ng, which has no Makefile until its configure script runs.
+# --zlib-compat gives the ordinary zlib API and symbol names, so libpng, htslib
+# and our own code need no change.  --static builds only libz.a, which is all we
+# link.  configure compiles every SIMD variant and picks one at run time, so a
+# single build serves both the AVX-512 machines and the AVX2 ones, and on ARM it
+# picks up NEON and the ARMv8 CRC instructions.  Done here, serially, for the
+# same reason as the htslib probe below: the parallel build must not race on it.
+# refs #38125
+if [ ! -e submodules/zlib-ng/Makefile ] ; then
+    echo "Note: configuring zlib-ng (serial, once)" >&2
+    # log kept outside the submodule so it does not show up as untracked there
+    (cd submodules/zlib-ng && ./configure --zlib-compat --static) \
+        > submodules/zlib-ng-configure.log 2>&1 \
+        || { echo "Error: zlib-ng configure failed, see submodules/zlib-ng-configure.log" >&2 ; exit 1 ; }
+    # zlib-ng's own .gitignore does not cover everything its configure writes, so
+    # ignore the rest locally rather than carrying a patch to upstream.  This is
+    # the submodule's private exclude file, not a tracked one.
+    zngExclude=$(cd submodules/zlib-ng && git rev-parse --git-path info/exclude 2>/dev/null || true)
+    if [ -n "$zngExclude" ] ; then
+        mkdir -p "$(dirname "$zngExclude")" 2>/dev/null || true
+        for f in zlib.h zlib_name_mangling.h test.c ; do
+            grep -qxF "$f" "$zngExclude" 2>/dev/null || echo "$f" >> "$zngExclude"
+        done
+    fi
+fi
+
 # Generate htslib's compiler-probe makefile fragment (htscodecs.mk) and config.h
 # SERIALLY, here, before the parallel ("make -j") build descends into htslib.
 # hts_probe_cc.sh probes SIMD (SSE4/AVX2/AVX512) support using fixed-name
 # conftest.* temp files in submodules/htslib/.  Under -j, a concurrent writer
 # can clobber conftest.c between its creation and compilation, so the AVX2 probe
 # "succeeds" with no -mavx2 flag; that records HTS_BUILD_AVX2=1 with an empty
 # HTS_CFLAGS_AVX2 in htscodecs.mk.  config.h then defines HAVE_AVX2, enabling
 # AVX2 source that is compiled without -mavx2 -> "target specific option
 # mismatch" build failure.  Both files are no-prerequisite make targets, so the
 # corrupt result is cached and reused until deleted.  Building them single-
 # threaded here guarantees the probe runs exactly once and the -j build never
 # triggers it.  Only done when they are missing, so it does not force needless
 # htslib recompiles on incremental builds.
 if [ -e submodules/htslib/Makefile ] && \
    { [ ! -e submodules/htslib/htscodecs.mk ] || [ ! -e submodules/htslib/config.h ]; } ; then