385cb594dbd4ff14124a64816d0a649458bff44d braney Wed Sep 2 12:15:29 2026 -0700 pngLevelBench: measure what each png compression level costs and saves, refs #38109 The compression level decision needs two numbers per level: the bytes the track image grows by, and the encode time it saves. A reader gains from a lower level only above B/S, which is a speed, so the two numbers turn each level into one break-even link speed to compare against the reader throughput the log reader reports. The encode matches lib/pngwrite.c exactly, RGBA with the row filter pinned to UP, and nothing is written to disk, so the time is the encode alone. The check that makes an offline measurement stand for what the CGI does is -check: at level 6, which is what libpng's default resolves to, this has to reproduce the file hgTracks wrote, byte for byte. It does, for every image tried. Also here are the three scripts around it: picking a traffic weighted corpus of real hgTracks URLs out of an access log, rendering them against a parked hgTracks, and aggregating the result into per level bytes, ms and break-even speed. The README gives the order they run in and the one known weakness, that a replayed URL renders the trackDb default track set rather than the reader's own. diff --git src/hg/oneShot/pngLevelBench/README src/hg/oneShot/pngLevelBench/README new file mode 100644 index 00000000000..348253b15dd --- /dev/null +++ src/hg/oneShot/pngLevelBench/README @@ -0,0 +1,67 @@ +pngLevelBench - what the png compression level costs and saves. Refs #38109. + +The decision in #38109 needs two numbers for every candidate zlib level: the +bytes the track image grows by (B) and the encode time it saves (S). A reader +gains from a lower level only when their throughput is above B / S, which is a +speed, so B and S turn each level into one break-even link speed. The reader +side of that comparison comes from hg/utils/pngTiming/pngTimingReport.py. + +Four pieces, in the order they run. + +1. pickUrls.py pick a corpus of real hgTracks URLs out of an access log. + + zcat /hive/data/inside/wwwstats/RR/2026/hgw1/access_log.20260823.gz \ + | grep -a '"GET /cgi-bin/hgTracks?' | awk '{print $7, $9}' | gzip > gets.gz + hgsql -N -e "show databases" > localDbs.txt + pickUrls.py gets.gz localDbs.txt corpusUrls.tsv 300 + + URLs are sampled in proportion to how often each was loaded, so the corpus + is traffic weighted. Custom track and hub URLs are dropped, since they + reach off the machine, and so are assemblies this machine does not have. + +2. render.sh render each URL against a parked hgTracks and keep the png. + + Set PORT to the ticket sandbox. It uses hgRenderTracks, which returns the + image itself rather than a page, and it sends an explicit pix= so the first + hit is a render and not addPixAndReloadPage(). + +3. pngLevelBench encode each corpus image at every level, 0 to 9. + + pngLevelBench -tab -reps=3 corpus/*.png > bench.tsv + pngLevelBench -check one.png # the validation, see below + + The encode matches lib/pngwrite.c exactly: RGBA, 8 bits, no interlace, row + filter pinned to UP. Only the level varies, and nothing is written to disk, + so the time is the encode alone. + +4. levelReport.py aggregate into per-level B, S and break-even speed. + + levelReport.py bench.tsv logSizes.tsv 6 + + logSizes.tsv is one real image size per line, from the same log: + + zcat access_log.*.gz | grep -a 'GET /trash/hgt/hgt_' \ + | awk '$9==200 && $10>1000 && $7 ~ /\.png/ {print $10}' > logSizes.tsv + + The `$7 ~ /\.png/` matters. The same directory holds the mouseover .json + files, which are small and numerous, and counting them drags every size + percentile down. + +What makes the numbers trustworthy, and what does not +----------------------------------------------------- +The validation is `-check`: at level 6, which is what libpng's default +resolves to, the encoder here must reproduce the file hgTracks wrote, byte for +byte. It did for all 290 images of the 2026-09-02 corpus. That is what lets +an offline measurement stand for what the CGI does. It also means the tool has +to link the same zlib the browser links: build it in a tree with the zlib-ng +submodule, and check `nm -D` shows a defined `deflate`. Linking the system +zlib instead (`make ZLIB=-lz`, supported by inc/common.mk:78) is how to measure +the difference between the two, and then the level 6 check will fail on purpose. + +The known weakness is the corpus, not the encoder. A replayed URL carries no +usable hgsid, so hgTracks renders the trackDb default track set rather than the +reader's own. The corpus therefore has a narrower size range than real +traffic: for the 2026-09-02 run, corpus p10 to p90 was 85 to 169 KB against 27 +to 326 KB in the log. levelReport.py reweights the corpus to match the log's +size histogram and says how much of the real traffic has no corpus image to +speak for it.