d48a1f1935917a45b20ae782f4a0ab53da13e6a9
braney
  Tue Sep 15 12:53:04 2026 -0700
hgTablesTest: skip an oversized page instead of dying inside the allocator, refs #38359

A dense file-backed track can hand back hundreds of megabytes for a single
five-megabyte test region.  hg38 hgdp returned 602MB, which took carefulAlloc
past its 500MB ceiling, and carefulAlloc exits the process where it stands
rather than errAborting, on the grounds that errAbort itself allocates.  So the
run ended with one line on stderr, nothing in the log, and every table still to
come forfeited.  The arm in quickSubmit meant to catch exactly this and name the
track had never once run.

htmlPage now takes an optional ceiling on the response it will read into memory.
Past it the fetch frees what it has read and errAborts naming the url, which the
robot's errCatch turns back into an ordinary return of no page.  The ceiling
defaults to none, which leaves hgNearTest, hgBlatTest and htmlCheck exactly as
they were.  hgTablesTest sets it to 100MB, a fifth of the allocator ceiling: the
dyString roughly doubles as it grows and the old buffer is still live while the
new one fills, and the parsed page then sits alongside its text.

An oversized page is logged and skipped, not counted as an error.  A track that
answers a 5Mb region with 600MB is one this robot cannot test, which is the same
situation the row count screen already catches before submitting; counting it
would put a failure in every weekly run and leave the summary as useless a gate
as the one that never failed.

The log is line buffered now as well.  Finding out that a run died partway
through is what this robot is for, and a block of buffered lines lost on the way
out is part of how the old failure left no trace of which track it was on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

diff --git src/lib/net.c src/lib/net.c
index 1063a4d7f80..82d7eed90b8 100644
--- src/lib/net.c
+++ src/lib/net.c
@@ -1535,43 +1535,61 @@
     else if (startsWith("ftp://",url))
 	return netGetOpenFtpSockets(url, retCtrlSocket);
     else    
 	errAbort("Sorry, can only netUrlOpen http, https and ftp currently, not '%s'", url);
     }
 return -1;    
 }
 
 int netUrlOpen(char *url)
 /* Return socket descriptor (low-level file handle) for read()ing url data,
  * or -1 if error.  Just close(result) when done. */
 {
 return netUrlOpenSockets(url, NULL);
 }
 
-struct dyString *netSlurpFile(int sd)
-/* Slurp file into dynamic string and return. */
+struct dyString *netSlurpFileMax(int sd, size_t maxSize)
+/* Slurp file into dynamic string and return.  If maxSize is nonzero and the
+ * data runs past it, stop reading, free what was read, and return NULL.  The
+ * caller still owns sd and should close it either way.  Freeing before the
+ * return matters to a caller running under pushCarefulMemHandler: the whole
+ * point of the cap is to stay under that ceiling, and a buffer abandoned here
+ * would count against it for the rest of the run. */
 {
 char buf[4*1024];
 int readSize;
 struct dyString *dy = dyStringNew(4*1024);
 
 /* Slurp file into dy and return. */
 while ((readSize = read(sd, buf, sizeof(buf))) > 0)
+    {
+    if (maxSize != 0 && (size_t)dy->stringSize + readSize > maxSize)
+        {
+        dyStringFree(&dy);
+        return NULL;
+        }
     dyStringAppendN(dy, buf, readSize);
+    }
 return dy;
 }
 
+struct dyString *netSlurpFile(int sd)
+/* Slurp file into dynamic string and return. */
+{
+return netSlurpFileMax(sd, 0);
+}
+
 struct dyString *netSlurpUrl(char *url)
 /* Go grab all of URL and return it as dynamic string. */
 {
 int sd = netUrlOpen(url);
 if (sd < 0)
     errAbort("netSlurpUrl: failed to open socket for [%s]", url);
 struct dyString *dy = netSlurpFile(sd);
 close(sd);
 return dy;
 }
 
 static void parseContentRange(char *x, ssize_t *rangeStart, ssize_t *rangeEnd)
 /* parse the content range information from response header value 
 	"bytes 763400000-763400112/763400113"
  */