14ecff0730596f8be147b255afc8acb99e9095e2
max
  Fri Sep 18 09:06:06 2026 -0700
hgHubConnect: an api key made on one geo mirror now works on all of them, behind syncHubApiKeys

#Preview2 week - bugs introduced now will need a build patch to fix
#Preview2 week - bugs introduced now will need a build patch to fix
Api keys live in the hgcentral of whichever mirror issued them, so a key made on
genome.ucsc.edu was rejected on genome-euro and genome-asia, and the botDelay
error told the user to go and make a second one. hgHubConnect now tells the other
nodes about a key as soon as it generates or revokes one: cjGenerateApiKey and
cjRevokeApiKey call syncApiKeyToOtherNodes(), which posts a hgHubSyncApiKey
cartJson request to each peer, and cjSyncApiKey() on the far side writes it into
that mirror's own table.

The peers come from hgcentral.gbNode via the new geoMirrorNotifyOtherNodes(), and
the request is signed with login.cookieSalt, which all of a site's mirrors already
share - it is what makes the login cookie verifiable on each of them. So neither a
list of mirror addresses nor a new shared secret needs provisioning. The notify is
best effort: a peer that is down is warn()ed about and skipped, never failing the
local generate or revoke, which has already committed by then.

All of it is off unless hg.conf says syncHubApiKeys=on. With the gate off the
sender returns at once, the receiver refuses the request outright rather than
merely being unreachable, and botDelay keeps printing the old server-specific
wording, which off is still the truth. Registered in hgConfCatalog.py as a gate,
to be flipped once this is released.

refs #38323

diff --git src/hg/lib/botDelay.c src/hg/lib/botDelay.c
index 9ea795e8523..93f62c54339 100644
--- src/hg/lib/botDelay.c
+++ src/hg/lib/botDelay.c
@@ -219,30 +219,39 @@
         // hgsid, and only if none of these is available, on IP address. Also, check
         // apiKey and cookieId if they are valid, check hgsid if the string looks OK.
         char *apiKey = cgiOptionalString("apiKey");
         if (apiKey)
             {
             // Here we do a mysql query before the bottleneck is complete. 
             // And this is better than handling the request without bottleneck
             // The connection is closed right away, so if the bottleneck leads to a long sleep, it won't tie up
             // the MariaDB server. The cost of opening a connection is less than 1msec.
             struct sqlConnection *conn = hConnectCentralNoCache();
             char *userName = hubSpaceUserNameForApiKey(conn, apiKey);
             sqlDisconnect(&conn);
 
             if (userName)
                 safef(botCheckString, 256, "apiKey%s %f", apiKey, fraction);
+            else if (cfgOptionBooleanDefault("syncHubApiKeys", FALSE))
+                // hgHubConnect copies a new or revoked key to the other geo mirror nodes,
+                // so a key made on any of them is valid here
+                hUserAbort("Invalid apiKey provided on URL. "
+                        "Make sure that the apiKey is valid, "
+                        "check https://genome.ucsc.edu/cgi-bin/hgHubConnect#hubDeveloper to create one "
+                        "or check this key. An apiKey created on any UCSC genome browser mirror "
+                        "(genome.ucsc.edu, genome-euro.ucsc.edu, genome-asia.ucsc.edu) works on all of "
+                        "them. If you have problems with the apiKey, contact us.");
             else
                 hUserAbort("Invalid apiKey provided on URL. "
                         "Make sure that the apiKey is valid, "
                         "check https://genome-euro.ucsc.edu/cgi-bin/hgHubConnect#hubDeveloper to create one "
                         "or check this key. Note that an apiKey for genome-euro must be created on "
                         "https://genome-euro.ucsc.edu/cgi-bin/hgHubConnect and the same for genome-asia or "
                         "other mirrors, apiKeys are server-specific. If you have problems with the apiKey, "
                         "contact us.");
             }
         else
             {
             if (isValidHguid(cookieUserId))
                 safef(botCheckString, 256, "uid%s %f", cookieUserId, fraction);
             else
                 {