ca1c644d399c9a9c007d58b1e26c1f94b8f18b22
lrnassar
  Fri Oct 24 16:11:30 2025 -0700
Removing some google user agents for (hopefully) better indexing, this was done by Max in the RR a few weeks ago. And disallowing json output from blat because google doesn't like to hit json, and also removing hgGeneGraph because it fails because of the cart requirement for google. No RM.

diff --git src/hg/htdocs/robots.rr.txt src/hg/htdocs/robots.rr.txt
index 8bf2d395853..c95bc3c0293 100644
--- src/hg/htdocs/robots.rr.txt
+++ src/hg/htdocs/robots.rr.txt
@@ -1,44 +1,43 @@
-User-agent: AdsBot-Google
 User-agent: AhrefsBot
 User-agent: Amazonbot
 User-agent: anthropic-ai
 User-agent: Applebot
 User-agent: AwarioRssBot
 User-agent: AwarioSmartBot
 User-agent: Bytedance
 User-agent: Bytespider
 User-agent: CCBot
 User-agent: ChatGPT-User
 User-agent: ClaudeBot
 User-agent: Claude-Web
 User-agent: cohere-ai
 User-agent: DataForSeoBot
 User-agent: Diffbot
 User-agent: FacebookBot
 User-agent: SemrushBot
 User-agent: FriendlyCrawler
-User-agent: Google-Extended
-User-agent: GoogleOther
 User-agent: GPTBot
 User-agent: img2dataset
 User-agent: ImagesiftBot
 User-agent: magpie-crawler
 User-agent: Meltwater
 User-agent: omgili
 User-agent: omgilibot
 User-agent: peer39_crawler
 User-agent: peer39_crawler/1.0
 User-agent: PerplexityBot
 User-agent: PiplBot
 User-agent: scoop.it
 User-agent: Seekr
 User-agent: YandexBot
 User-agent: YouBot
 Disallow: /
 
 User-agent: *
 Crawl-delay: 5
 Disallow: /admin/stats/
 Disallow: /goldenPath/certificate.html
 Disallow: /goldenPath/certificates/
 Disallow: /cgi-bin/hgTracks*.customText*.
+Disallow: /cgi-bin/hgBlat*output=json*
+Disallow: /cgi-bin/hgGeneGraph*