bbabbd5d2566d47d923d51dbe350634783455999 mspeir Sun Oct 26 12:14:52 2025 -0700 change soe to gi, refs #35031 diff --git src/hg/htdocs/goldenPath/help/cloud.html src/hg/htdocs/goldenPath/help/cloud.html index a73904f2b0b..33b081f0f80 100755 --- src/hg/htdocs/goldenPath/help/cloud.html +++ src/hg/htdocs/goldenPath/help/cloud.html @@ -87,58 +87,58 @@ <ul class=gbsNoBullet> <li><a href="#tag_name">Where can I learn more about Amazon Tools?</a></li> </ul> </div> </div> </div> <!-- ====Cloud Data Section============================== --> <a id="S3"></a> <h2>What is the Amazon s3://genome-browser bucket?</h2> <p> S3 stands for Simple Storage Service, and it is the name for cloud storage in Amazon Web Services (AWS). The data available through S3 is essentially stored in a folder called a bucket, and files are called objects. The s3://genome-browser bucket is a copy of the main data available on our -UCSC Genome Browser Download website: <a href="https://hgdownload.soe.ucsc.edu/downloads.html" -target="_blank">https://hgdownload.soe.ucsc.edu/downloads.html</a></p> +UCSC Genome Browser Download website: <a href="https://hgdownload.gi.ucsc.edu/downloads.html" +target="_blank">https://hgdownload.gi.ucsc.edu/downloads.html</a></p> <p> By placing our Download server files in an S3 bucket, developers working in the cloud can more easily integrate with UCSC data. You can learn more about how S3-object-based storage works, and its advantages of being accessible anywhere across the world with low latency and high durability by reviewing <a href="https://docs.aws.amazon.com/s3/" target="_blank">Amazon's S3 documentation</a>.</p> <a id="S3_contents"></a> <h2>What specific files are in the s3://genome-browser bucket?</h2> <p> -The data mirrors our <a href="https://hgdownload.soe.ucsc.edu/downloads.html" +The data mirrors our <a href="https://hgdownload.gi.ucsc.edu/downloads.html" target="_blank">UCSC Genome Browser Download</a> website's main rsync directories: <pre> UCSC Human Golden Path Downloads s3://genome-browser/goldenPath UCSC Human Genome Browser Gbdb Data Files s3://genome-browser/gbdb UCSC Human Genome Raw Mysql Tables s3://genome-browser/mysql UCSC Human Genome Web Site CGI Binaries s3://genome-browser/cgi-bin UCSC Human Genome Web Site Htdocs s3://genome-browser/htdocs </pre></p> <p> <ul> <li>The <b>goldenPath</b> directory is organized by assembly name, and represents the file structure on our Download server, which includes README.txt files. For instance, the sequence data for the human hg38 assembly would be found in this location with an instructive README.txt: <code>goldenPath/hg38/bigZips/README.txt</code>. The README.txt, also -available on the <a href="https://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/README.txt" +available on the <a href="https://hgdownload.gi.ucsc.edu/goldenPath/hg38/bigZips/README.txt" target="_blank">Download website</a>, informs that the most recent patch-inclusive sequence is found in <code>goldenPath/hg38/bigZips/latest/</code>.</li> <li>The <b>gbdb</b> directory, also organized by assembly name, provides access to genome browser database files in binary format used by the browser software. For instance, the underlying binary indexed sequence data for the hg38 databases used in the display in the UCSC Genome Browser would be located in the following location, <code>gbdb/hg38/hg38.2bit</code>, matching the file in the <code>goldenPath/hg38/bigZips/latest/</code> directory, reflecting how these files are operated on by the UCSC Genome Browser software in order to display assembly sequence when browsing.</li> <li>The <b>mysql</b> directory, also organized by assembly name, provides access to MySQL database tableName.MYD files, and their related tableName.MYI index and tableName.frm format files, providing a copy of the tables used by the main Browser site.</li> @@ -217,101 +217,101 @@ end</td></tr> <tr><th>/getData/track</th><td>(genome or (hubUrl and genome)) and track</td><td>chrom, (start and end), maxItemsOutput, jsonOutputArrays</td></tr> </table></p> <p> By reviewing <a href="http://genome.ucsc.edu/goldenPath/help/api.html#list_examples" target="_blank">example data access URLs</a> demonstrating of list and getData functions and further <a href="http://genome.ucsc.edu/goldenPath/help/api.html#Practical_examples" target="_blank">practical examples URLs</a> of extracting specific track data items you can learn more about the ways of using the API to extract data.</p> <!-- ========== What is the Download Server and does one use it? ============================== --> <a id="download"></a> <h2>What is the Download server and how does one use it?</h2> <p> -The UCSC Genome Browser Download website, <a href="https://hgdownload.soe.ucsc.edu/downloads.html" -target="_blank">hgdownload.soe.ucsc.edu</a>, is the source of the data +The UCSC Genome Browser Download website, <a href="https://hgdownload.gi.ucsc.edu/downloads.html" +target="_blank">hgdownload.gi.ucsc.edu</a>, is the source of the data hosted in the Amazon s3://genome-browser bucket. It can be viewed in a web browser to access specific download files, or the data can be copied with rysnc commands.</p> <h3>Examples</h3> <p> For instance, the following <a href="https://en.wikipedia.org/wiki/Rsync" target="_blank">rsync</a> command will show you the various rysnc directories available on our Download server: <pre> -$ rsync -a -P rsync://hgdownload.soe.ucsc.edu/ +$ rsync -a -P rsync://hgdownload.gi.ucsc.edu/ genome UCSC Human Genome Downloads sars UCSC Human Genome SARS Downloads htdocs UCSC Human Genome Web Site Htdocs goldenPath UCSC Human Golden Path Downloads cgi-bin UCSC Human Genome Web Site CGI Binaries x86_64 cgi-bin-i386 UCSC Human Genome Web Site CGI Binaries i386 gbdb UCSC Human Genome Browser Gbdb Config Files archives UCSC Human Genome Browser Archived Config Files mysql UCSC Human Genome Raw Mysql Tables gbib UCSC Genome Browser in a Box hubs UCSC Genome Browser Public Hubs </pre> <ul> <li>For instance, here is an example of accessing a README file in the <code>goldenPath/</code>Downloads directory:<br> -<code>rsync -a -P rsync://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/README.txt ./</code></li> +<code>rsync -a -P rsync://hgdownload.gi.ucsc.edu/goldenPath/hg38/bigZips/README.txt ./</code></li> <li>And here is an example link that would access the <code>gbdb/</code> binary data directory for the human hg38 assembly 2bit file:<br> -<code>rsync -a -P rsync://hgdownload.soe.ucsc.edu/gbdb/hg38/hg38.2bit ./</code></li> +<code>rsync -a -P rsync://hgdownload.gi.ucsc.edu/gbdb/hg38/hg38.2bit ./</code></li> <li>And here is an example link that would access our publications html page from the bucket's <code>htdocs/</code> hypertext document directory:</br> -<code>rsync -a -P rsync://hgdownload.soe.ucsc.edu/htdocs/goldenPath/pubs.html ./</code></li> +<code>rsync -a -P rsync://hgdownload.gi.ucsc.edu/htdocs/goldenPath/pubs.html ./</code></li> </ul></p> <p> Many of these rsync directories exist to support the Genome Browser in a Cloud (<a target="_blank" href="gbic.html">GBiC</a>) and the Genome Browser in a Box (<a target="_blank" href="gbib.html">GBiB</a>) software products discussed below. Also note that there is a mirror of the download server available in Europe so the above rysnc commands can also be pointed to the <code>hgdownload-euro</code> locations. <ul><li>For instance here is a command to access data from the Europe location:<br> <code>rsync -a -P rsync://hgdownload-euro.soe.ucsc.edu/gbdb/hg38/hg38.2bit ./</code></li> </ul> <!-- ========== What is the MySQL server and how does one use it? ============================== --> <a id="MySQL"></a> <h2>What is the MySQL server and how does one use it?</h2> <p> The UCSC Genome Browser uses <a href="https://en.wikipedia.org/wiki/MariaDB" target="_blank">MariaDB</a> (fork of MySQL) as the backend database server and maintains -a public server at <b>genome-mysql.soe.ucsc.edu</b> to allow direct queries.</p> +a public server at <b>genome-mysql.gi.ucsc.edu</b> to allow direct queries.</p> <h3>Examples</h3> <p> <ul> <li>For instance, here is an example of accessing the hg38 human assembly database and selecting from the table <code>trackDb</code> all the entries in the group (grp) "genes" and ordering those entries by tableName:<br> <code> -mysql -h genome-mysql.soe.ucsc.edu -u genome -NBe 'select tableName from trackDb where grp = "genes" order by tableName' hg38 +mysql -h genome-mysql.gi.ucsc.edu -u genome -NBe 'select tableName from trackDb where grp = "genes" order by tableName' hg38 </code></li> <li>And here is an example of accessing a specific Transcription Factor Binding Site (TFBS) table <code>wgEncodeRegTfbsClusteredV3</code> on the human hg19 assembly and selecting entries from a 500 base pair region on chr1:<br> <code> -mysql --user=genome --host=genome-mysql.soe.ucsc.edu -A -Ne 'select chrom,chromStart,chromEnd,name,score +mysql --user=genome --host=genome-mysql.gi.ucsc.edu -A -Ne 'select chrom,chromStart,chromEnd,name,score from wgEncodeRegTfbsClusteredV3 where chrom = "chr1" and chromStart > 10000 and chromEnd < 10500;' hg19 </code></li> <li>And here is an example query that will pull all the long non-coding entries (lncRNA) from the <code>wgEncodeGencodeBasicV39</code> table on the hg38 genome:<br> <code> -mysql -u genome -h genome-mysql.soe.ucsc.edu hg38 -e 'select g.name,a.transcriptType from wgEncodeGencodeBasicV39 g, +mysql -u genome -h genome-mysql.gi.ucsc.edu hg38 -e 'select g.name,a.transcriptType from wgEncodeGencodeBasicV39 g, wgEncodeGencodeAttrsV39 a where (g.name = a.transcriptId) and (a.transcriptType = "lncRNA");' </code></li> </ul></p> <p> See the <a target="_blank" href="mysql.html">Downloading Data using MariaDB (MySQL)</a> for more information. Also, there is a mirror of the MariaDb server available in Europe so commands can also be pointed to the <code>genome-euro-mysql</code> location. <ul><li>For instance here is a command to access hg38 data from the Europe location:<br> <code>mysql -h genome-mysql-euro.soe.ucsc.edu -u genome -NBe 'show tables' hg38</code></li> </ul> </p> <!-- ====Cloud Software Section============================== -->