bbabbd5d2566d47d923d51dbe350634783455999 mspeir Sun Oct 26 12:14:52 2025 -0700 change soe to gi, refs #35031 diff --git src/hg/htdocs/goldenPath/help/mirror.html src/hg/htdocs/goldenPath/help/mirror.html index 835b440af95..fb46960b025 100755 --- src/hg/htdocs/goldenPath/help/mirror.html +++ src/hg/htdocs/goldenPath/help/mirror.html @@ -1,389 +1,389 @@ <!DOCTYPE HTML> <!-- DO NOT EDIT THE HTDOCS VERSION OF THIS FILE. THIS FILE IS AUTOMATICALLY GENERATED FROM A MARKDOWN FILE IN kent/src/product. MAKE ANY EDITS TO THIS PAGE THERE, THEN RUN MAKE in kent/src/product/mirrorDocs, AND FOLLOW THE INSTRUCTIONS TO COMMIT THIS PAGE INTO git. Please read the file kent/src/product/Note-To-QA.txt for details. --> <!--#set var="TITLE" value="Genome Browser Manual Installation" --> <!--#set var="ROOT" value="../.." --> <!-- Relative paths to support mirror sites with non-standard GB docs install --> <!--#include virtual="$ROOT/inc/gbPageStart.html" --> <h1>Installation of a UCSC Genome Browser on a local machine ("mirror")</h1> <h2>Contents</h2> <h6><a href='#considerations-before-installing-a-genome-browser'>Considerations before installing a Genome Browser</a></h6> <h6><a href='#installing-a-genome-browser-locally-with-the-gbic-installer'>Installing a Genome Browser locally with the GBiC installer</a></h6> <h6><a href='#docker-installation-instructions'>Docker installation instructions</a></h6> <h6><a href='#manual-installation-instructions'>Manual installation instructions</a></h6> <h6><a href='#using-udr-to-speed-up-downloads'>Using UDR to speed up downloads</a></h6> <h6><a href='#the-genome-mirror-mailing-list'>The genome-mirror mailing list</a></h6> <h6><a href='#what-happened-to-genome-browser-in-a-box-gbib'>What happened to Genome Browser in a Box (GBiB)?</a></h6> <a name='considerations-before-installing-a-genome-browser'></a> <h2>Considerations before installing a Genome Browser</h2> <p> Like most web servers, running a Genome Browser installation at your institution, even for your own department, requires a Unix machine, disk space (6TB for hg19), and the resources to update the site and underlying OS regularly. You may want to consider these alternatives before embarking on a full UCSC Genome Browser installation directly on your server. For information about operating in the cloud, visit the <a href='http://genome.ucsc.edu/goldenPath/help/cloud.html' title=''>Cloud Data and Software Resources</a> help page. </p> <ol> <li><p> <strong>Embed the Genome Browser graphic in your web page</strong> </p> <p> If you only want to include a genome browser view into your webpage for already existing genomes, you can use an <iframe> tag and point it to <a href='http://genome.ucsc.edu/cgi-bin/hgRenderTracks' title=''>http://genome.ucsc.edu/cgi-bin/hgRenderTracks</a>, which will show only the main browser graphic without any decorations. </p> <p> You can then use various parameters to adapt this graphic to your use case (e.g. set the displayed position, switch tracks on/off or highlight a range with a color), see our <a href='https://genome.ucsc.edu/goldenpath/help/customTrack.html#SHARE' title=''>manual pages</a> for a list of the parameters. </p></li> <li><p> <strong>Use Assembly Hubs</strong> </p> <p> <a href='https://genome.ucsc.edu/goldenPath/help/hgTrackHubHelp.html#Assembly' title=''>Assembly Hubs</a>: Assembly Hubs allow you to prepare any FASTA file, add annotations and use the Genome Browser to visualize it. All you need is a webserver where you save the indexed genome sequence and files to annotate it, e.g. in BAM, bigBed or bigWig format. </p> <ul> <li>Pros: <ul> <li>No installation, no updates, no long-term UNIX support necessary.</li> <li>Stable for many years, the link to the assembly hub can be put into a manuscript.</li> <li>For commercial users, no license is required.</li> </ul></li> <li>Cons: <ul> <li>You need access to a public webserver to store the files.</li> <li>Data is rendered at UCSC. Private data can be password-protected and loaded through https and restricted to UCSC's IP address, but has to be located on the web and accessible by the UCSC webserver.</li> <li>For BLAT searches in your genome, you have to start a BLAT server yourself on a Unix server.</li> <li>If your hub includes a high number of annotation files or HAL (multiple alignment) files and is located far from Santa Cruz, the display performance of assembly hubs may be slower than a local mirror. This issue can be resolved by using a UCSC mirror closer to the assembly hub (e.g. use genome-euro.ucsc.edu for assembly hubs located on servers in Europe, or genome-asia.ucsc.edu for those in Asia). Alternatively, you can improve performance by moving your hub data to a webspace provider closer to Santa Cruz or by using a content distribution network, where all content is mirrored and the closest location is chosen by your provider.</li> </ul></li> </ul></li> <li><p> <strong>Install locally with the Genome Browser installation script (GBIC)</strong> </p> <p> We recommend this only if none of the above options fulfill your needs. Our <a href='https://genome.ucsc.edu/goldenpath/help/gbic.html' title=''>GBIC installation</a> script will install a full local mirror of the UCSC website, for the assemblies you select. GBiC can also be used as a Docker container. See the <a href='#docker-installation-instructions' title=''>Docker installation instructions</a> below. We support mirror site installations as time allows, and have many functional mirrors of the Genome Browser worldwide. For details, see the <a href='#installing-a-genome-browser-locally-with-the-gbic-installer' title=''>section below</a>. </p> <ul> <li>Pros: <ul> <li>Relatively simple to install on a virtual machine or cloud instance: just run the script.</li> <li>Best performance when rendering local BAM/bigWig/bigBed files.</li> <li>For commercial users: easier click-through licensing compared to a full multi-seat, manual license.</li> </ul></li> <li>Cons: <ul> <li>To keep up with changes in the Genome Browser, you will have to install linux packages and update the linux distribution yourself in the future and run the installation script with the 'update' command if you want to take advantage of new features in the Genome Browser.</li> <li>Preferably run on a linux server that is not used otherwise.</li> <li>By default requires opening at least three outgoing ports to UCSC for MariaDB and BLAT in your firewall. Once all data is downloaded and BLAT setup locally, no open ports are needed anymore.</li> <li>For maximum browsing speed, can require up to 2-6TB to store all genome annotation tracks.</li> </ul></li> </ul></li> <li><p> <strong>Install manually yourself, by following installation instructions</strong> </p> <p> We do provide <a href='https://genome.ucsc.edu/goldenpath/help/mirrorManual.html' title=''>step-by-step instructions</a> for local installation, but do not recommend this, if any other system works for you. The internet also has various pages with instructions, but they are often out of date and may cause problems later. For details on manual instructions, see the <a href='#manual-installation-instructions' title=''>section below</a>. </p> <ul> <li>Pros: <ul> <li>You will understand the complete setup of the Genome Browser.</li> <li>For commercial users: license agreements can be customized to your needs.</li> </ul></li> <li>Cons: <ul> <li>Not easy to setup, even for experienced Unix administrators.</li> <li>Will probably require some support via the <a href='mailto::genome-mirror@soe.ucsc.edu' title=''>genome-mirror</a> mailing list.</li> <li>To keep up with changes in the Genome Browser, you will have to install linux packages and update the linux distribution yourself in the future and apply UCSC data updates yourself using rsync or MariaDB table loads</li> <li>Configuration changes on our side may break your setup.</li> <li>For maximum browsing speed, can require up to 2-6TB to store all genome annotation tracks.</li> <li>For commercial users: license agreements take longer to negotiate, no click-through license.</li> </ul></li> </ul></li> </ol> <p> A license is required for commercial download and/or installation of the Genome Browser binaries and source code. No license is needed for academic, nonprofit, and personal use. To purchase a license, see our <a href='https://genome.ucsc.edu/license/index.html' title=''>license Instructions</a> or visit the <a href='https://genome-store.ucsc.edu/' title=''>Genome Browser store</a>. </p> <a name='installing-a-genome-browser-locally-with-the-gbic-installer'></a> <h2>Installing a Genome Browser locally with the GBiC installer</h2> <p> If you do not want to use our prepared virtual machine Genome-Browser-in-a-Box, we provide a Genome Browser in the Cloud (GBiC) <a href='https://genome.ucsc.edu/goldenPath/help/gbic.html' title=''>installation program</a> that sets up a fully functional mirror on all major Linux distributions. </p> <p> The GBiC program has been tested and confirmed to work with Ubuntu 18/20/22/24 LTS, Rocky 9.5, and Fedora 30/35/41. </p> <p> Preferably, the installation should be performed on a fresh Linux installation, as it deactivates the default site config file in Apache and fills the MariaDB directory with numerous databases. The easiest way to accomplish this is to run the Genome Browser in the Cloud program in a new virtual machine. The program also works on Docker and cloud computing virtual machines, and has been tested on those sold by Amazon, Microsoft and Google. </p> <p> Like GBiB, the mirror installed by the GBiC can load the annotation data either from UCSC or a local database copy. If you load data from UCSC and use a cloud computing provider, it is highly advisable to run your instances in the US West Coast / San Francisco Bay Area or San Jose data centers; otherwise data-loading may be very slow. </p> <p> To run the installation program, please see the <a href='https://genome.ucsc.edu/goldenPath/help/gbic.html' title=''>GBiC user guide</a>. </p> <a name='docker-installation-instructions'></a> <h2>Docker installation instructions</h2> <p> Download the Dockerfile to a new directory and build the docker image. This works on Windows, OSX and Linux, as long as Docker is installed: </p> <pre><code>mkdir browserDocker && cd browserDocker wget https://raw.githubusercontent.com/ucscGenomeBrowser/kent/master/src/product/installer/docker/Dockerfile docker build . -t gbimage</code></pre> <p> You can then run the gbimage image that you just built as a new container in daemon mode (-d) and export its port to localhost: </p> <pre><code>docker run -d --name genomeBrowser -p 8080:80 gbimage</code></pre> <p> The Apache server is running on port 8080 then and you should be able to access it via https://localhost:8080 </p> <a name='manual-installation-instructions'></a> <h2>Manual installation instructions</h2> <p> See <a href='mirrorManual.html' title=''>mirrorManual.html</a>: If the installation program does not work on your linux distribution or you prefer to make adaptations to your mirror, we provide these <a href='mirrorManual.html' title=''>step-by-step installation instructions</a> that cover the configuration of Apache, MariaDB, the Genome Browser CGIs, temporary file removal and other topics, like data loading through proxies. </p> <p> The following external websites were not created by UCSC staff and are of varying quality, but may be helpful when installing on unusual platforms. </p> <ul> <li><a href='http://enotacoes.wordpress.com/2009/09/03/installing-a-minimal-ucsc-mirror-in-ubuntu-jaunty-64-bits/' title=''>Installation on Ubuntu</a></li> <li><a href='http://www.bioinformatics.uni-muenster.de/download/ucsc' title=''>Installation into a FreeBSD jail</a>, by Norbert Grundmann, Universitaet Muenster, Germany</li> <li><a href='http://bergmanlab.ls.manchester.ac.uk/?p=32' title=''>Compilation of the Kent source tree on OSX with fink</a> by Casey Bergman, Manchester, UK</li> <li><a href='http://www.oliverelliott.org/article/bioinformatics/tut_genomebrowser/' title=''>Browser installation and new genome setup</a>, by Oliver Elliott, Columbia University, USA</li> <li><a href='http://genomewiki.ucsc.edu/index.php/Browser_Installation' title=''>Installation notes in our wiki</a></li> </ul> <a name='using-udr-to-speed-up-downloads'></a> <h2>Using UDR to speed up downloads</h2> <p> <a href='https://github.com/LabAdvComp/UDR' title=''>UDR</a> (UDT Enabled Rsync) is a download protocol that is very efficent at sending large amounts of data over long distances. UDR utilizes rsync as the transport mechanism, but sends the data over the UDT protocol. UDR is not written or managed by UCSC. It is an open source tool created by the <a href='http://www.labcomputing.org/' title=''>Laboratory for Advanced Computing</a> at the University of Chicago. It has been tested under Linux, FreeBSD and Mac OSX, but may work under other UNIX variants. The source code can be obtained through <a href='https://github.com/LabAdvComp/UDR' title=''>GitHub</a>. When using the GBIC installation program, the <code>-u</code> option will use UDR for all downloads. </p> <p> If you manually download data only occasionally, there is no need to change your method; continue to visit our download server to download the files you need. This new protocol has been put in place primarily to facilitate quick downloads of huge amounts of data over long distances. </p> <p> With typical TCP-based protocols like http, ftp, and rsync, the transfer speed slows as the distance between the download source and destination increases. Protocols like UDT/UDR allow for many UDP packets to be sent in batch, thus allowing for much higher transmission speeds over long distances. UDR will be especially useful for users who are downloading from places distant to California. The US East Coast and the international community will likely see much higher download speeds when using UDR vs. rsync, http or ftp. </p> <p> If you need help building the UDR binaries or have questions about how UDR functions, please read the documentation on the GitHub page and if necessary, contact the UDR authors via the GitHub page. We recommend reading the documentation on the UDR GitHub page to better understand how UDR works. UDR is written in C++. It is Open Source and is released under the Apache 2.0 License. In order for it to work, you must have rsync installed on your system. </p> <p> For your convenience, we offer a binary distribution of UDR for Red Hat Enterprise Linux 6.x (or variants such as CentOS 6 or Scientific -Linux 6). You'll find both a 64-bit and 32-bit rpm <a href='http://hgdownload.soe.ucsc.edu/admin/udr' title=''>here</a>. +Linux 6). You'll find both a 64-bit and 32-bit rpm <a href='http://hgdownload.gi.ucsc.edu/admin/udr' title=''>here</a>. </p> <p> Once you have a working UDR binary, either by building from source or by installing the rpm, you can download files from either of our our download servers in a fashion similar to rsync. For example, using rsync, all of the MariaDB tables for the hg19 database can be downloaded using either one of the following two commands: </p> -<pre><code>rsync -avP rsync://hgdownload.soe.ucsc.edu/mysql/hg19/ /my/local/hg19/ -rsync -avP hgdownload.soe.ucsc.edu::mysql/hg19/ /my/local/hg19/</code></pre> +<pre><code>rsync -avP rsync://hgdownload.gi.ucsc.edu/mysql/hg19/ /my/local/hg19/ +rsync -avP hgdownload.gi.ucsc.edu::mysql/hg19/ /my/local/hg19/</code></pre> <p> Using UDR is very similar. The UDR syntax for downloading the same data would be: </p> -<pre><code>udr rsync -avP hgdownload.soe.ucsc.edu::mysql/hg19/ /my/local/hg19/</code></pre> +<pre><code>udr rsync -avP hgdownload.gi.ucsc.edu::mysql/hg19/ /my/local/hg19/</code></pre> <a name='the-genome-mirror-mailing-list'></a> <h2>The genome-mirror mailing list</h2> <p> For questions about installing and mirroring the UCSC Genome Browser, contact the UCSC mailing list <a href='mailto::genome-mirror@soe.ucsc.edu' title=''>genome-mirror@soe.ucsc.edu</a>. <strong><span class="gbsWarnText"> Messages sent to this address will be posted the moderated genome-mirror mailing list, which is archived on a SEARCHABLE PUBLIC </span><a href='http://groups.google.com/a/soe.ucsc.edu/group/genome-mirror' title=''>Google Groups forum</a></strong>. </p> <a name='what-happened-to-genome-browser-in-a-box-gbib'></a> <h2>What happened to Genome Browser in a Box (GBiB)?</h2> <p> Support for the Genome Browser in a Box (GBiB) product <strong>product ended in 2025</strong>. We recommend use of the GBiC <a href='/goldenPath/help/mirror.html#docker-installation-instructions' title=''>dockerfile</a> instead. The information below is kept for historical purposes. </p> <ol> <li><p> <strong>Use Genome Browser in a Box</strong> </p> <p> <a href='https://genome.ucsc.edu/goldenPath/help/gbib.html' title=''>Genome Browser in a Box</a> (GBiB): is a fully configured virtual machine that includes Apache and MariaDB, and has behavior identical to the UCSC website. GBiB loads genome data from the UCSC download servers on the fly. Website and data updates are applied automatically every two weeks. By default, GBiB uses the VirtualBox virtualization software, so it can be run on any operating system, Windows, OSX and Linux. It does not require VirtualBox, the virtual machine image can easily be converted to e.g., VMWare or HyperV. For increased privacy, you can download the genomes and annotation tracks you need and use your GBiB off-line even on a laptop. </p> <ul> <li>Pros: <ul> <li>Relatively simple to install: download and double-click.</li> <li>By default, software and data updates are managed remotely by UCSC, via an update script run every week.</li> <li>Best performance when rendering local BAM/bigWig/bigBed files.</li> <li>No Unix webserver needed, runs on any OS.</li> <li>For commercial users: easier click-through licensing compared to a full multi-seat, manual license.</li> </ul></li> <li>Cons: <ul> <li>Requires the free VirtualBox software or a commercial Virtualization system.</li> <li>By default requires opening at least three outgoing ports to UCSC for MariaDB, Rsync downloads and BLAT in your firewall. Once all data is downloaded, no open ports are needed.</li> <li>For maximum browsing speed, can require up to 2-6TB to store all genome annotation tracks, like a manual local installation.</li> </ul></li> </ul></li> </ol> <!--#include virtual="$ROOT/inc/gbPageEnd.html" -->