e68feeac1d9b49e6792edb0d6665f961c14572d6 max Mon Feb 8 05:02:32 2021 -0800 fixing the mirrorManual moving the git clone step before the installation step, #26934 diff --git src/product/mirrorManual.txt src/product/mirrorManual.txt index 440905f..844f600 100644 --- src/product/mirrorManual.txt +++ src/product/mirrorManual.txt @@ -108,41 +108,89 @@ For example, to get the size of all of the files for hg19, you would use the following command: rsync -hna --stats rsync://hgdownload.soe.ucsc.edu/gbdb/hg19/ | egrep "Number of files:|total size is" After running that command, you should see output like this: Number of files: 54886 total size is 6515.70G speedup is 5181080.38 (DRY RUN) The next command will give you the size of the entire mySQL database, but can be changed to get the size for a particular assembly: rsync -hna --stats rsync://hgdownload.soe.ucsc.edu/mysql/ | egrep "Number of files:|total size is" +# Local Git repository +Use the following procedures to create your own personal copy of the kent source +tree where you can have your own edits that are not part of the development at +UCSC. This is useful for mirror sites that have their own customizations in +the source tree for local circumstances. + +Install Git software version 1.6.2.2 or later. See the Git Community Handbook + installation (<https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>) and setup + (<href='https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>) instructions, as well + as our Installing Git (<http://genomewiki.ucsc.edu/index.php/Installing_git>) + Genomewiki page. + +Start an initial Git local repository: + + git clone git://genome-source.soe.ucsc.edu/kent.git + +or, if a firewall prevents git daemon port 9418, use: + + git clone http://genome-source.soe.ucsc.edu/kent.git + +The kent source tree will be imported to the current working directory in a +directory named ./kent/. + +Track the beta branch at UCSC repository: Most users want to use the beta branch, which has tested, released versions of + the browser. To create a beta tracking branch: + + cd kent + git checkout -t -b beta origin/beta + +The -b creates a local branch with name "beta", and checks it out. +The -t makes it a tracking branch, so that 'git pull' brings in updates from +origin/beta, the "real" beta branch in our public central read-only repository. + +To get the latest UCSC release, from anywhere within the kent source tree: + + git pull + +Updates: UCSC generally updates the origin/beta branch every three weeks. If you are + updating database tables for a mirror site, we recommend that you update the + source at the same time, as source code is sometimes modified to include + operations on new columns that have been added to database tables. + +For instructions on keeping local tracks separate from UCSC Genome Browser +tracks created at UCSC and mirrored from there, see the section "Adding tracks +to the browser" below. + # Installing the UCSC Genome browser Note: we offer Genome-Browser-in-a-Box (GBIB), a fully configured virtual machine image that can be converted for VirtualBox, VMWare, Hyper-V and other popular environments. We also offer Genome-Browser-in-the-Cloud (GBIC) an shell script that installs a genome browser in most main Linux distributions (Most Debian and Redhat-based ones, like Ubuntu and CentOS). -See https://hgwdev-max.gi.ucsc.edu/goldenPath/help/mirror.html +See https://genome.ucsc.edu/goldenPath/help/mirror.html Scripts to perform all of the functions below can be found in -the directory here: src/products/scripts/ +the directory https://github.com/ucscGenomeBrowser/kent/tree/master/src/product/scripts. +In a git clone of the kent repository, the scripts +are located in src/product/scripts. Confirm the following: 1. Apache web server is installed and working, http://localhost/ provides the Apache default home page from your machine NOTE: The browser static html web pages require the Apache XBitHack option to be enabled to allow SSI <!--#include ... --> statements to function. Add 'Options +Includes' for your html directory, your httpd.conf file entry looks like: XBitHack on <Directory /usr/local/apache/htdocs> Options +Includes </Directory> @@ -664,30 +712,76 @@ GRANT ALL PRIVILEGES ON *.* TO 'someuser'@'%' REQUIRE SUBJECT '/C=US/ST=CA/L=Santa Cruz/O=YourCompany/OU=YourDivision/CN=someuser/emailAddress=someuser@YourCompany.com' AND ISSUER '/C=US/ST=CA/L=Santa Cruz/O=YourCompany/OU=YourDivision/CN=YourCompanyCA/emailAddress=admin@YourCompany.com' AND CIPHER 'DHE-RSA-AES256-SHA'; You can see the cert details like this: openssl x509 -in /somepath/someuser-cert.pem -text In later versions of MySQL, it is a requirement that the CN of the CA cert must DIFFER from the CN of the user and server certs. Further MySQL SSL documentation is available from <https://dev.mysql.com/doc/refman/5.6/en/creating-ssl-files-using-openssl.html> +# Local Git repository +Use the following procedures to create your own personal copy of the kent source +tree where you can have your own edits that are not part of the development at +UCSC. This is useful for mirror sites that have their own customizations in +the source tree for local circumstances. + +Install Git software version 1.6.2.2 or later. See the Git Community Handbook + installation (<https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>) and setup + (<href='https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>) instructions, as well + as our Installing Git (<http://genomewiki.ucsc.edu/index.php/Installing_git>) + Genomewiki page. + +Start an initial Git local repository: + + git clone git://genome-source.soe.ucsc.edu/kent.git + +or, if a firewall prevents git daemon port 9418, use: + + git clone http://genome-source.soe.ucsc.edu/kent.git + +The kent source tree will be imported to the current working directory in a +directory named ./kent/. + +Track the beta branch at UCSC repository: Most users want to use the beta branch, which has tested, released versions of + the browser. To create a beta tracking branch: + + cd kent + git checkout -t -b beta origin/beta + +The -b creates a local branch with name "beta", and checks it out. +The -t makes it a tracking branch, so that 'git pull' brings in updates from +origin/beta, the "real" beta branch in our public central read-only repository. + +To get the latest UCSC release, from anywhere within the kent source tree: + + git pull + +Updates: UCSC generally updates the origin/beta branch every three weeks. If you are + updating database tables for a mirror site, we recommend that you update the + source at the same time, as source code is sometimes modified to include + operations on new columns that have been added to database tables. + +See also: the README files in the source tree directory src/product/README.*. +For instructions on keeping local tracks separate from UCSC Genome Browser +tracks, see src/product/README.trackDb. + # Adding a custom genome to the browser Please note that setting up an [assembly hub](http://genomewiki.ucsc.edu/index.php/Assembly_Hubs) is a lot easier than adding a genome to a local mirror. The browser can be made to operate with a bare minimum of tables for the purpose of demonstrating the CGI binaries are functioning. The only tables you need to load for this are: 1. all tables in the hgcentral database 2. six tables in the human genome Create an empty hgcentral database: @@ -1086,76 +1180,30 @@ generally looks like this: hPrintf("<FORM ACTION=\"%s\" NAME=\"mainForm\" METHOD=%s>\n", getScriptName(), cartUsualString(cart, "formMethod", "POST")); If you add &formMethod=GET and a subsequently fetched form is still posting, you might need to alter the "<FORM..." statement to use the cartUsualString. The hg18 hgTracks config page generates a GET URL that is too long for FireFox, so after debugging hgTables, you will probably want to add &formMethod=POST to an URL (or clear cart, load session etc). One thing that does not work with GET is "upload file" inputs. -# Local Git repository -Use the following procedures to create your own personal copy of the kent source -tree where you can have your own edits that are not part of the development at -UCSC. This is useful for mirror sites that have their own customizations in -the source tree for local circumstances. - -Install Git software version 1.6.2.2 or later. See the Git Community Handbook - installation (<https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>) and setup - (<href='https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup>) instructions, as well - as our Installing Git (<http://genomewiki.ucsc.edu/index.php/Installing_git>) - Genomewiki page. - -Start an initial Git local repository: - - git clone git://genome-source.soe.ucsc.edu/kent.git - -or, if a firewall prevents git daemon port 9418, use: - - git clone http://genome-source.soe.ucsc.edu/kent.git - -The kent source tree will be imported to the current working directory in a -directory named ./kent/. - -Track the beta branch at UCSC repository: Most users want to use the beta branch, which has tested, released versions of - the browser. To create a beta tracking branch: - - cd kent - git checkout -t -b beta origin/beta - -The -b creates a local branch with name "beta", and checks it out. -The -t makes it a tracking branch, so that 'git pull' brings in updates from -origin/beta, the "real" beta branch in our public central read-only repository. - -To get the latest UCSC release, from anywhere within the kent source tree: - - git pull - -Updates: UCSC generally updates the origin/beta branch every three weeks. If you are - updating database tables for a mirror site, we recommend that you update the - source at the same time, as source code is sometimes modified to include - operations on new columns that have been added to database tables. - -See also: the README files in the source tree directory src/product/README.*. -For instructions on keeping local tracks separate from UCSC Genome Browser -tracks, see src/product/README.trackDb. - # Proxy support net.c now has support for http(s) proxy servers which may be required by some installations to get through the firewall to external resources such as (but not limited to) for example bigWig or bigBed data via custom track bigDataUrl. One must add the setting "httpProxy", "httpsProxy", "ftpProxy" to hg.conf httpProxy=http://someProxyServer:3128 httpsProxy=http://someProxyServer:3128 ftpProxy=ftp://127.0.0.1:2121 If the proxy server requires BASIC authentication