Please note, if you are working with a genome that has already been submitted to the NCBI Assembly system, it may already be available in the UCSC Genome Browser.
Please examine the GenArk Assembly Hub collection to see if your genome of interest is already available. In the case it cannot be found there, you can use the UCSC Assembly Request page to request a genome assembly be added to the UCSC Genome Browser.
The Assembly Hub function allows you to display your novel genome sequence using the UCSC Genome Browser.
To display your novel genome sequence, use a web server at your institution (or free services like Cyverse), for usage behind a firewall you can also load them locally through GBiB to supply your files to the UCSC Genome Browser. Note that hosting hub files on HTTP is highly recommended and much more efficient than FTP. You then establish a hierarchy of directories and files to host your novel genome sequence. For example:
+To display your novel genome sequence, use a web server at your institution (or free services +like Cyverse), +for usage behind a firewall you can also load them locally through docker +to supply your files to the UCSC Genome Browser. Note that hosting hub files on HTTP is highly +recommended and much more efficient than FTP. You then establish a hierarchy of directories and +files to host your novel genome sequence. For example:
myHub/ - directory to organize your files on this hub
hub.txt - primary reference text file to define the hub, refers to:
genomes.txt - definitions for each genome assembly on this hub
newOrg1/ - directory of files for this specific genome assembly
newOrg1.2bit - ‘2bit’ file constructed from your fasta sequence
description.html - information about this assembly for users
trackDb.txt - definitions for tracks on this genome assembly
groups.txt - definitions for track groups on this assembly
bigWig and bigBed files - data for tracks on this assembly
external track hub data tracks
The URL to reference this hub would be: http://yourLab.yourInstitution.edu/myHub/hub.txt
Note: there is now a useOneFile on hub setting that allows the hub properties to be specified in a single file. More information about this setting can be found on the Genome Browser User Guide.
You can view a working example hierarchy of files at: Plants
A smaller slice of this hub is represented in a Quick Start Guide to Assembly Hubs.
You can build direct links to the genome(s) in your assembly hub:
The initial file hub.txt is the primary URL reference for your assembly hub. The format of the file:
hub hubName shortLabel genome longLabel Comment describing this hub contents genomesFile genomes.txt email contactEmail@institution.edu descriptionUrl aboutHub.html
shortLabel is the name that will appear in the genome pull-down menu at the UCSC gateway page. Example: Plants.
genomesFile is a reference to the next definition file in this chain that will describe the assemblies and tracks available at this hub. Typically genomes.txt is at the same directory level as this hub.txt, however it can also be a relative path reference to a different directory level.
The email address provides users a contact point for queries related to this assembly hub.
The descriptionUrl provides a relative path or URL link to a webpage describing the overall hub.
The genomes.txt file provides the references to the genome assemblies and tracks available at this assembly hub. The example file indicates the typical contents:
genome ricCom1 trackDb ricCom1/trackDb.txt groups ricCom1/groups.txt description July 2011 Castor bean twoBitPath ricCom1/ricCom1.2bit organism Ricinus communis defaultPos E09R7372:1000000-2000000 orderKey 4800 scientificName Ricinus communis htmlPath ricCom1/description.html transBlat yourLab.yourInstitution.edu 17777 blat yourLab.yourInstitution.edu 17777 isPcr yourLab.yourInstitution.edu 17779
There can be multiple assembly definitions in this single file. Separate these stanzas with blank lines. The references to other files are relative path references. In this example there is a sub-directory here called ricCom1 which contains the files for this specific assembly.
Note that it is strongly encouraged to give each of your genomes stanza's a line for defaultPos, scientificName, organism, description (along with other above settings) so that when your hub is attached it will load a specified default location and have text to be more easily searched from the Gateway page.
The .2bit file is constructed from the fasta sequence for the assembly. The kent source program faToTwoBit is used to construct this file. Download the program from the downloads section of the Browser. For example:
faToTwoBit ricCom1.fa ricCom1.2bit
Use the twoBitInfo to verify the sequences in this assembly and create a chrom.sizes file which is not used in the hub, but is useful in later processing to construct the big* files:
twoBitInfo ricCom1.2bit stdout | sort -k2rn > ricCom1.chrom.sizes
The .2bit commands can function with the .2bit file at a URL:
twoBitInfo -udcDir=http://genome-test.gi.ucsc.edu/~hiram/hubs/Plants/ricCom1/ricCom1.2bit stdout | sort -k2nr > ricCom1.chrom.sizes
Sequence can be extracted from the .2bit file with the twoBitToFa command, for example:
twoBitToFa -seq=chrCp -udcDir=http://genome-test.gi.ucsc.edu/~hiram/hubs/Plants/ricCom1/ricCom1.2bit stdout > ricCom1.chrCp.fa
The groups.txt file defines the grouping of track controls under the primary genome browser image display. The example referenced here has the usual definitions as found in the UCSC Genome Browser.
Each group is defined, for example the Mapping group:
name map label Mapping priority 2 defaultIsClosed 0
Tracks are defined in the trackDb.txt where each stanza describes how tracks are displayed (shortLabel/longLabel/color/visibility) and other information such as what group the track should belong to (referencing the groups.txt) and if any additional html should display when one clicks into the track or a track item:
track gap_ longLabel Gap shortLabel Gap priority 11 visibility dense color 0,0,0 bigDataUrl bbi/ricCom1.gap.bb type bigBed 4 group map html ../trackDescriptions/gap
For more informations about the syntax of the trackDb.txt file, use UCSC's Hub Track Database Definition page. It helps to have a cluster super computer to process the genomes to construct tracks. It can be done for small genomes on single computers that have multiple cores. The process for each track is unique. Please note the continuing document: Browser Track Construction for a discussion of constructing tracks for your assembly hub.
Assembly hubs can have a Cytoband track that can allow for quicker navigation of individual chromosomes and display banding pattern information if known.
A quick version of the track can be built using the existing chrom.sizes files for your assembly (the banding options include gneg, gpos25, gpos50, gpos75, gpos100, acen, gvar, or stalk):
cat araTha1.chrom.sizes | sort -k1,1 -k2,2n | awk '{print $1,0,$2,$1,"gneg"}' > cytoBandIdeo.bed
The resulting bed file can be turned into a big bed and given a .as file (example here) to inform the browser it is not a normal bed.
bedToBigBed -type=bed4 cytoBandIdeo.bed -as=cytoBand.as araTha1.chrom.sizes cytoBandIdeo.bigBed
In the trackDb, as long as the track is named cytoBandIdeo (track cytoBandIdeo example) it will load in the assembly hub.
There are resources for automatically building assembly hubs available from G-OnRamp and MakeHub.
There is also a collection of Example NCBI assembly hubs that are already working and can either be used or copied as a template to build further hubs.
G-OnRamp is a Galaxy workflow that turns a genome assembly and RNA-Seq data into a Genome Browser with multiple evidence tracks. Because G-OnRamp is based on the Galaxy platform, developing some familiarity with the key concepts and functionalities of Galaxy would be beneficial prior to using G-OnRamp. Here is a link to their instruction page that gives an overview of their process.
MakeHub is a command line tool for the fully automatic generation of track data hubs for visualizing genomes with the UCSC genome browser. More information can be found on their GitHub page.
Here are some quick steps to load an example hub from this collection, and an attempt to explain how to look at the files behind the hub.
Now to investigate the files behind the hub to understand the process involved:
BLAT servers (gfServer) are configured as either dedicated or dynamic servers. Dedicated BLAT serves index a genome when started and remain running in memory to quickly respond to request. Dynamic BLAT servers pre-index genomes to files and are run on demand to handle a BLAT request and then exit.
Dedicated gfServer are easier to configure and faster to respond. However, the server continually uses memory. A dynamic gfServer is more appropriate with multiple assemblies and infrequent use. Their response time is usually acceptable; however, it varies with the speed of the disk containing the index. With repeated access, the operating system will cache the indexes in memory, improving response time.
By running your own BLAT server, you can add lines to the genomes.txt file of your assembly hub to enable the browser to access the server and activate blat searches.
Please see Running your own gfServer for details on installing and configuring both dedicated and dynamic gfServers.
transBlat yourServer.yourInstitution.edu 17777 blat yourServer.yourInstitution.edu 17779 isPcr yourServer.yourInstitution.edu 17779
Please see more about configuring your blat gfServer to replicate the UCSC Browser's settings, which will also have information about optimizing PCR results. The Source Downloads page offers access to utilities with pre-compiled binaries such as gfserver found in a blat/ directory for your machine type here and further blat documentation here, and the gfServer usage statement for further options.
-Please also know you can set up gfservers on a GBiB and run it locally. Please see this GBiB assembly blat step-by-step set up page for details.
+Please also know you can set up gfservers on docker and run it locally. +
Note: You can stop your instance of gfServer with a command. For example:
gfServer stop localhost 17860
You can see this error if you have the translatedBlat / nucleotideBlat port numbers the wrong way around:
Expecting 6 words from server got 2
The following is an example of an error message when attempting to run a DNA sequence query via the web-based BLAT tool after loading a hub, after starting a gfServer instance (from the same dir as the 2bit file). For example, a command to start an instance of gfServer:
gfServer start localhost 17779 -stepSize=5 contigsRenamed.2bit &
Example of a possible error message, from web-based BLAT after attempting a web-based BLAT query:
Error in TCP non-blocking connect() 111 - Connection refused Operation now in progress Sorry, the BLAT/iPCR server seems to be down. Please try again later.
Check the following:
First, make sure your gfServer instance is running.
Type the following command to check for your running gfServer process:
ps aux | grep gfServer
In your genomes.txt file, does your twoBitPath/filename match what you specified in your command to start gfServer?
In your genomes.txt file, is the location of the instance to your gfServer correct?
To check this, you can cd into the directory where you started your gfServer, then type the command:
hostname -i
Your result should be an IP address, for example, '132.249.245.79'.
Now you can test the connection to your port that you specified, with a simple telnet command.
Type in the following command: telnet yourIP yourPort. For example:
telnet 132.249.245.79 17777
The results should read, "Connected to 132.249.245.79".
Otherwise, if gfServer isn't running or if you typed the wrong location in your telnet command, telnet will say, "Connection refused."
In this example, check your genomes.txt file, and make sure your blat line reads, "blat 132.249.245.79 17777".
You may need to change your genomes.txt file from, for example, "blat localhost 17777" to "blat 132.249.245.79 17777" (use your specific IP/host name where gfServer is running).
To request status from the gfServer process, run: gfServer status yourLocation yourPort.
For example:
$ gfServer status 132.249.245.79 17777
You should see output like this:
version 36x2 type nucleotide host localhost port 17777 tileSize 11 stepSize 5 minMatch 2 pcr requests 0 blat requests 0 bases 0 misses 0 noSig 1 trimmed 0 warnings 0
The best troubleshooting test is to take the webpage out of the equation, and use the command line utility, gfClient, to run the query on your instance of gfServer. If you can successfully connect gfClient to gfServer, you will know that your location and port specification are correct.
From the directory that holds your hub's .2bit file (should be the same directory where your instance of gfServer was launched), perform a query using gfClient:
You can type "gfClient" on your command line to see the usage statement.
Use the following command: gfClient yourLocation yourPort pathOf2bitFile yourFastaQuery.fa nameOfOutputFile.psl
FYI: For testing with gfClient, you only need the gfServer binary on your server, not blat.
For example:
gfClient localhost 17777 . query.fa gfOutput.psl
Note the "." after the port, to specify that the query will use the .2bit file in the current directory. After running this command, take a look at the gfOutput.psl file. If successful, you will see BLAT results.
Another example:
Note: In the example below, "yourServer.yourInstitution.edu" is the name of their machine where you run the gfServer command.
From the test machine: Test the DNA alignment, where test.fa is some sequence to find:
gfClient yourServer.yourInstitution.edu 17779 `pwd` test.fa dnaTestOut.psl
From the test machine: Test the protein alignment, where proteinSequence.fa is the sequence to find:
gfClient -t=dnaX -q=prot yourServer.yourInstitution.edu 17779 `pwd` proteinSequence.fa proteinOut.psl
pwd says to find the yourAssembly.2bit file in this directory.A dynamic BLAT server is specified with the "dynamic" argument to the blat, transBlat, isPcr definitions in the hub genomes.txt file, followed by the gfServer root-relative path of the directory containing the 2bit and gfidx files.
For example:
blat yourServer.yourInstitution.edu 4096 dynamic yourAssembly transBlat yourServer.yourInstitution.edu 4096 dynamic yourAssembly isPcr yourServer.yourInstitution.edu 4096 dynamic yourAssembly
The genome and gfServer indexes would be:
$rootdir/yourAssembly/yourAssembly.2bit $rootdir/yourAssembly/yourAssembly.untrans.gfidx $rootdir/yourAssembly/yourAssembly.trans.gfidx
See Building gfServer indexes for instructions in building the index.
For large hubs, it is possible to have more deeply nest directory, for instance, the following NCBI convention:
blat yourServer.yourInstitution.edu 4096 dynamic GCF/000/181/335/GCF_000181335.3 transBlat yourServer.yourInstitution.edu 4096 dynamic GCF/000/181/335/GCF_000181335.3 isPcr yourServer.yourInstitution.edu 4096 dynamic GCF/000/181/335/GCF_000181335.3
Which will reference these genome files and indexes:
$rootdir/GCF/000/181/335/GCF_000181335.3/GCF_000181335.3.2bit $rootdir/GCF/000/181/335/GCF_000181335.3/GCF_000181335.3.untrans.gfidx $rootdir/GCF/000/181/335/GCF_000181335.3/GCF_000181335.3.trans.gfidx
A query without specifying a genome is an "I am alive" check:
% gfServer status myserver 4040 version 37x1 serverType dynamic
Specifying a genome checks that is is valid and gives information on how to the index was built:
% gfServer -genome=mm10 -genomeDataDir=test/mm10 status myserver 4040 version 37x1 serverType dynamic type nucleotide tileSize 11 stepSize 5 minMatch 2
Using -trans checks the translated index:
% gfServer -genome=mm10 -genomeDataDir=test/mm10 -trans status myserver 4040 version 37x1 serverType dynamic type translated tileSize 4 stepSize 4 minMatch 3