353a34ac7e7638457db3f55e57452069113d860b braney Sun Sep 6 13:34:28 2026 -0700 Add a bigNet track type, a net of alignments in a bigBed, refs #20824 Track hubs have had no way to show a real net. The usual stand-in is a net rendered as a maf, which loses the level structure that makes a net useful for establishing orthologous sequence. bigNet holds the netAlign columns in a bigBed, so a hub can carry the net itself. The format is bed6+20: the target in chrom/chromStart/chromEnd, the query sequence in name, the query strand in strand, then level and the rest of the netAlign fields. The trackDb line is type bigNet mirroring type netAlign. chainTrack is the plain trackDb name of the bigChain track in the same hub; hgc adds the hub prefix itself. chainNetLoadRangeHub() builds a chainNet from a bigBed range query and hands it to the same helpToNet() the SQL path uses, so the nesting is rebuilt the same way. netDraw picks its loader off tg->isBigBed and the drawing code below that is untouched. genericNetClick does the same for the details page and follows the named chain track for the alignment. Also bounds the level walk in helpToNet() by help->maxDepth. It could read one past the end of the levels array. netToBigNet converts a net file to bedToBigBed input. It writes the tab line itself rather than calling bigNetTabOut, because autoSql prints a double with %g and that drops digits off a chain score. diff --git src/hg/htdocs/goldenPath/help/bigNet.html src/hg/htdocs/goldenPath/help/bigNet.html new file mode 100644 index 00000000000..ae12e195fc6 --- /dev/null +++ src/hg/htdocs/goldenPath/help/bigNet.html @@ -0,0 +1,176 @@ + + + + + + + +

bigNet Track Format

+A net describes which parts of one genome line up with which parts of another. It is built from +chains, and it sorts them into levels. The top level holds the best chain +over each part of the target genome. Inside the gaps of that chain sit the next best chains, and so +on. The net is how the Genome Browser shows orthology between two assemblies, because it says which +alignment is the real counterpart of a region rather than just one of many possible ones.

+The bigNet format holds the same data as a net file, but compressed and +indexed as a bigBed. Net files are converted to bigNet +files with the program bedToBigBed, run with the -as option to pull in a +special autoSql +(.as) file that defines the fields of the bigNet.

+The bigNet files are in an indexed binary format. The main advantage of this format is that only +those portions of the file needed to display a particular region are transferred to the Genome +Browser server. The bigNet file stays on your own web-accessible server (http, https or ftp), not on +the UCSC server, and only the portion needed for the currently displayed chromosomal position is +locally cached as a "sparse file". If you do not have access to a web-accessible server +and need hosting space for your bigNet files, please see the +Hosting section of the Track Hub Help documentation.

+A bigNet track is used in a track hub. It is not available as a +custom track.

+ + +

bigNet format definition

+The following autoSql definition is used to specify bigNet files. Save it as bigNet.as. It +is pulled in when the bedToBigBed utility is run with the -as=bigNet.as +option. The same file is in the UCSC source tree at src/hg/lib/bigNet.as.

    table bigNet
+    "bigNet - a net of pairwise alignments in bigBed format"
+        (
+        string chrom;       "Reference sequence chromosome or scaffold"
+        uint   chromStart;  "Start position in chromosome"
+        uint   chromEnd;    "End position in chromosome"
+        string name;        "Name of the query sequence"
+        uint   score;       "Score (0-1000)"
+        char[1] strand;     "+ or - for strand of the query sequence"
+        uint   level;       "Depth in the net.  Odd levels are fills, even levels are gaps"
+        uint   qStart;      "Start of alignment on query sequence"
+        uint   qEnd;        "End of alignment on query sequence"
+        uint   chainId;     "Id of the chain that fills this gap.  0 for a gap"
+        uint   ali;         "Bases in gap-free alignments"
+        double chainScore;  "Score from the chain"
+        string type;        "Syntenic type: gap, top, syn, nonSyn or inv"
+        int    qOver;       "Overlap with parent gap on query side.  -1 for undefined"
+        int    qFar;        "Distance from parent gap on query side.  -1 for undefined"
+        int    qDup;        "Bases with two or more copies in query.  -1 for undefined"
+        int    tN;          "Unsequenced bases on target.  -1 for undefined"
+        int    qN;          "Unsequenced bases on query.  -1 for undefined"
+        int    tR;          "RepeatMasker bases on target.  -1 for undefined"
+        int    qR;          "RepeatMasker bases on query.  -1 for undefined"
+        int    tNewR;       "Lineage specific repeat bases on target.  -1 for undefined"
+        int    qNewR;       "Lineage specific repeat bases on query.  -1 for undefined"
+        int    tOldR;       "Bases of ancient repeats on target.  -1 for undefined"
+        int    qOldR;       "Bases of ancient repeats on query.  -1 for undefined"
+        int    tTrf;        "Bases of tandem repeats on target.  -1 for undefined"
+        int    qTrf;        "Bases of tandem repeats on query.  -1 for undefined"
+        )

+One row is one fill or one gap of the net. The level field records how deep it sits. +Odd levels are fills and even levels are gaps, so a top level chain has level 1, the +gaps inside it have level 2, and the chains that fill those gaps have +level 3. The Genome Browser rebuilds the nesting from these levels and the target +coordinates, so a row must be contained by the row one level above it.

+The last thirteen fields are filled in by the netClass and netSyntenic +programs. If you have not run them, those fields hold -1 and the Genome Browser leaves them out of +the details page.

+Note that the bedToBigBed utility uses a substantial amount of memory: approximately +25% more RAM than the uncompressed BED input file.

+ +

Creating a bigNet track

+A bigNet track shows only the net. To see the alignment behind a piece of the net, the browser needs +the chains as well, so a bigNet track is paired with a +bigChain track built from the same alignment. Build the bigChain track +first, then follow these steps.

+Step 1. +Download the bedToBigBed and netToBigNet programs from the UCSC +binary utilities directory.

+Step 2. +Use the fetchChromSizes script from the +same directory to create a +chrom.sizes file for the target assembly (e.g., hg38). Alternatively, download the +chrom.sizes file for any assembly hosted at UCSC from our +downloads page (click on "Full data +set" for any assembly).

+Step 3. +Use the netToBigNet utility to turn the net file into the input for +bedToBigBed:

netToBigNet hg38.mm39.net bigNet.pre

+netToBigNet sorts its output, so it is ready for the next step. It stops with an error +if the net is missing the fields that netClass and netSyntenic add. Pass +-warn to convert such a net anyway.

+Step 4. +Create the bigNet file using the bedToBigBed utility:

bedToBigBed -type=bed6+20 -as=bigNet.as -tab bigNet.pre hg38.chrom.sizes bigNet.bb

+Step 5. +Move the newly created bigNet file (bigNet.bb) to a web-accessible http, https or ftp +location.

+Step 6. +Add the track to the trackDb.txt file of a +track hub. The type line names the query assembly and the chain +track that holds the alignments:

track myChains
+shortLabel Mouse Chain
+longLabel Mouse (mm39) chained alignments
+type bigChain mm39
+bigDataUrl bigChain.bb
+linkDataUrl bigChain.link.bb
+otherDb mm39
+visibility pack
+
+track myNet
+shortLabel Mouse Net
+longLabel Mouse (mm39) alignment net
+type bigNet mm39 myChains
+bigDataUrl bigNet.bb
+otherDb mm39
+visibility full

+The second word of the type line is the query assembly. The third word is the name of the chain +track in the same hub. The browser follows it when you click an item in the net, so that the details +page can show the alignment and link to the other assembly.

+The bedToBigBed program can be run with several additional options. For a full list of +the available options, type bedToBigBed (with no arguments) on the command line to +display the usage message.

+ +

Sharing your data with others

+If you would like to share your bigNet data track with a colleague, learn how to create a URL by +looking at Example 6 on this page.

+ +

Extracting data from the bigNet format

+Because the bigNet format is an extension of bigBed, the +bigBedToBed program can be used to extract data from a bigNet file. This program can be +downloaded from the binary utilities +directory. It can be run to extract all of the data or a portion of it:

bigBedToBed http://myorg.edu/mylab/bigNet.bb -chrom=chr21 -start=0 -end=1000000 out.txt

+You can also use the bigBedSummary and bigBedInfo programs from that +directory to see a summary of the file or its header and field list.

+ +

Troubleshooting

+If you encounter an error when you run the bedToBigBed program, check your input file +for data coordinates that extend past the end of the chromosome. If these are present, run the +bedClip program (available +here) to remove the problematic row:

bedClip bigNet.pre chrom.sizes bigNet.clipped.pre

+If the track draws but every item sits on one row, check that the level field is +filled in and that each row is contained by the row one level above it. The browser draws a row at +the depth its level names, and a level that has no parent is drawn at the top.

+ +