99599c7130790107ff0de9f043930da6aa7fddf1
angie
  Mon Nov 16 16:35:58 2020 -0800
Scripts for automating SARS-CoV-2 Phylogeny tracks (refs #26530): fetching
sequences and metadata from several public sources, mapping GISAID IDs to
public seq IDs, downloading the latest release of the phylogenetic tree from
github.com/roblanf/sarscov2phylo/ , making VCFs from GISAID and public
sequences, and using github.com/yatisht/usher to resolve ambiguous alleles,
make protobuf files for hgPhyloPlace, and add public sequences that have not
been mapped to GISAID sequences to the sarscov2phylo tree for a comprehensive
public tree+VCF.

This is still not fully otto-mated because certain crucial inputs like
GISAID sequences still must be downloaded using a web browser, but the goal
is to automate as much as possible and maybe someday have it fully cron-driven.

There are two main top-level scripts which call other scripts, which may in turn
call scripts, in this hierarchy:

updateIdMapping.sh
getCogUk.sh
getNcbi.sh
searchAllSarsCov2BioSample.sh
bioSampleIdToText.sh
bioSampleTextToTab.pl
gbMetadataAddBioSample.pl
fixNcbiFastaNames.pl

updateSarsCov2Phylo.sh
getRelease.sh
processRelease.sh
cladeLineageColors.pl
mapPublic.sh
extractUnmappedPublic.sh
addUnmappedPublic.sh

many of the above:
util.sh

publicCredits.sh will hopefully be folded into updateSarsCov2Phylo.sh when I
figure out how to automate fetching of author/institution metadata from NCBI
and COG-UK.

diff --git src/hg/utils/otto/sarscov2phylo/searchAllSarsCov2BioSample.sh src/hg/utils/otto/sarscov2phylo/searchAllSarsCov2BioSample.sh
new file mode 100755
index 0000000..10eecb2
--- /dev/null
+++ src/hg/utils/otto/sarscov2phylo/searchAllSarsCov2BioSample.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+set -beEu -o pipefail
+
+# Entrez search for all SARS-CoV-2 sequences with length >= 29,000
+# "Severe acute respiratory syndrome coronavirus 2"[Organism] AND ("29000"[SLEN] : "35000"[SLEN])
+query='%22Severe%20acute%20respiratory%20syndrome%20coronavirus%202%22%5BOrganism%5D'
+
+tool=searchAllSarsCov2.sh
+email="$USER%40soe.ucsc.edu"
+
+# Assemble the esearch URL
+base="https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
+url="${base}esearch.fcgi?db=biosample&term=$query&tool=$tool&email=$email&retmax=1000000"
+
+curl -s -S "$url" \
+| grep "<Id>" \
+| sed -re 's@\s*<Id>([0-9]+)</Id>@\1@;' \
+    > all.biosample.gids.txt