File Changes for angie

v427_base to v428_preview (2022-02-21 to 2022-02-28) v428

• src/hg/utils/otto/sarscov2phylo/combineMetadata.sh
  • lines changed 5, context: html, text, full: html, text
    9734d13d2c7ee8004001a7553145e65af763796a Tue Feb 22 11:06:40 2022 -0800
    Extract Nextstrain clade from full nextclade output now generated by getNcbi.sh and getCogUk.sh.
• src/hg/utils/otto/sarscov2phylo/extractPublicTree.sh
  • lines changed 1, context: html, text, full: html, text
    ecbbcd78daf00901ca5eac7cde487effd342a376 Thu Jan 13 11:33:31 2022 -0800
    Increase max-path-length so we don't lop off all of BA.2 and BA.3 which due to Delta contam & back-muts are on super-long paths.
  • lines changed 2, context: html, text, full: html, text
    d898714060e14c8a185e3fde3d84dafe32f93baa Tue Feb 22 11:05:07 2022 -0800
    Speed up compression by using more threads.
• src/hg/utils/otto/sarscov2phylo/findDropoutContam.pl
  • lines changed 122, context: html, text, full: html, text
    fcaec96345053d29cc56fa0aa8b13f07ecd8951d Mon Feb 14 13:43:12 2022 -0800
    Use Nextclade's new output columns privateNucMutations.{reversionSubstitutions,labeledSubstitutions} to identify partial or contaminated Omicron sequences for removal.
• src/hg/utils/otto/sarscov2phylo/findRefBackfill.pl
  • lines changed 29, context: html, text, full: html, text
    3c022c2be4485ff1f369617943da75bf51e9da70 Tue Feb 22 10:52:54 2022 -0800
    Identify sequences with an implausibly low number of substitutions given sampling month so we can exclude them from the tree.
• src/hg/utils/otto/sarscov2phylo/getCogUk.sh
  • lines changed 13, context: html, text, full: html, text
    35e80a80d8967505aca7819e6c50a224e7ab4338 Thu Jan 13 11:38:00 2022 -0800
    Fasta fetches were timing out. Compressed versions of most files are available, use those. If a fetch fails, resume at the point it failed. Increase split fasta size for nextclade run - more efficient.
  • lines changed 15, context: html, text, full: html, text
    573695f19d9d0217bf98e7e7d06114835a592388 Tue Feb 22 11:02:20 2022 -0800
    Use COG-UK's new compressed download files. Use latest version of nextclade and keep full output.
• src/hg/utils/otto/sarscov2phylo/getNcbi.sh
  • lines changed 1, context: html, text, full: html, text
    72739f248f43d3685eb2ca5fd464f73102fb77a4 Thu Jan 13 11:36:29 2022 -0800
    Increase split fasta size for nextclade run - more efficient.
  • lines changed 23, context: html, text, full: html, text
    b038619ca5e7c1bf6ab5810bfb004dc4c702266b Tue Feb 22 11:02:56 2022 -0800
    Use latest version of nextclade and keep full output.
• src/hg/utils/otto/sarscov2phylo/gisaidFromChunks.sh
  • lines changed 6, context: html, text, full: html, text
    5f1c3399aaa5c1bcd341470a156cb528073546d3 Thu Jan 13 11:39:32 2022 -0800
    Uniqueify to prevent dups from several possible sources.
  • lines changed 6, context: html, text, full: html, text
    3760b502d693a5e3e1540f5eb190c87a6a0e965f Tue Feb 22 11:04:44 2022 -0800
    Strip quotes from nextclade tsv output, speed up compression by using more threads & speed up sed by usin conditional.
• src/hg/utils/otto/sarscov2phylo/includeRecombinants.tsv
  • lines changed 51, context: html, text, full: html, text
    c3ac1c095777767dff7ad31b94d4e71318a6db67 Tue Feb 22 10:55:09 2022 -0800
    Use new scripts findDropoutContam.pl and findRefBackfill.pl to identify problematic sequences using nextclade annotations. Exclude those sequences (with an exception for reported recombinants, via new file includeRecombinants.tsv) from the tree.
  • lines changed 99, context: html, text, full: html, text
    41d0b771bd70603cad81d2985e31c8c4c81d223b Sun Feb 27 11:11:48 2022 -0800
    Add some IDs from recent pango-designation requests.
• src/hg/utils/otto/sarscov2phylo/makeNewMaskedVcf.sh
  • lines changed 4, context: html, text, full: html, text
    a6d479158de520ee012585952f425eec7324bbad Thu Jan 13 11:48:43 2022 -0800
    Exclude problematic sequences upstream of alignment -- avoid waste of time & CPU.
  • lines changed 29, context: html, text, full: html, text
    c3ac1c095777767dff7ad31b94d4e71318a6db67 Tue Feb 22 10:55:09 2022 -0800
    Use new scripts findDropoutContam.pl and findRefBackfill.pl to identify problematic sequences using nextclade annotations. Exclude those sequences (with an exception for reported recombinants, via new file includeRecombinants.tsv) from the tree.
• src/hg/utils/otto/sarscov2phylo/maskDelta.sh
  • lines changed 27, context: html, text, full: html, text
    00842797b9fda733465b9bf732648a492d133cac Wed Jan 12 10:00:10 2022 -0800
    Also mask 21302, 21304 and 21305 in Delta. Mask convergent deletion of 11291-11296 plus 3 bases to the left in Alpha, Beta and Gamma to prevent spurious 'SNVs' at deletion site from messing up the tree; may need to add this for Omicron eventually.
  • lines changed 5, context: html, text, full: html, text
    b96a4b4d3d3b9d7dd2e7a1934ff4e4e57fb1348b Tue Feb 22 11:07:42 2022 -0800
    Mask two new problematic-in-Delta positions (https://github.com/cov-lineages/pango-designation/issues/435)
• src/hg/utils/otto/sarscov2phylo/nextstrain.clade-mutations.tsv
  • lines changed 1, context: html, text, full: html, text
    b9cad96bd7acdb271b5672d6c7703b8819f70d06 Thu Jan 13 11:46:44 2022 -0800
    Tweak Nextstrain 21M (base Omicron) mutations.
  • lines changed 1, context: html, text, full: html, text
    4ef0e866319b686e8087578651bc4ed50586f1d9 Thu Jan 20 16:49:53 2022 -0800
    Mask more Omicron sites as the branch continues to get scrambled by new seqs with reference backfilling & contamination.
• src/hg/utils/otto/sarscov2phylo/pango.clade-mutations.tsv
  • lines changed 2, context: html, text, full: html, text
    cc81e6f15173ad421b5b2ce99756579e5bfedb9c Tue Jan 11 14:56:06 2022 -0800
    Path tweaks for BB.1 and AV.1
  • lines changed 3, context: html, text, full: html, text
    52fa1b2103b94d28b62bc7a25f67bd4d228fa25a Wed Jan 12 09:54:40 2022 -0800
    Added BA.1.1; renamed BB.1 to B.1.621.2 - pango-designation v1.2.122. Also mask AY.32's C21304T because 21302, 21304, 21305 are masked in Delta now (leave AY.89 unchanged so it doesn't conflict with AY.4; slated to be withdrawn.)
  • lines changed 1, context: html, text, full: html, text
    3d412101b5e9aaac062cb53e48c9fe859622fd05 Thu Jan 13 09:42:15 2022 -0800
    AY.89 has been withdrawn - pango-designation v1.2.123
  • lines changed 2, context: html, text, full: html, text
    4ef0e866319b686e8087578651bc4ed50586f1d9 Thu Jan 20 16:49:53 2022 -0800
    Mask more Omicron sites as the branch continues to get scrambled by new seqs with reference backfilling & contamination.
  • lines changed 1, context: html, text, full: html, text
    48a5cc05092c7a1b99fd7a1dffa6df4448724fe7 Thu Feb 3 17:43:51 2022 -0800
    Redo of B.1.1.529 mutations: require mutations common to BA.* except especially dropout-prone locations -- use Ns for those, and add Ns for all mtuations specific to one or two BA.*s.
  • lines changed 11, context: html, text, full: html, text
    4f080ddd47a4464800ae1f5244ccf430b08def3e Thu Feb 3 23:21:56 2022 -0800
    Added new lineages AY.3.4, AY.4.2.4, AY.4.11, AY.4.12, AY.24.1, AY.25.2, AY.25.3, AY.103.1, AY.103.2, AY.122.4 and AY.125.1 - pango-designation v1.2.124
  • lines changed 6, context: html, text, full: html, text
    329c8f653135142be828bf24aecc080eef030318 Tue Feb 15 14:24:20 2022 -0800
    Added AY.4.13, AY.4.14, AY.4.15, AY.4.16, AY.4.17 and AY.124.1.1 -- pango-designation v1.2.125
  • lines changed 6, context: html, text, full: html, text
    32275764350ba48ecfc7ba000716d0f5f886c94d Tue Feb 15 14:48:29 2022 -0800
    Added AY.5.6, AY.25.1.2, AY.39.3, AY.43.8, AY.122.5 and AY.127.2 -- pango-designation v1.2.126
  • lines changed 1, context: html, text, full: html, text
    00b35af5594324410825b5886ef475708a4075e4 Tue Feb 15 15:28:28 2022 -0800
    Added AY.43.9 -- pango-designation v1.2.127
  • lines changed 15, context: html, text, full: html, text
    409232efcfdb35f6fbeca6f009a39466a0d0006c Wed Feb 16 23:45:49 2022 -0800
    Added BA.1.[2-15] except BA.1.11 which is a reversion mirage :(, also BA.1.13.1 and BA.1.15.1 -- pango-designation v1.2.128
  • lines changed 232, context: html, text, full: html, text
    505a2b837cb74dc1ce22743580762cd340e4954b Mon Feb 21 11:52:10 2022 -0800
    Mask A28461G in AY.* lineages; remove AY.96 -- pango-designation v1.2.129
• src/hg/utils/otto/sarscov2phylo/pangoHash.sh
  • lines changed 33, context: html, text, full: html, text
    3ac59e25cc8fb9d681fcb5d52224151b41f36ce5 Mon Feb 14 12:15:00 2022 -0800
    Align, mask and compute md5sum in same was as pangolin, for constructing cache file.
• src/hg/utils/otto/sarscov2phylo/runNextclade.sh
  • lines changed 40, context: html, text, full: html, text
    1d99af58ff1f767eb3497d666eeacc1ab8ac166a Mon Feb 14 12:12:37 2022 -0800
    Script for running nextclade on compressed or uncompressed fasta, keeping full TSV & aligned fasta output, suitable for running on cluster.
• src/hg/utils/otto/sarscov2phylo/updateCombinedTree.sh
  • lines changed 1, context: html, text, full: html, text
    ecbbcd78daf00901ca5eac7cde487effd342a376 Thu Jan 13 11:33:31 2022 -0800
    Increase max-path-length so we don't lop off all of BA.2 and BA.3 which due to Delta contam & back-muts are on super-long paths.
• src/hg/utils/otto/sarscov2phylo/updateIdMapping.sh
  • lines changed 2, context: html, text, full: html, text
    1345e556ff3da44f7efcf83adc5b67868e536044 Tue Feb 22 11:08:53 2022 -0800
    ncov-ingest file with accession mapping is now gzipped.
• src/hg/utils/otto/sarscov2phylo/updateLineageTreePb.sh
  • lines changed 2, context: html, text, full: html, text
    03e6d9865fa98c05dda6bbf5d780b0a2b41d389c Tue Feb 22 11:11:08 2022 -0800
    Remove B.1.1.529 from lineageTree.pb to match pangoLEARN behavior & avoid applying Omicron label to sequences of very poor quality -- requested by Rachel Colquhoun & Andrew Rambaut.
• src/hg/utils/otto/sarscov2phylo/updatePublic.sh
  • lines changed 10, context: html, text, full: html, text
    cf519593dadf5adb63837c46b3e242f3c1ab08c3 Thu Jan 13 11:43:51 2022 -0800
    Fetch from different sources in parallel. Guard against non-zero return from grep (thx Jonathan!)
• src/hg/utils/otto/sarscov2phylo/usherClusterRun.sh
  • lines changed 4, context: html, text, full: html, text
    d97010481bcf72f263f4a6a21010e46bf667c272 Thu Jan 13 11:45:32 2022 -0800
    Shorten time limit for matOptimize cluster run to 2 hours (probably could be 1) because it usually doesn't recognize when it's done.
  • lines changed 6, context: html, text, full: html, text
    665fe7dc2ae7d018811a15f79c42bb29904d8630 Tue Feb 22 11:12:28 2022 -0800
    Add a para push to wait loop, to help with rare problem of a node failure. Clean up intermediate protobuf files-to-be-merged.
• src/hg/utils/otto/sarscov2phylo/util.sh
  • lines changed 6, context: html, text, full: html, text
    716924d36cd6d25551bf5f590cb3f587e987df8b Tue Feb 22 10:56:18 2022 -0800
    Yet more GenBank title craziness to filter out
• src/inc/common.mk
  • lines changed 5, context: html, text, full: html, text
    5430222afb16da6e196ba2bc67bb216f2975f84e Fri Feb 25 15:32:03 2022 -0800
    Removing definition of SSLDIR because it is no longer used in common.mk or any makefile. SSL_DIR is used elsewhere in common.mk.
  • lines changed 9, context: html, text, full: html, text
    1713494a42debdf12ab248a85ec5e4fc828f42b0 Sun Feb 27 11:10:42 2022 -0800
    When CONDA_BUILD is set, skip freetype and get openssl static libs from ${PREFIX}/lib.