Commits for angie

v323_preview2 to v323_base (2015-09-28 to 2015-10-05) v323

• hFindTableInfoWithConn was checking table existence when chrom was given,
  but not when chrom was NULL, and so it failed to check existence for a
  table when looking in hgFixed. Now it always checks existence.
  Also, it uses hTableExists (cached) instead of sqlTableExists.
  refs #16112
  
  • src/hg/lib/hdb.c - lines changed 6, context: html, text, full: html, text
• Add test cases (and bug fixes!) to make sure that we get the desired
  behavior when an insertion is adjacent to a non-insertion, and when
  an insertion falls at the start or end of the search region (if applicable):
  
  - Include insertions that fall at the start or end of the search region
  
  - If the primary row is an insertion, keep secondary non-insertion rows
  to the left and right.
  
  - If the primary row is a non-insertion, keep secondary insertion rows
  at its start and end.
  
  (i.e. keep insertions at boundaries -- but don't let any non-insertions
  slip through)
  
  The VCF logic is more complicated because VCF indels always include an
  extra base to the left, so they appear to start before they actually do,
  and can be interspersed with non-indels that start there.
  
  • src/hg/lib/annoStreamDb.c - lines changed 164, context: html, text, full: html, text
  • src/hg/lib/bed.sql - lines changed 1, context: html, text, full: html, text
  • src/hg/lib/tests/annoGratorTester.c - lines changed 64, context: html, text, full: html, text
  • src/hg/lib/tests/annoGratorTests.mk - lines changed 15, context: html, text, full: html, text
  • src/hg/lib/tests/expected/annoGrator/insertions.txt - lines changed 212, context: html, text, full: html, text
  • src/hg/lib/tests/input/annoGrator/insertionsPrimary.bb - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
  • src/hg/lib/tests/input/annoGrator/insertionsPrimary.bed - lines changed 16, context: html, text, full: html, text
  • src/hg/lib/tests/input/annoGrator/insertionsPrimary.vcf - lines changed 18, context: html, text, full: html, text
  • src/hg/lib/tests/input/annoGrator/insertionsSecondary.bb - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
  • src/hg/lib/tests/input/annoGrator/insertionsSecondary.bed - lines changed 25, context: html, text, full: html, text
  • src/hg/lib/tests/input/annoGrator/insertionsSecondary.vcf - lines changed 43, context: html, text, full: html, text
  • src/lib/annoGrator.c - lines changed 49, context: html, text, full: html, text
  • src/lib/annoStreamTab.c - lines changed 36, context: html, text, full: html, text
  • src/lib/annoStreamVcf.c - lines changed 24, context: html, text, full: html, text
  • src/lib/bigBed.c - lines changed 9, context: html, text, full: html, text
• Added another test case: first item in primary is an insertion and first item
  in secondary is adjacent on the left.
  
  • src/hg/lib/tests/expected/annoGrator/insertions.txt - lines changed 4, context: html, text, full: html, text
  • src/hg/lib/tests/input/annoGrator/insertionsPrimary.bb - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
  • src/hg/lib/tests/input/annoGrator/insertionsPrimary.bed - lines changed 2, context: html, text, full: html, text
  • src/hg/lib/tests/input/annoGrator/insertionsPrimary.vcf - lines changed 9, context: html, text, full: html, text
  • src/hg/lib/tests/input/annoGrator/insertionsSecondary.bb - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
  • src/hg/lib/tests/input/annoGrator/insertionsSecondary.bed - lines changed 2, context: html, text, full: html, text
  • src/hg/lib/tests/input/annoGrator/insertionsSecondary.vcf - lines changed 5, context: html, text, full: html, text
• Test & fix VCF+tabix querying of insertion items.
  
  • src/hg/lib/tests/annoGratorTester.c - lines changed 10, context: html, text, full: html, text
  • src/hg/lib/tests/expected/annoGrator/insertions.txt - lines changed 96, context: html, text, full: html, text
  • src/hg/lib/tests/input/annoGrator/insertionsPrimary.vcf.gz - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
  • src/hg/lib/tests/input/annoGrator/insertionsPrimary.vcf.gz.tbi - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
  • src/hg/lib/tests/input/annoGrator/insertionsSecondary.vcf - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
  • src/hg/lib/tests/input/annoGrator/insertionsSecondary.vcf.gz - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
  • src/hg/lib/tests/input/annoGrator/insertionsSecondary.vcf.gz.tbi - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
  • src/lib/annoStreamVcf.c - lines changed 4, context: html, text, full: html, text
• On very long queries like selected fields of snp144 chr1 (>11M rows),
  hgIntegrator was spending about half of the time repetitively building
  up sourceName.columnName strings and looking them up in a hash to tell
  whether or not to include each column's data in the output. To avoid
  the waste, now it builds up an array (one for each source, primary +
  grators) of arrays of flags (one flag for each column) so it can do
  an array lookup instead of string-munging and hash lookup.
  ~2x speedup for selected fields of snp144 chr1 (4 minutes -> 2 minutes).
  mysql time is about half that; much of the other half is copying data
  in annoStreamDb.c:bufferRowsFromSqlQuery.
  
  • src/lib/annoFormatTab.c - lines changed 72, context: html, text, full: html, text