All File Changes

v325_base to v326_preview (2015-11-16 to 2015-11-23) v326

• src/hg/cirm/cdw/cdwWebBrowse/cdwWebBrowse.c
  • lines changed 6, context: html, text, full: html, text
    changes after code review, freeing memory, refs #16383
    
  • lines changed 1, context: html, text, full: html, text
    giving cdw datasets their own subdir, refs #16300
    
• src/hg/cirm/cdw/lib/cdwDataset.as
  • lines changed 1, context: html, text, full: html, text
    extending dataset description refs #16300
    
• src/hg/cirm/cdw/lib/cdwDataset.sql
  • lines changed 1, context: html, text, full: html, text
    extending dataset description refs #16300
    
• src/hg/cirm/cdw/lib/makefile
  • lines changed 1, context: html, text, full: html, text
    extending dataset description refs #16300
    
• src/hg/cirm/cdw/wrangle/addMetaDataToJson/addMetaDataToJson
  • lines changed 43, context: html, text, full: html, text
    finishing up the program, making it more general and added in a function to change the corresponding .html file. refs #16182
    
• src/hg/cirm/cdw/wrangle/condenseMatrix/condenseMatrix
  • lines changed 6, context: html, text, full: html, text
    Changing the condense to a sum rather than an average, refs #16182
    
• src/hg/cirm/cdw/wrangle/kallistoOnFastqMani/kallistoOnFastqMani
  • lines changed 49, context: html, text, full: html, text
    Adding changes to a suprogram in the rna seq pipeline, the one that handles Kallisto. refs #16244
    
• src/hg/cirm/cdw/wrangle/kallistoToMatrix/kallistoToMatrix
  • lines changed 145, context: html, text, full: html, text
    adding a fast tool to convert kallisto output to a matrix, refs #16300
    
  • lines changed 53, context: html, text, full: html, text
    map transcripts to genes, refs #16300
    
  • lines changed 28, context: html, text, full: html, text
    mapping genes that don't have symbols, refs #16300
    
• src/hg/cirm/cdw/wrangle/kallistoToMatrix/makefile
  • lines changed 4, context: html, text, full: html, text
    adding a fast tool to convert kallisto output to a matrix, refs #16300
    
• src/hg/cirm/cdw/wrangle/makefile
  • lines changed 2, context: html, text, full: html, text
    adding the new tool to the wrangle makefile
    
• src/hg/cirm/cdw/wrangle/rnaSeqToClustersPipeline/rnaSeqToClustersPipeline
  • lines changed 91, context: html, text, full: html, text
    Adding in some changes to make the pipeline more universal, refs #16244
    
• src/hg/cirm/cdw/wrangle/rnaSeqToClustersPipeline/tests/mouse/README
  • lines changed 10, context: html, text, full: html, text
    Adding a mouse test data set for the rna seq pipeline, it should be very small. refs #16244
    
• src/hg/cirm/cdw/wrangle/rnaSeqToClustersPipeline/tests/mouse/fastqFiles/fq1-R1-001.fastq.gz
  • lines changed 0, context: html, text, full: html, text
    Adding a mouse test data set for the rna seq pipeline, it should be very small. refs #16244
    
• src/hg/cirm/cdw/wrangle/rnaSeqToClustersPipeline/tests/mouse/fastqFiles/fq1-R2-001.fastq.gz
  • lines changed 0, context: html, text, full: html, text
    Adding a mouse test data set for the rna seq pipeline, it should be very small. refs #16244
    
• src/hg/cirm/cdw/wrangle/rnaSeqToClustersPipeline/tests/mouse/fastqFiles/fq2-R1-001.fastq.gz
  • lines changed 0, context: html, text, full: html, text
    Adding a mouse test data set for the rna seq pipeline, it should be very small. refs #16244
    
• src/hg/cirm/cdw/wrangle/rnaSeqToClustersPipeline/tests/mouse/fastqFiles/fq2-R2-001.fastq.gz
  • lines changed 0, context: html, text, full: html, text
    Adding a mouse test data set for the rna seq pipeline, it should be very small. refs #16244
    
• src/hg/cirm/cdw/wrangle/rnaSeqToClustersPipeline/tests/mouse/manifest.txt
  • lines changed 5, context: html, text, full: html, text
    Adding a mouse test data set for the rna seq pipeline, it should be very small. refs #16244
    
• src/hg/expMatrixToJson/expMatrixToJson.c
  • lines changed 31, context: html, text, full: html, text
    Added new javascript code to the c program, refs #16341 and #16216
    
• src/hg/gsBig/gsBig.c
  • lines changed 4, context: html, text, full: html, text
    Fix for issue where gsBig failed to merge segments after it had skipped over one containing all Ns, refs #16220
    
• src/hg/hgCustom/hgCustom.c
  • lines changed 1, context: html, text, full: html, text
    User requested allowing longer URL
    
• src/hg/hgIntegrator/hgIntegrator.c
  • lines changed 348, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
  • lines changed 5, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
• src/hg/hgTracks/gtexTracks.c
  • lines changed 123, context: html, text, full: html, text
    Initial code for variable height layout of GTEx gene track
    
  • lines changed 33, context: html, text, full: html, text
    Mostly working with variable heigt item layout in full mode.
    
  • lines changed 41, context: html, text, full: html, text
    Start on variable height pack layout.
    
  • lines changed 36, context: html, text, full: html, text
    Finish variable height item packing. Adds an optional rowSizes field to spaceSaver. refs #15645
    
  • lines changed 44, context: html, text, full: html, text
    Add difference graph feature to GTEx track. refs #15645
    
• src/hg/hgTracks/hgTracks.c
  • lines changed 1, context: html, text, full: html, text
    adding a keyboard shortcut for the get DNA menu, s t for now, may change
    later, refs #16397
    
  • lines changed 1, context: html, text, full: html, text
    adding help for new keyboard shortcut, refs #16397
    
  • lines changed 1, context: html, text, full: html, text
    Finish variable height item packing. Adds an optional rowSizes field to spaceSaver. refs #15645
    
• src/hg/hgTracks/hgTracks.h
  • lines changed 4, context: html, text, full: html, text
    Initial code for variable height layout of GTEx gene track
    
• src/hg/hgTracks/simpleTracks.c
  • lines changed 22, context: html, text, full: html, text
    Initial code for variable height layout of GTEx gene track
    
  • lines changed 16, context: html, text, full: html, text
    Finish variable height item packing. Adds an optional rowSizes field to spaceSaver. refs #15645
    
• src/hg/hgVai/hgVai.c
  • lines changed 7, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
  • lines changed 27, context: html, text, full: html, text
    Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
    
• src/hg/hgVarAnnogrator/hgVarAnnogrator.c
  • lines changed 22, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
• src/hg/hgc/gtexClick.c
  • lines changed 8, context: html, text, full: html, text
    Fix hgc crash when a tissue has only zero-valued expression for a gene. refs #15645
    
• src/hg/htdocs/goldenPath/help/hubQuickStartAssembly.html
  • lines changed 1, context: html, text, full: html, text
    Adding some bolded text to Assembly Hub Quickstart Guide
    
• src/hg/htdocs/goldenPath/pubs.html
  • lines changed 10, context: html, text, full: html, text
    Adding newest NAR database paper to pubs page, no redmine
    
• src/hg/htdocs/staff.html
  • lines changed 92, context: html, text, full: html, text
    Updating staff page with new hires and updated positions, refs #16418
    
  • lines changed 5, context: html, text, full: html, text
    Updating staff page with small fixes, refs #16418
    
• src/hg/inc/annoStreamDb.h
  • lines changed 16, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
• src/hg/inc/cartJson.h
  • lines changed 7, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
• src/hg/inc/gtexUi.h
  • lines changed 5, context: html, text, full: html, text
    Add difference graph feature to GTEx track. refs #15645
    
• src/hg/inc/hAnno.h
  • lines changed 8, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
  • lines changed 5, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
• src/hg/inc/hashJoin.h
  • lines changed 42, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
• src/hg/inc/hdb.h
  • lines changed 5, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
• src/hg/inc/hui.h
  • lines changed 5, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
• src/hg/inc/joinMixer.h
  • lines changed 40, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
• src/hg/inc/joiner.h
  • lines changed 26, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
• src/hg/js/external/immutable.3.2.1.min.js
  • lines changed 0, context: html, text, full: html, text
    Update Immutable js library to latest version.
    
• src/hg/js/external/immutable.3.7.4.min.js
  • lines changed 0, context: html, text, full: html, text
    Update Immutable js library to latest version.
    
• src/hg/js/external/lodash.3.10.0.compat.min.js
  • lines changed 0, context: html, text, full: html, text
    Update lodash js library to latest version.
    
• src/hg/js/external/lodash.3.3.1.compat.min.js
  • lines changed 0, context: html, text, full: html, text
    Update lodash js library to latest version.
    
• src/hg/js/external/makefile
  • lines changed 1, context: html, text, full: html, text
    Update lodash js library to latest version.
    
  • lines changed 1, context: html, text, full: html, text
    Update Immutable js library to latest version.
    
• src/hg/js/hgTracks.js
  • lines changed 7, context: html, text, full: html, text
    adding a keyboard shortcut for the get DNA menu, s t for now, may change
    later, refs #16397
    
• src/hg/js/model/hgIntegrator/hgIntegratorModel.js
  • lines changed 612, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
• src/hg/js/model/lib/ImModel.js
  • lines changed 49, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
• src/hg/js/react/hgIntegrator/hgIntegrator.jsx
  • lines changed 156, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
• src/hg/lib/annoFormatVep.c
  • lines changed 2, context: html, text, full: html, text
    Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
    
• src/hg/lib/annoGratorGpVar.c
  • lines changed 1, context: html, text, full: html, text
    Interface change: removing setAutoSqlObject method from streamer and
    exposing annoGratorSetAutoSqlObject because asObj should never be
    externally imposed, but a streamer/grator may have a need to bootstrap
    itself without an asObj and then install one.
    refs #15544
    
• src/hg/lib/annoStreamDb.c
  • lines changed 51, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
  • lines changed 11, context: html, text, full: html, text
    Libifying lmCloneRow, adding lmCloneRowExt for future use by annoStreamDb.
    
  • lines changed 572, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
• src/hg/lib/annoStreamDbFactorSource.c
  • lines changed 1, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
  • lines changed 8, context: html, text, full: html, text
    Interface change: removing setAutoSqlObject method from streamer and
    exposing annoGratorSetAutoSqlObject because asObj should never be
    externally imposed, but a streamer/grator may have a need to bootstrap
    itself without an asObj and then install one.
    refs #15544
    
• src/hg/lib/annoStreamDbKnownGene.c
  • lines changed 1, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
  • lines changed 8, context: html, text, full: html, text
    Interface change: removing setAutoSqlObject method from streamer and
    exposing annoGratorSetAutoSqlObject because asObj should never be
    externally imposed, but a streamer/grator may have a need to bootstrap
    itself without an asObj and then install one.
    refs #15544
    
  • lines changed 170, context: html, text, full: html, text
    Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
    
• src/hg/lib/annoStreamWig.c
  • lines changed 1, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
• src/hg/lib/cart.c
  • lines changed 2, context: html, text, full: html, text
    Enlarging buffers to prevent overflow, refs #13549
    
• src/hg/lib/cartJson.c
  • lines changed 10, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
• src/hg/lib/gtexUi.c
  • lines changed 22, context: html, text, full: html, text
    Add difference graph feature to GTEx track. refs #15645
    
• src/hg/lib/hAnno.c
  • lines changed 7, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
  • lines changed 11, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
  • lines changed 13, context: html, text, full: html, text
    Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
    
• src/hg/lib/hashJoin.c
  • lines changed 312, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
• src/hg/lib/hdb.c
  • lines changed 18, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
• src/hg/lib/hui.c
  • lines changed 43, context: html, text, full: html, text
    In case asFromTableDescriptions is called many times, cache the result of sqlTableExists to avoid unnecessary repeated queries.
    
  • lines changed 25, context: html, text, full: html, text
    Data Integrator UI for adding related SQL tables and fields for SQL-based
    tracks using all.joiner. This is only the UI -- the implementation will be
    in a subsequent commit. Main changes:
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getRelatedTables
    - the getFields command used to expect a simple comma-sep list of tracks.
    Now it expects a list of ';'-sep track blobs, each of which may include
    multiple comma-sep tables (track plus related tables).
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - fieldSelect now supports multiple tables per track
    - js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
    dialog (FieldSelect) for adding related tables and showing their available
    fields w/checkboxes; corresponding new UI state and events.
    - js/model/hgIntegrator/hgIntegratorModel.js:
    - new handlers for UI events and cartJson data for related tables/fields
    - hgIntegrator.c: new code to return information about what tables (if any)
    are related to each selected track, and what fields are in selected related
    tables.
    
    Other changes:
    - hgIntegratorModel.js:
    - split monolithic handleCartVar into separate handlers
    - add new 'hgi_uiChoices' cart var to remember things that may not pertain
    to the current query, such as a user's related tables/fields selections
    for a table that is not included in the current query.
    - convert between hgi_querySpec and internal UI state representation
    (querySpecUpdateUiChoices, toQuerySpecJS)
    - cart.js: added uiChoices convenience functions, stringifying of cartSet val
    - cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
    - new command getQueryState: bundle querySpec and uiChoices from the cart
    since they need to be processed together
    - UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
    - split UI state's querySpec object into separate dataSources and
    outFileOptions objects (no point in bundling those in the UI)
    
    Thanks to Kate and Matt for helpful suggestions about the UI.
    
    refs #15544
    
• src/hg/lib/joinMixer.c
  • lines changed 363, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
• src/hg/lib/joiner.c
  • lines changed 2, context: html, text, full: html, text
    Use hTableExists (cached) instead of sqlTableExists.
    
  • lines changed 132, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
• src/hg/lib/jsHelper.c
  • lines changed 1, context: html, text, full: html, text
    Update lodash js library to latest version.
    
  • lines changed 1, context: html, text, full: html, text
    Update Immutable js library to latest version.
    
• src/hg/lib/makefile
  • lines changed 2, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
  • lines changed 1, context: html, text, full: html, text
    Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
    
• src/hg/lib/tests/annoGratorTester.c
  • lines changed 3, context: html, text, full: html, text
    Change to interface of annoStreamDbNew: instead of making the caller
    pass in an asObject, the caller may now pass in a parsed JSON config
    object (or NULL). annoStreamDbNew will use this config object to
    determine whether related tables need to be joined with the track table,
    and produce its own asObj. It doesn't actually do that yet -- this
    is just an interface change that will enable it to do so.
    refs #15544
    
• src/hg/makeDb/doc/hg19.txt
  • lines changed 29, context: html, text, full: html, text
    Scripts and documentation for adding ISCA Benign cumulative gain/loss tracks, refs #16122
    
• src/hg/makeDb/doc/hg38/hg38Patch5.txt
  • lines changed 659, context: html, text, full: html, text
    text from hg38Patch3 sed edited for patch5 build refs #16261
    
  • lines changed 2, context: html, text, full: html, text
    Beginning work on hg38p5, refs #16261
    
• src/hg/makeDb/doc/mm10.txt
  • lines changed 240, context: html, text, full: html, text
    start SNP142 on mm10 #15934
    
  • lines changed 130, context: html, text, full: html, text
    dbSnp 142 for mm10 #15934
    
  • lines changed 73, context: html, text, full: html, text
    dbSnp 142 mm10 sequence with substitutions #15934
    
• src/hg/makeDb/genbank/etc/genbank.conf
  • lines changed 1, context: html, text, full: html, text
    Fixing typo in file name, refs #16220
    
  • lines changed 1, context: html, text, full: html, text
    Fixing file location mistake, refs #16220
    
• src/hg/makeDb/hgLoadSeq/hgLoadSeq.c
  • lines changed 1, context: html, text, full: html, text
    fix core dump with -test argument #16034
    
• src/hg/makeDb/schema/all.joiner
  • lines changed 1, context: html, text, full: html, text
    Changing minCheck on hg38 GencodeV23 per Marks request, refs #16128
    
  • lines changed 2, context: html, text, full: html, text
    Added coriellDelDup table to hg38 tablesIgnored, refs #15534
    
  • lines changed 1, context: html, text, full: html, text
    Adding gorGor4 to all.joiner, refs #16220
    
• src/hg/makeDb/trackDb/geneid.html
  • lines changed 19, context: html, text, full: html, text
    Added references refs#15616
    
• src/hg/makeDb/trackDb/human/hg19/tgpPhase3.html
  • lines changed 3, context: html, text, full: html, text
    Changes based on CR, refs #16380
    
• src/hg/makeDb/trackDb/human/hg19/tgpPhase3Accessibility.html
  • lines changed 2, context: html, text, full: html, text
    Changes based on CR, refs #16380
    
• src/hg/makeDb/trackDb/human/hg19/wgEncodeGencodeSuper.html
  • lines changed 1, context: html, text, full: html, text
    Changing html char code for one letter and rearranging PI info for hg38 and mm10 pages, refs #16388
    
• src/hg/makeDb/trackDb/human/hg38/nonCodingRNAs.ra
  • lines changed 10, context: html, text, full: html, text
    Adding the tRNA track to the non-coding RNA supertrack refs #11694
    
• src/hg/makeDb/trackDb/human/hg38/trackDb.wgEncode.ra
  • lines changed 1, context: html, text, full: html, text
    Changing release tags for new Gencode V23 release, refs #16128
    
• src/hg/makeDb/trackDb/human/hg38/wgEncodeGencodeSuper.html
  • lines changed 10, context: html, text, full: html, text
    Changing html char code for one letter and rearranging PI info for hg38 and mm10 pages, refs #16388
    
• src/hg/makeDb/trackDb/human/hg38/wgEncodeGencodeV20.ra
  • lines changed 1, context: html, text, full: html, text
    Changing release tags for new Gencode V23 release, refs #16128
    
• src/hg/makeDb/trackDb/human/hg38/wgEncodeGencodeV22.ra
  • lines changed 1, context: html, text, full: html, text
    Changing release tags for new Gencode V23 release, refs #16128
    
• src/hg/makeDb/trackDb/human/hg38/wgEncodeGencodeV23.html
  • lines changed 8, context: html, text, full: html, text
    Fixed link from V32 to V23, updated PI and Co-PI, moved Rachel from UCSC to former members
    
  • lines changed 1, context: html, text, full: html, text
    Updating Havana manual annotation group link for hg38 V23 release, refs #16128
    
  • lines changed 1, context: html, text, full: html, text
    Changing html char code for one letter and rearranging PI info for hg38 and mm10 pages, refs #16388
    
• src/hg/makeDb/trackDb/human/iscaComposite.html
  • lines changed 3, context: html, text, full: html, text
    Scripts and documentation for adding ISCA Benign cumulative gain/loss tracks, refs #16122
    
• src/hg/makeDb/trackDb/human/iscaCompositeOld.html
  • lines changed 146, context: html, text, full: html, text
    Scripts and documentation for adding ISCA Benign cumulative gain/loss tracks, refs #16122
    
• src/hg/makeDb/trackDb/human/snp142.bugFix.html
  • lines changed 2, context: html, text, full: html, text
    added an s to 1000 Genomes and a missing comma, refs #16204
    
• src/hg/makeDb/trackDb/human/trackDb.ra
  • lines changed 32, context: html, text, full: html, text
    Scripts and documentation for adding ISCA Benign cumulative gain/loss tracks, refs #16122
    
  • lines changed 4, context: html, text, full: html, text
    removed release alpha tags for snp142 tracks, refs #16204
    
  • lines changed 79, context: html, text, full: html, text
    removed old snp142 entries, refs #16204
    
• src/hg/makeDb/trackDb/mouse/mm10/wgEncodeGencodeSuper.html
  • lines changed 10, context: html, text, full: html, text
    Changing html char code for one letter and rearranging PI info for hg38 and mm10 pages, refs #16388
    
• src/hg/makeDb/trackDb/mouse/mm10/wgEncodeGencodeVM4.ra
  • lines changed 3, context: html, text, full: html, text
    changing default visbilties for mm10 GENCODE VM4 track, refs #16129
    
• src/hg/makeDb/trackDb/mouse/snp142.html
  • lines changed 36, context: html, text, full: html, text
    trackDb for snp142 on mm10 #15934
    
• src/hg/makeDb/trackDb/mouse/snp142.shared.html
  • lines changed 471, context: html, text, full: html, text
    trackDb for snp142 on mm10 #15934
    
• src/hg/makeDb/trackDb/mouse/snp142Common.html
  • lines changed 28, context: html, text, full: html, text
    trackDb for snp142 on mm10 #15934
    
• src/hg/makeDb/trackDb/mouse/snp142Mult.html
  • lines changed 25, context: html, text, full: html, text
    trackDb for snp142 on mm10 #15934
    
• src/hg/makeDb/trackDb/mouse/trackDb.ra
  • lines changed 60, context: html, text, full: html, text
    trackDb for snp142 on mm10 #15934
    
• src/hg/utils/automation/doCpgIslands.pl
  • lines changed 72, context: html, text, full: html, text
    more efficient use of resources to improve performance refs #16078
    
• src/hg/utils/automation/makeDownloads.pl
  • lines changed 1, context: html, text, full: html, text
    include refs #15209
    
• src/hg/utils/otto/isca/buildISCA.csh
  • lines changed 10, context: html, text, full: html, text
    Scripts and documentation for adding ISCA Benign cumulative gain/loss tracks, refs #16122
    
• src/hg/utils/otto/isca/isca.tables
  • lines changed 2, context: html, text, full: html, text
    Scripts and documentation for adding ISCA Benign cumulative gain/loss tracks, refs #16122
    
• src/hg/utils/otto/isca/iscaWrapper.sh
  • lines changed 1, context: html, text, full: html, text
    Changing who gets errors from the ISCA build process
    
• src/hg/utils/otto/isca/validateISCA.sh
  • lines changed 2, context: html, text, full: html, text
    Scripts and documentation for adding ISCA Benign cumulative gain/loss tracks, refs #16122
    
• src/inc/annoGrator.h
  • lines changed 3, context: html, text, full: html, text
    Interface change: removing setAutoSqlObject method from streamer and
    exposing annoGratorSetAutoSqlObject because asObj should never be
    externally imposed, but a streamer/grator may have a need to bootstrap
    itself without an asObj and then install one.
    refs #15544
    
• src/inc/annoStreamer.h
  • lines changed 3, context: html, text, full: html, text
    Interface change: removing setAutoSqlObject method from streamer and
    exposing annoGratorSetAutoSqlObject because asObj should never be
    externally imposed, but a streamer/grator may have a need to bootstrap
    itself without an asObj and then install one.
    refs #15544
    
• src/inc/jsonWrite.h
  • lines changed 1, context: html, text, full: html, text
    When a NULL tag ("var") was passed to jsonListStart, it wasn't printing the
    preceding comma if one was needed. To make NULL tag universally supported
    by everything that calls jsonWriteTag, change jsonWriteTag to always print
    the preceding comma if necessary, and print the tag only if non-NULL.
    
• src/inc/localmem.h
  • lines changed 8, context: html, text, full: html, text
    Libifying lmCloneRow, adding lmCloneRowExt for future use by annoStreamDb.
    
• src/inc/net.h
  • lines changed 1, context: html, text, full: html, text
    Making buf size into #def, upsizing another buffer, refs #16406
    
• src/inc/obscure.h
  • lines changed 5, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
• src/inc/spaceSaver.h
  • lines changed 9, context: html, text, full: html, text
    Finish variable height item packing. Adds an optional rowSizes field to spaceSaver. refs #15645
    
• src/lib/annoFormatTab.c
  • lines changed 2, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
• src/lib/annoGrator.c
  • lines changed 17, context: html, text, full: html, text
    Interface change: removing setAutoSqlObject method from streamer and
    exposing annoGratorSetAutoSqlObject because asObj should never be
    externally imposed, but a streamer/grator may have a need to bootstrap
    itself without an asObj and then install one.
    refs #15544
    
• src/lib/annoRow.c
  • lines changed 6, context: html, text, full: html, text
    Libifying lmCloneRow, adding lmCloneRowExt for future use by annoStreamDb.
    
• src/lib/annoStreamer.c
  • lines changed 6, context: html, text, full: html, text
    Interface change: removing setAutoSqlObject method from streamer and
    exposing annoGratorSetAutoSqlObject because asObj should never be
    externally imposed, but a streamer/grator may have a need to bootstrap
    itself without an asObj and then install one.
    refs #15544
    
• src/lib/jsonWrite.c
  • lines changed 14, context: html, text, full: html, text
    When a NULL tag ("var") was passed to jsonListStart, it wasn't printing the
    preceding comma if one was needed. To make NULL tag universally supported
    by everything that calls jsonWriteTag, change jsonWriteTag to always print
    the preceding comma if necessary, and print the tag only if non-NULL.
    
• src/lib/localmem.c
  • lines changed 21, context: html, text, full: html, text
    Libifying lmCloneRow, adding lmCloneRowExt for future use by annoStreamDb.
    
• src/lib/net.c
  • lines changed 1, context: html, text, full: html, text
    User requested allowing longer URL
    
  • lines changed 1, context: html, text, full: html, text
    increasing buf size, refs #16406
    
  • lines changed 2, context: html, text, full: html, text
    Making buf size into #def, upsizing another buffer, refs #16406
    
• src/lib/obscure.c
  • lines changed 15, context: html, text, full: html, text
    Back-end implementation of Data Integrator's support for related tables and
    fields using all.joiner. Most joins are implemented using a new module,
    hashJoin.c; but SQL joins are used in certain cases when hash joins are
    impractical and SQL joins are actually faster. A new module joinMixer
    determines which joins should be implemented by hashJoin vs SQL, and computes
    row indices for hashJoin objects to find keys (from SQL or other hashJoins)
    and store results. The SQL join info from joinMixer is translated into SQL
    queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
    adding the fields from related tables after the fields of the main track table.
    
    Main changes:
    
    - annoStreamDb.c
    - main table SQL query now uses <table>.<field> instead of just <field> to
    avoid clashes with same field name in different tables
    - SQL joins return multiple rows for a single main table row when there are
    multiple matching rows in a related table; these rows need to be squashed
    into one row with the multiple matches comma-separated, both to match
    hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
    - as mentioned above, generate joining SQL queries when necessary and
    generate own asObj including selected fields from related tables.
    - parse JSON config object with relatedTables spec from UI via hgi_querySpec
    
    hashJoin basically slurps a related table into a big hash of keys to values,
    perform lookups (possibly of multiple keys), and formats each column's results.
    It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
    collapse adjacent duplicate matches, commas at end of matches from multiple
    key lookups, reversed order of multiple match values. hgTables/joining.c uses
    arrays of slNames, but in order to avoid all that allocation I'm just glomming
    into an array of reused dyStrings.
    
    joinMixer takes a list of fields to include in output, gets a list of joins
    to be performed (from joinerRouteThroughAll), applies some simple rough
    heuristics to guess whether a join is practical in SQL, and decides which joins
    to do by SQL and which to do by hashJoin. It plans a row format with several
    groups of fields in this order: main table fields, related table fields to
    appear in the output, related table fields needed by hashJoins, hashJoin
    result fields needed by other hashJoins, and hashJoin result fields to appear
    in output. It initializes hashJoins with precomputed row indexes and also
    provides a mapping from big-row columns to the columns that appear in output.
    
    Thanks to Matt for testing on demo6 during development.
    
    refs #15544
    
• src/lib/spaceSaver.c
  • lines changed 29, context: html, text, full: html, text
    Finish variable height item packing. Adds an optional rowSizes field to spaceSaver. refs #15645
    
• src/lib/udc.c
  • lines changed 145, context: html, text, full: html, text
    instrument the UDC layer #16387
    
• src/utils/faTrimRead/faTrimRead.c
  • lines changed 3, context: html, text, full: html, text
    fixup gcc warnings for -Wunused-but-set-variable refs #16121
    
• src/utils/git-reports/git-reports.c
  • lines changed 30, context: html, text, full: html, text
    Adding Full Authors info with full email addresses to git-reports.
    
• src/utils/newPythonProg/newPythonProg.c
  • lines changed 1, context: html, text, full: html, text
    Changing the defaul makefile target from normal to all at Kate's advice, refs# 16352
    
• src/utils/qa/weeklybld/buildEnv.csh
  • lines changed 2, context: html, text, full: html, text
    v326 preview1 refs #16419
    
• src/utils/textHist2/textHist2.c
  • lines changed 2, context: html, text, full: html, text
    fixup gcc warnings for -Wunused-but-set-variable refs #16121
    

• lines changed: 6620
• files changed: 160