File Changes for angie
switch to commits view, user indexv325_base to v326_preview (2015-11-16 to 2015-11-23) v326
- src/hg/hgIntegrator/hgIntegrator.c
- lines changed 348, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- lines changed 5, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- src/hg/hgVai/hgVai.c
- lines changed 7, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- lines changed 27, context: html, text, full: html, text
Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
- src/hg/hgVarAnnogrator/hgVarAnnogrator.c
- lines changed 22, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- src/hg/inc/annoStreamDb.h
- lines changed 16, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- src/hg/inc/cartJson.h
- lines changed 7, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- src/hg/inc/hAnno.h
- lines changed 8, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- lines changed 5, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- src/hg/inc/hashJoin.h
- lines changed 42, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- src/hg/inc/hdb.h
- lines changed 5, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- src/hg/inc/hui.h
- lines changed 5, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- src/hg/inc/joinMixer.h
- lines changed 40, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- src/hg/inc/joiner.h
- lines changed 26, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- src/hg/js/external/immutable.3.2.1.min.js
- lines changed 0, context: html, text, full: html, text
Update Immutable js library to latest version.
- src/hg/js/external/immutable.3.7.4.min.js
- lines changed 0, context: html, text, full: html, text
Update Immutable js library to latest version.
- src/hg/js/external/lodash.3.10.0.compat.min.js
- lines changed 0, context: html, text, full: html, text
Update lodash js library to latest version.
- src/hg/js/external/lodash.3.3.1.compat.min.js
- lines changed 0, context: html, text, full: html, text
Update lodash js library to latest version.
- src/hg/js/external/makefile
- lines changed 1, context: html, text, full: html, text
Update lodash js library to latest version.
- lines changed 1, context: html, text, full: html, text
Update Immutable js library to latest version.
- src/hg/js/model/hgIntegrator/hgIntegratorModel.js
- lines changed 612, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- src/hg/js/model/lib/ImModel.js
- lines changed 49, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- src/hg/js/react/hgIntegrator/hgIntegrator.jsx
- lines changed 156, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- src/hg/lib/annoFormatVep.c
- lines changed 2, context: html, text, full: html, text
Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
- src/hg/lib/annoGratorGpVar.c
- lines changed 1, context: html, text, full: html, text
Interface change: removing setAutoSqlObject method from streamer and
exposing annoGratorSetAutoSqlObject because asObj should never be
externally imposed, but a streamer/grator may have a need to bootstrap
itself without an asObj and then install one.
refs #15544
- src/hg/lib/annoStreamDb.c
- lines changed 51, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- lines changed 11, context: html, text, full: html, text
Libifying lmCloneRow, adding lmCloneRowExt for future use by annoStreamDb.
- lines changed 572, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- src/hg/lib/annoStreamDbFactorSource.c
- lines changed 1, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- lines changed 8, context: html, text, full: html, text
Interface change: removing setAutoSqlObject method from streamer and
exposing annoGratorSetAutoSqlObject because asObj should never be
externally imposed, but a streamer/grator may have a need to bootstrap
itself without an asObj and then install one.
refs #15544
- src/hg/lib/annoStreamDbKnownGene.c
- lines changed 1, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- lines changed 8, context: html, text, full: html, text
Interface change: removing setAutoSqlObject method from streamer and
exposing annoGratorSetAutoSqlObject because asObj should never be
externally imposed, but a streamer/grator may have a need to bootstrap
itself without an asObj and then install one.
refs #15544
- lines changed 170, context: html, text, full: html, text
Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
- src/hg/lib/annoStreamWig.c
- lines changed 1, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- src/hg/lib/cartJson.c
- lines changed 10, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- src/hg/lib/hAnno.c
- lines changed 7, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- lines changed 11, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- lines changed 13, context: html, text, full: html, text
Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
- src/hg/lib/hashJoin.c
- lines changed 312, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- src/hg/lib/hdb.c
- lines changed 18, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- src/hg/lib/hui.c
- lines changed 43, context: html, text, full: html, text
In case asFromTableDescriptions is called many times, cache the result of sqlTableExists to avoid unnecessary repeated queries.
- lines changed 25, context: html, text, full: html, text
Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- src/hg/lib/joinMixer.c
- lines changed 363, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- src/hg/lib/joiner.c
- lines changed 2, context: html, text, full: html, text
Use hTableExists (cached) instead of sqlTableExists.
- lines changed 132, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- src/hg/lib/jsHelper.c
- lines changed 1, context: html, text, full: html, text
Update lodash js library to latest version.
- lines changed 1, context: html, text, full: html, text
Update Immutable js library to latest version.
- src/hg/lib/makefile
- lines changed 2, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- lines changed 1, context: html, text, full: html, text
Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
- src/hg/lib/tests/annoGratorTester.c
- lines changed 3, context: html, text, full: html, text
Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- src/inc/annoGrator.h
- lines changed 3, context: html, text, full: html, text
Interface change: removing setAutoSqlObject method from streamer and
exposing annoGratorSetAutoSqlObject because asObj should never be
externally imposed, but a streamer/grator may have a need to bootstrap
itself without an asObj and then install one.
refs #15544
- src/inc/annoStreamer.h
- lines changed 3, context: html, text, full: html, text
Interface change: removing setAutoSqlObject method from streamer and
exposing annoGratorSetAutoSqlObject because asObj should never be
externally imposed, but a streamer/grator may have a need to bootstrap
itself without an asObj and then install one.
refs #15544
- src/inc/jsonWrite.h
- lines changed 1, context: html, text, full: html, text
When a NULL tag ("var") was passed to jsonListStart, it wasn't printing the
preceding comma if one was needed. To make NULL tag universally supported
by everything that calls jsonWriteTag, change jsonWriteTag to always print
the preceding comma if necessary, and print the tag only if non-NULL.
- src/inc/localmem.h
- lines changed 8, context: html, text, full: html, text
Libifying lmCloneRow, adding lmCloneRowExt for future use by annoStreamDb.
- src/inc/obscure.h
- lines changed 5, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- src/lib/annoFormatTab.c
- lines changed 2, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- src/lib/annoGrator.c
- lines changed 17, context: html, text, full: html, text
Interface change: removing setAutoSqlObject method from streamer and
exposing annoGratorSetAutoSqlObject because asObj should never be
externally imposed, but a streamer/grator may have a need to bootstrap
itself without an asObj and then install one.
refs #15544
- src/lib/annoRow.c
- lines changed 6, context: html, text, full: html, text
Libifying lmCloneRow, adding lmCloneRowExt for future use by annoStreamDb.
- src/lib/annoStreamer.c
- lines changed 6, context: html, text, full: html, text
Interface change: removing setAutoSqlObject method from streamer and
exposing annoGratorSetAutoSqlObject because asObj should never be
externally imposed, but a streamer/grator may have a need to bootstrap
itself without an asObj and then install one.
refs #15544
- src/lib/jsonWrite.c
- lines changed 14, context: html, text, full: html, text
When a NULL tag ("var") was passed to jsonListStart, it wasn't printing the
preceding comma if one was needed. To make NULL tag universally supported
by everything that calls jsonWriteTag, change jsonWriteTag to always print
the preceding comma if necessary, and print the tag only if non-NULL.
- src/lib/localmem.c
- lines changed 21, context: html, text, full: html, text
Libifying lmCloneRow, adding lmCloneRowExt for future use by annoStreamDb.
- src/lib/obscure.c
- lines changed 15, context: html, text, full: html, text
Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
switch to commits view, user index