Commits for angie
switch to files view, user index
v325_base to v326_preview (2015-11-16 to 2015-11-23) v326
- Update lodash js library to latest version.
- src/hg/js/external/lodash.3.10.0.compat.min.js - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
- src/hg/js/external/lodash.3.3.1.compat.min.js - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
- src/hg/js/external/makefile - lines changed 1, context: html, text, full: html, text
- Update Immutable js library to latest version.
- src/hg/js/external/immutable.3.2.1.min.js - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
- src/hg/js/external/immutable.3.7.4.min.js - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
- src/hg/js/external/makefile - lines changed 1, context: html, text, full: html, text
- When a NULL tag ("var") was passed to jsonListStart, it wasn't printing the
preceding comma if one was needed. To make NULL tag universally supported
by everything that calls jsonWriteTag, change jsonWriteTag to always print
the preceding comma if necessary, and print the tag only if non-NULL.
- In case asFromTableDescriptions is called many times, cache the result of sqlTableExists to avoid unnecessary repeated queries.
- Data Integrator UI for adding related SQL tables and fields for SQL-based
tracks using all.joiner. This is only the UI -- the implementation will be
in a subsequent commit. Main changes:
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getRelatedTables
- the getFields command used to expect a simple comma-sep list of tracks.
Now it expects a list of ';'-sep track blobs, each of which may include
multiple comma-sep tables (track plus related tables).
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- fieldSelect now supports multiple tables per track
- js/react/hgIntegrator/hgIntegrator.jsx: new subsections of 'Choose fields...'
dialog (FieldSelect) for adding related tables and showing their available
fields w/checkboxes; corresponding new UI state and events.
- js/model/hgIntegrator/hgIntegratorModel.js:
- new handlers for UI events and cartJson data for related tables/fields
- hgIntegrator.c: new code to return information about what tables (if any)
are related to each selected track, and what fields are in selected related
tables.
Other changes:
- hgIntegratorModel.js:
- split monolithic handleCartVar into separate handlers
- add new 'hgi_uiChoices' cart var to remember things that may not pertain
to the current query, such as a user's related tables/fields selections
for a table that is not included in the current query.
- convert between hgi_querySpec and internal UI state representation
(querySpecUpdateUiChoices, toQuerySpecJS)
- cart.js: added uiChoices convenience functions, stringifying of cartSet val
- cartJson interface between hgIntegrator.c and hgIntegratorModel.js:
- new command getQueryState: bundle querySpec and uiChoices from the cart
since they need to be processed together
- UI state interface between hgIntegrator.jsx and hgIntegratorModel.js:
- split UI state's querySpec object into separate dataSources and
outFileOptions objects (no point in bundling those in the UI)
Thanks to Kate and Matt for helpful suggestions about the UI.
refs #15544
- src/hg/hgIntegrator/hgIntegrator.c - lines changed 348, context: html, text, full: html, text
- src/hg/js/model/hgIntegrator/hgIntegratorModel.js - lines changed 612, context: html, text, full: html, text
- src/hg/js/model/lib/ImModel.js - lines changed 49, context: html, text, full: html, text
- src/hg/js/react/hgIntegrator/hgIntegrator.jsx - lines changed 156, context: html, text, full: html, text
- Change to interface of annoStreamDbNew: instead of making the caller
pass in an asObject, the caller may now pass in a parsed JSON config
object (or NULL). annoStreamDbNew will use this config object to
determine whether related tables need to be joined with the track table,
and produce its own asObj. It doesn't actually do that yet -- this
is just an interface change that will enable it to do so.
refs #15544
- src/hg/hgIntegrator/hgIntegrator.c - lines changed 5, context: html, text, full: html, text
- src/hg/hgVarAnnogrator/hgVarAnnogrator.c - lines changed 22, context: html, text, full: html, text
- src/hg/inc/annoStreamDb.h - lines changed 16, context: html, text, full: html, text
- src/hg/lib/annoStreamDb.c - lines changed 51, context: html, text, full: html, text
- src/hg/lib/annoStreamDbFactorSource.c - lines changed 1, context: html, text, full: html, text
- src/hg/lib/annoStreamDbKnownGene.c - lines changed 1, context: html, text, full: html, text
- src/hg/lib/annoStreamWig.c - lines changed 1, context: html, text, full: html, text
- src/hg/lib/tests/annoGratorTester.c - lines changed 3, context: html, text, full: html, text
- Interface change: removing setAutoSqlObject method from streamer and
exposing annoGratorSetAutoSqlObject because asObj should never be
externally imposed, but a streamer/grator may have a need to bootstrap
itself without an asObj and then install one.
refs #15544
- src/hg/lib/annoGratorGpVar.c - lines changed 1, context: html, text, full: html, text
- src/hg/lib/annoStreamDbFactorSource.c - lines changed 8, context: html, text, full: html, text
- src/hg/lib/annoStreamDbKnownGene.c - lines changed 8, context: html, text, full: html, text
- Use hTableExists (cached) instead of sqlTableExists.
- Libifying lmCloneRow, adding lmCloneRowExt for future use by annoStreamDb.
- src/hg/lib/annoStreamDb.c - lines changed 11, context: html, text, full: html, text
- Back-end implementation of Data Integrator's support for related tables and
fields using all.joiner. Most joins are implemented using a new module,
hashJoin.c; but SQL joins are used in certain cases when hash joins are
impractical and SQL joins are actually faster. A new module joinMixer
determines which joins should be implemented by hashJoin vs SQL, and computes
row indices for hashJoin objects to find keys (from SQL or other hashJoins)
and store results. The SQL join info from joinMixer is translated into SQL
queries in annoStreamDb. annoStreamDb also generates its own autoSql asObject,
adding the fields from related tables after the fields of the main track table.
Main changes:
- annoStreamDb.c
- main table SQL query now uses <table>.<field> instead of just <field> to
avoid clashes with same field name in different tables
- SQL joins return multiple rows for a single main table row when there are
multiple matching rows in a related table; these rows need to be squashed
into one row with the multiple matches comma-separated, both to match
hgTables behavior and to avoid overflow of rowBuf. (glomSqlDup)
- as mentioned above, generate joining SQL queries when necessary and
generate own asObj including selected fields from related tables.
- parse JSON config object with relatedTables spec from UI via hgi_querySpec
hashJoin basically slurps a related table into a big hash of keys to values,
perform lookups (possibly of multiple keys), and formats each column's results.
It includes a lot of tweaks to match hgTables/joining.c output char-for-char:
collapse adjacent duplicate matches, commas at end of matches from multiple
key lookups, reversed order of multiple match values. hgTables/joining.c uses
arrays of slNames, but in order to avoid all that allocation I'm just glomming
into an array of reused dyStrings.
joinMixer takes a list of fields to include in output, gets a list of joins
to be performed (from joinerRouteThroughAll), applies some simple rough
heuristics to guess whether a join is practical in SQL, and decides which joins
to do by SQL and which to do by hashJoin. It plans a row format with several
groups of fields in this order: main table fields, related table fields to
appear in the output, related table fields needed by hashJoins, hashJoin
result fields needed by other hashJoins, and hashJoin result fields to appear
in output. It initializes hashJoins with precomputed row indexes and also
provides a mapping from big-row columns to the columns that appear in output.
Thanks to Matt for testing on demo6 during development.
refs #15544
- src/hg/inc/joinMixer.h - lines changed 40, context: html, text, full: html, text
- src/hg/lib/annoStreamDb.c - lines changed 572, context: html, text, full: html, text
- src/hg/lib/hashJoin.c - lines changed 312, context: html, text, full: html, text
- src/hg/lib/joinMixer.c - lines changed 363, context: html, text, full: html, text
- src/lib/annoFormatTab.c - lines changed 2, context: html, text, full: html, text
- Use annoStreamDb with config to add kgXref.geneSymbol to knownGene columns instead of annoStreamDbKnownGene.
- src/hg/lib/annoFormatVep.c - lines changed 2, context: html, text, full: html, text
- src/hg/lib/annoStreamDbKnownGene.c - lines changed 170, context: html, text, full: html, text
switch to files view, user index