Commits for kent

v282_base to v283_preview (2013-04-16 to 2013-04-23) v283

• Adding twoBitTotalSizeNoN.
  • src/inc/twoBit.h - lines changed 3, context: html, text, full: html, text
  • src/lib/twoBit.c - lines changed 13, context: html, text, full: html, text
• Adding new edwQaPairCorrelate table, and including non-N base count in edwAssembly.
  • src/hg/encode3/encodeDataWarehouse/inc/encodeDataWarehouse.h - lines changed 86, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.as - lines changed 15, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.c - lines changed 204, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.sql - lines changed 17, context: html, text, full: html, text
• Including non-N base count in edwAssembly.
  • src/hg/encode3/encodeDataWarehouse/edwAddAssembly/edwAddAssembly.c - lines changed 5, context: html, text, full: html, text
• Starting to implement replicate analysis.
  • src/hg/encode3/encodeDataWarehouse/edwMakeEnrichments/edwMakeEnrichments.c - lines changed 40, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/inc/edwLib.h - lines changed 13, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/edwLib.c - lines changed 45, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/redoSchema - lines changed 2, context: html, text, full: html, text
• Starting to implement replicate analysis.
  • src/hg/encode3/encodeDataWarehouse/edwMakeReplicateQa/edwMakeReplicateQa.c - lines changed 200, context: html, text, full: html, text
• Separating edwParseSubmitFile into it's own function, and making it check each field so as to give error messages earlier before time consuming uploads.
  • src/hg/encode3/encodeDataWarehouse/edwSubmit/edwSubmit.c - lines changed 137, context: html, text, full: html, text
• Adding a view shortcut for pairCorrelate
  • src/hg/encode3/encodeDataWarehouse/lib/resetEdw - lines changed 1, context: html, text, full: html, text
• Removing some debugging statements.
  • src/hg/encode3/encodeDataWarehouse/lib/edwLib.c - lines changed 1, context: html, text, full: html, text
• Removing some debugging statments and adding more bigBed variants.
  • src/hg/encode3/encodeDataWarehouse/edwMakeValidFile/edwMakeValidFile.c - lines changed 6, context: html, text, full: html, text
• Adding bigWig support.
  • src/hg/encode3/encodeDataWarehouse/edwMakeReplicateQa/edwMakeReplicateQa.c - lines changed 111, context: html, text, full: html, text
• Adding bigBedVariants.
  • src/hg/encode3/encodeDataWarehouse/edwMakeReplicateQa/edwMakeReplicateQa.c - lines changed 5, context: html, text, full: html, text
• Centralizing bigBed variant format detection.
  • src/hg/encode3/encodeDataWarehouse/edwMakeEnrichments/edwMakeEnrichments.c - lines changed 5, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/edwMakeReplicateQa/edwMakeReplicateQa.c - lines changed 5, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/edwMakeValidFile/edwMakeValidFile.c - lines changed 5, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/inc/edwLib.h - lines changed 3, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/edwLib.c - lines changed 9, context: html, text, full: html, text
• Adding more enrichment targets.
  • src/hg/encode3/encodeDataWarehouse/lib/resetEdw - lines changed 10, context: html, text, full: html, text
• Removing warning.
  • src/lib/gff.c - lines changed 5, context: html, text, full: html, text
• Adding more supported enrichments. Making it actually try to load files again after parsing submission.
  • src/hg/encode3/encodeDataWarehouse/edwSubmit/edwSubmit.c - lines changed 6, context: html, text, full: html, text
• Finishing a place Fan missed in implementing -withNull
  • src/hg/autoSql/autoSql.c - lines changed 11, context: html, text, full: html, text
• Adding a separate parameter for file name to report errors on for benefit of edwSubmit which want's to report pre-upload URL where error occurs.
  • src/inc/fieldedTable.h - lines changed 4, context: html, text, full: html, text
  • src/lib/fieldedTable.c - lines changed 27, context: html, text, full: html, text
• Adding shuffleArrayOfChars library function.
  • src/inc/obscure.h - lines changed 3, context: html, text, full: html, text
  • src/lib/obscure.c - lines changed 16, context: html, text, full: html, text
• Splitting edwQaPairCorrelate into two tables.
  • src/hg/encode3/encodeDataWarehouse/inc/encodeDataWarehouse.h - lines changed 121, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.as - lines changed 13, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.c - lines changed 227, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.sql - lines changed 14, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/resetEdw - lines changed 1, context: html, text, full: html, text
• Adding edwWriteErrToStderrAndTable library function.
  • src/hg/encode3/encodeDataWarehouse/inc/edwLib.h - lines changed 3, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/edwLib.c - lines changed 14, context: html, text, full: html, text
• Improving error reporting for missing required fields in submission file.
  • src/hg/encode3/encodeDataWarehouse/edwSubmit/edwSubmit.c - lines changed 15, context: html, text, full: html, text
• Files that fail validation get reason why filled in in edwFile.errorMessage. Improving status output a little.
  • src/hg/encode3/encodeDataWarehouse/edwMakeValidFile/edwMakeValidFile.c - lines changed 105, context: html, text, full: html, text
• Splitting qaPairCorrelate into two tables - one for sample overlap, one for correlation. Adding several new correlation fields.
  • src/hg/encode3/encodeDataWarehouse/edwMakeReplicateQa/edwMakeReplicateQa.c - lines changed 151, context: html, text, full: html, text
• Fixed a source of rounding error, measured timing of parts because it's slow, and implemented a ~2x faster algorithm.
  • src/hg/encode3/encodeDataWarehouse/edwMakeEnrichments/edwMakeEnrichments.c - lines changed 97, context: html, text, full: html, text
• Adding defaultZeros flag - this seems to be needed with strictor MySQL configs when inserting partial records.
  • src/hg/autoSql/autoSql.c - lines changed 22, context: html, text, full: html, text
• Adding -defaultZeros to make it work in San Diego.
  • src/hg/encode3/encodeDataWarehouse/lib/redoSchema - lines changed 2, context: html, text, full: html, text
• Adding -defaultZeros to make it work in San Diego.
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.sql - lines changed 119, context: html, text, full: html, text
• Moving to San Diego paths.
  • src/hg/encode3/encodeDataWarehouse/lib/edwLib.c - lines changed 3, context: html, text, full: html, text
• Improving status output.
  • src/hg/encode3/encodeDataWarehouse/edwMakeEnrichments/edwMakeEnrichments.c - lines changed 3, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/edwMakeValidFile/edwMakeValidFile.c - lines changed 1, context: html, text, full: html, text
• Improving status output.
  • src/hg/encode3/encodeDataWarehouse/edwMakeReplicateQa/edwMakeReplicateQa.c - lines changed 4, context: html, text, full: html, text
• First cut of qa job scheduler.
  • src/hg/encode3/encodeDataWarehouse/edwRunner/edwRunner.c - lines changed 211, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/edwRunner/makefile - lines changed 4, context: html, text, full: html, text
• Making it have a few less points where it's waiting when it could be doing something.
  • src/hg/encode3/encodeDataWarehouse/edwRunner/edwRunner.c - lines changed 46, context: html, text, full: html, text
• Improving file list for tag maker.
  • src/makefile - lines changed 1, context: html, text, full: html, text
• Adding edwJob and removing edwQaAgent and edwQaRun to manage jobs.
  • src/hg/encode3/encodeDataWarehouse/inc/encodeDataWarehouse.h - lines changed 240, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.as - lines changed 33, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.c - lines changed 575, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.sql - lines changed 36, context: html, text, full: html, text
• Adding views for correatios and crossEnrichments. Removing edwQaAgent stuff.
  • src/hg/encode3/encodeDataWarehouse/lib/resetEdw - lines changed 8, context: html, text, full: html, text
• Adding function to fetch user record given emailaddress and functions to add jobs to be run.
  • src/hg/encode3/encodeDataWarehouse/inc/edwLib.h - lines changed 12, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/edwLib.c - lines changed 38, context: html, text, full: html, text
• Making it keep track of last ID to save a little database time.
  • src/hg/encode3/encodeDataWarehouse/edwRunner/edwRunner.c - lines changed 8, context: html, text, full: html, text
• Removing little edwSubmit from resetEdw script.
  • src/hg/encode3/encodeDataWarehouse/lib/resetEdw - lines changed 1, context: html, text, full: html, text
• Making command line for edwSubmit use email address rather than username + password. This is just a temporary thing while working out edwWebSubmit.
  • src/hg/encode3/encodeDataWarehouse/edwSubmit/edwSubmit.c - lines changed 35, context: html, text, full: html, text
• Making assembly and enrichment target adders put their files on QA job list.
  • src/hg/encode3/encodeDataWarehouse/edwAddAssembly/edwAddAssembly.c - lines changed 2, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/edwAddQaEnrichTarget/edwAddQaEnrichTarget.c - lines changed 2, context: html, text, full: html, text
• Removing some debugging code and turning some into logging code.
  • src/hg/encode3/encodeDataWarehouse/edwMakeValidFile/edwMakeValidFile.c - lines changed 2, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/edwMakeValidFile/sorted.bb - lines changed 0, context: html, text, full: html, text (a binary file or whitespace-only change shows no diff)
  • src/hg/encode3/encodeDataWarehouse/edwMakeValidFile/sorted.bed - lines changed 331449, context: html, text, full: html, text
• Improving comments and turning some debugging uglyfs into verbose-2's.
  • src/hg/encode3/encodeDataWarehouse/edwRunner/edwRunner.c - lines changed 5, context: html, text, full: html, text
• Exporting edwSingleFileTimeout.
  • src/hg/encode3/encodeDataWarehouse/inc/edwLib.h - lines changed 1, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/edwLib.c - lines changed 3, context: html, text, full: html, text
• Checking for empty input and not crashing!
  • src/hg/encode3/encodeDataWarehouse/edwMakeEnrichments/edwMakeEnrichments.c - lines changed 5, context: html, text, full: html, text
• Adapting this to be run initiated from web. Using a database profile so can have read/write access to mysql encodeDataWarehouse. Making it so will attempt not to be run twice on the same URL at once.
  • src/hg/encode3/encodeDataWarehouse/edwSubmit/edwSubmit.c - lines changed 65, context: html, text, full: html, text
• Fixed type of next pointer in cgiParsedVars. Ref #10621
  • src/inc/cheapcgi.h - lines changed 1, context: html, text, full: html, text
• Cleaning out old cruft from freen dir. Making freen test named pipes.
  • src/hg/oneShot/freen/blatUp.out - lines changed 37, context: html, text, full: html, text
  • src/hg/oneShot/freen/freen.c - lines changed 33, context: html, text, full: html, text
  • src/hg/oneShot/freen/freen.doc - lines changed 40, context: html, text, full: html, text
  • src/hg/oneShot/freen/hg7.out - lines changed 224, context: html, text, full: html, text
• Removing a temp file when it's no longer needed.
  • src/hg/encode3/encodeDataWarehouse/edwRunner/edwRunner.c - lines changed 1, context: html, text, full: html, text
• Adding some fields to edwSubmit to make it so web app can report progress better.
  • src/hg/encode3/encodeDataWarehouse/inc/encodeDataWarehouse.h - lines changed 5, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.as - lines changed 21, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.c - lines changed 23, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.sql - lines changed 4, context: html, text, full: html, text
• Forcing remake of all edw source on schema change.
  • src/hg/encode3/encodeDataWarehouse/lib/redoSchema - lines changed 4, context: html, text, full: html, text
• Making resetEdw create fifo for submission spooler if it doesn't exist.
  • src/hg/encode3/encodeDataWarehouse/lib/resetEdw - lines changed 7, context: html, text, full: html, text
• Moving edwMostRecentSubmission to library.
  • src/hg/encode3/encodeDataWarehouse/inc/edwLib.h - lines changed 3, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/edwLib.c - lines changed 19, context: html, text, full: html, text
• A number of changes to put more status info on submission in progress into database so web app can report on it.
  • src/hg/encode3/encodeDataWarehouse/edwSubmit/edwSubmit.c - lines changed 85, context: html, text, full: html, text
• First cut of a web submission ap. It communicates with command line edwSubmit program via a named pipe managed by edwSubmitSpooler, which also manages multiple edwSubmit processes so as to allow more than one, but not too many.
  • src/hg/encode3/encodeDataWarehouse/edwSubmitSpooler/edwSubmitSpooler.c - lines changed 98, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/edwSubmitSpooler/makefile - lines changed 3, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/edwWebSubmit/edwWebSubmit.c - lines changed 217, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/edwWebSubmit/makefile - lines changed 16, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/makefile - lines changed 3, context: html, text, full: html, text
• Making Eurie a user.
  • src/hg/encode3/encodeDataWarehouse/lib/resetEdw - lines changed 1, context: html, text, full: html, text
• Making edwWriteErrToTable - similar to an existing function but doesn't also write to stdout.
  • src/hg/encode3/encodeDataWarehouse/inc/edwLib.h - lines changed 3, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/edwLib.c - lines changed 10, context: html, text, full: html, text
• Reporting file level errors in the submission as well as file tables.
  • src/hg/encode3/encodeDataWarehouse/edwSubmit/edwSubmit.c - lines changed 18, context: html, text, full: html, text
• Improving error messages and removing quirky javascript (doc.write seems to be very frowned upon) in header. Improving status display during and after upload.
  • src/hg/encode3/encodeDataWarehouse/edwWebSubmit/edwWebSubmit.c - lines changed 38, context: html, text, full: html, text
• Adding index and auto keywords to autoSql columns.
  • src/hg/autoSql/autoSql.c - lines changed 41, context: html, text, full: html, text
  • src/inc/asParse.h - lines changed 10, context: html, text, full: html, text
  • src/lib/asParse.c - lines changed 43, context: html, text, full: html, text
• Adding docs and test cases for index and autoincrement feature.
  • src/hg/autoSql/autoSql.doc - lines changed 34, context: html, text, full: html, text
  • src/hg/autoSql/doc.as - lines changed 5, context: html, text, full: html, text
  • src/hg/autoSql/tests/expected/index.sql - lines changed 30, context: html, text, full: html, text
  • src/hg/autoSql/tests/input/index.as - lines changed 18, context: html, text, full: html, text
  • src/hg/autoSql/tests/makefile - lines changed 5, context: html, text, full: html, text
• Added -addBin option at Angie's request.
  • src/hg/autoSql/autoSql.c - lines changed 35, context: html, text, full: html, text
  • src/hg/autoSql/tests/expected/binTest.c - lines changed 388, context: html, text, full: html, text
  • src/hg/autoSql/tests/expected/binTest.h - lines changed 173, context: html, text, full: html, text
  • src/hg/autoSql/tests/expected/binTest.sql - lines changed 37, context: html, text, full: html, text
  • src/hg/autoSql/tests/input/binTest.as - lines changed 30, context: html, text, full: html, text
  • src/hg/autoSql/tests/makefile - lines changed 7, context: html, text, full: html, text
  • src/inc/asParse.h - lines changed 4, context: html, text, full: html, text
  • src/lib/asParse.c - lines changed 16, context: html, text, full: html, text
• Correcting English a little.
  • src/hg/autoSql/autoSql.doc - lines changed 1, context: html, text, full: html, text
• Polishing upload tracking part of page, and leaving validations for later.
  • src/hg/encode3/encodeDataWarehouse/edwWebSubmit/edwWebSubmit.c - lines changed 129, context: html, text, full: html, text
• A little more polish
  • src/hg/encode3/encodeDataWarehouse/edwWebSubmit/edwWebSubmit.c - lines changed 7, context: html, text, full: html, text
• Removed a few lines not needed since improvements in autoSql.
  • src/hg/encode3/encodeDataWarehouse/lib/redoSchema - lines changed 2, context: html, text, full: html, text
• Adding indexes.
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.as - lines changed 36, context: html, text, full: html, text
  • src/hg/encode3/encodeDataWarehouse/lib/encodeDataWarehouse.sql - lines changed 33, context: html, text, full: html, text
• Bumped default process count to 5.
  • src/hg/encode3/encodeDataWarehouse/edwSubmitSpooler/edwSubmitSpooler.c - lines changed 1, context: html, text, full: html, text
• Changing file permissions to be world readable and group writable. Improving an error message.
  • src/hg/encode3/encodeDataWarehouse/edwSubmit/edwSubmit.c - lines changed 5, context: html, text, full: html, text