Wire dnaseq mergeExperiments: config, cache, variation load, webservices publish#93
Open
jbrestel wants to merge 14 commits into
Open
Wire dnaseq mergeExperiments: config, cache, variation load, webservices publish#93jbrestel wants to merge 14 commits into
jbrestel wants to merge 14 commits into
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…y from experiment configs Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ead experimentConfigGlob and absolute cacheFile Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ng plugin Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dirs to dnaseq webservices Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…atch copy Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t-db and variation load, webservices publish Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g line) Not needed in this workflow; the nextflow-side processSeqVars will no longer require the undone_strains_file input. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Merge processes are singletons over collect()'d inputs, so maxForks only gates the few independent early branches. Serialize to keep peak memory flat (snpEff JVM, Julia processSeqVars). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sions Drop the experiment-name prefix from staged filenames so mergeCoverageBeds column names match the bare VCF sample names that processSequenceVariations.jl looks up (the prefix only ever reached data via coverage, silently breaking coverage lookups). Sample names must be unique across experiments; enumerate all streams first and error with the offending sample/experiments before creating any symlinks, instead of letting ln -sf silently clobber a collision. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The merge config's reference_strain was blank: it queried ref_strain_abbrev, which is not set for reference-strain organisms (the only ones dnaseq runs for). Use the organism's own strain_abbrev via the shared OrganismInfo object instead, replacing the bespoke DbiDatabase connection. Plumb gusConfigFile through the merge subgraph so getOrganismInfo can be used. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The generated config globbed coverage/*.coverage.bed.gz (literal dot), but gathered files are named <sample>_coverage.bed.gz (underscore), so the glob matched nothing and the merge ran with no coverage data. Correct the separator to _coverage.bed.gz. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace RunNextflowWithEntry with RunNextflowLocal in dnaSeqMergeExperiments.xml (analysisDir->workingDir, config path relative to workflowDataDir, drop gitBranch). Remove the now-unused RunNextflowWithEntry.pm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the
mergeExperimentsnextflow entry into the dnaseq ReFlow workflow and loads/publishes its output.MakeDnaSeqMergeExperimentsNextflowConfig.pm): drops 6 params the currentdnaseq-nextflowmergeExperimentsentry no longer consumes (cacheFileDir,varscanDirectory,varscanFilePath,webServicesDir,extDbRlsSpec,organism_abbrev), makescacheFilean absolute path, and adds the now-requiredploidy— derived by reading the per-experiment nextflow configs and validating they agree (dies on mismatch). Ploidy-from-configs is a documented workaround; it belongs onapidb.organismlong-term.dnaseq.xmltouches an empty cache file before the merge (survives re-runs; lives outside the ephemeral_mergeExperimentsdir), passes it as--cache_file, and copies the fresh output back afterward.InsertVariationFeaturesWorkflowStep wrappingApiCommonData::Load::Plugin::InsertVariationFeatures, plus aCreateExtDbAndDbRlsstep creating<org>_dnaSeqVariations|do_not_care.CopyDnaSeqMergeResultsToWebServicesstep publishesmerged.ann.vcf.gz(+.tbi) →<org>/dnaseq/vcf/andhsss_readFreqN→<org>/dnaseq/readFreqN(prefix stripped), into the samednaseq/base bigwigs use.dnaSeqMergeExperiments.xml) updated to match; touches theundoneStrainsinput the run requires.Design/plan docs under
docs/superpowers/.Test Plan
Static verification (done):
perl -con all step classes; ploidy helper harness (consistent → value, mismatch → abort, empty/missing → die);xmllinton both XMLs; merge-output test-data artifacts present for every consumer; cross-file integration review.Integration (requires cluster + DB, not yet run):
dnaseqfor an organism with ≥2 experiments; confirm merge output, cache populated afterward,InsertVariationFeaturesloads<org>_dnaSeqVariations, and<org>/dnaseq/{vcf,readFreqN}populated.mergeCoverageBeds_coverage.bed.gzvs.coverage.bed.gznaming observation to the dnaseq-nextflow repo.🤖 Generated with Claude Code