Load mergeExperiments variation .dat files into ApiDB variation tables

[jbrestel]

Summary

Adds a GUS plugin that loads the mergeExperiments (dnaseq-nextflow) variation outputs — variationFeature.dat, transcript_product.dat, snpeff.dat — into ApiDB.VariationFeature, VariationTranscriptProduct, and VariationEffect.

• ApiCommonData::Load::VariationLoader — a pure module (no GUS, no DBI) defining the canonical column orders and the three file transforms. Unit-testable without a database.
• ApiCommonData::Load::Plugin::InsertVariationFeatures — resolves the external database release, builds an organism-scoped transcript source_id → na_feature_id map, validates that every sequence id belongs to the organism's genome, transforms each file to a temp file, and loads all three in a single psql transaction (BEGIN; DELETE; \copy ×3; COMMIT; with ON_ERROR_STOP=1).

Key design points

• One transaction across a parent table and two FK-children (ON DELETE CASCADE). A failure in any \copy rolls back the whole load, so a failed reload leaves the prior data intact rather than a half-populated set. Verified.
• source_id = Variant_<sequence_source_id>_<location> — a stable, opaque identifier. A constant prefix, deliberately not variant_type (which is strain-set-derived and would churn the id across re-merges).
• Undo via undoPreprocess, not undoTables — these tables carry no row_alg_invocation_id, so undo recovers extDbRlsSpec from core.AlgorithmParam and deletes by external_database_release_id (children cascade).
• Intergenic snpeff rows (empty transcript_id, ~60% of rows) load a NULL na_feature_id; a non-empty but unresolvable transcript id is fatal.

Companion change (separate repo)

Requires ApidbSchema branch dnaseq-tables: adds VariationFeature.source_id VARCHAR(100) NOT NULL UNIQUE and drops VariationTranscriptProduct.downstream_of_frameshift_strain_ids. That branch needs its own PR/merge.

Test Plan

• Unit: Load/t/variationLoader.t — 34 assertions, no DB (transforms, header parsing, source_id, fatal-vs-NULL semantics, and full-row column-mapping assertions guarding the column-list↔transform lockstep).
• Unit: Load/t/insertVariationFeatures_unit.t — 4 assertions, no DB (single-line \copy output, schema-identifier guard).
• Integration: Load/t/insertVariationFeatures_integration.t — 10 assertions against a live Postgres stub schema; loads the full sample dataset (1504 / 781 / 1978 rows), verifies the intergenic NULL split (797 non-null), and the cascade-delete undo path.
• Not automated: the full ga plugin run needs a workflow context and an organism present in the target DB (the test DB has no L. major). Documented in the design spec for whoever has such an instance.

Design spec and implementation plan are in docs/superpowers/specs/ and docs/superpowers/plans/.

🤖 Generated with Claude Code