Feature 2339 enhance data comparison workflow#2341
Open
jh-RLI wants to merge 27 commits into
Open
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…://github.com/OpenEnergyPlatform/oeplatform into feature-2339-enhance-data-comparison-workflow
…s disk ontop loads mapping.obda only at startup, so mapping edits and branch switches silently leave a running container serving the old mapping (newly mapped tables return 0 rows). The script restarts the container, waits for the SPARQL endpoint, and cross-checks every table mapped on disk against the endpoint via the table-name predicate; --check verifies without restarting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Registry: three AMIRIS facet dimensions (data_role, transaction_role, substance) on the generic is-about predicate with disjoint enums (amiris_facets value space); per-substance aggregation hint (sum|mean); raw time_step dimension (xsd:dateTime, detected via isAbout -> OEO_00140043). Loader passes aggregation through to the contract and serves the generator's unmapped_columns.json report beside it. Generator: gen_test_mapping.py gains a metadata-driven AMIRIS mode (`amiris` argv) — reads each table's served oemetadata, classifies isAbout concepts through the registry enums, detects the unit shape (field-level unit -> constant triple) and the time column, derives scenario_year from the DECLARED temporal block (never EXTRACT(YEAR): the settlement fencepost row would misbook into 2020), and reports everything unmappable instead of guessing. Wide tables emit one data point per (row, measure column) so multi-substance rows stay double-count-safe. Mapping: 64 generated blocks for the 11 amiris_germany2019_* tables appended to mapping.obda. Verified via SPARQL after ontop reload: dimension ASKs true, scenario_year = 2019 only, no fuel triples (reported unmapped instead), mean 2019 day-ahead price 37.98 EUR/MWh, merged multi-source query returns AMIRIS beside ariadne. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ty contract Registry (beta) tab now hosts three structurally different multi-source selection variants behind ?variant=A|B|C (0 = previous single-source view), dev builds only, floating switcher, shared state across variants: A gallery (browse-first cards) - B workbench rail (comparison-first) - C sentence builder (guided, verdict-as-narrative) Shared semantics per the wayfinder decisions: merged result set with source as first-class dimension and default group-by (WF-02, incl. the no-stacking- across-sources guardrail), unmapped_columns badges + chart footnotes (WF-05), granularity ladder with declared-scenario_year yearly buckets and registry aggregation hints (WF-06, BIND-based GROUP BY + xsd:double cast per WF-08), live comparability verdict (WF-12). comparability.js is NOT throwaway: pure WF-12 verdict function (merge | aggregate_first | blocked(reason), fixed reason vocabulary, selection corollary) with the WF-13 convertible-unit seam (inert until the registry serves units:). Everything under prototype_multisource/ is throwaway; see PROTOTYPE-NOTES.md for run instructions, live demo selections and the assumptions awaiting maintainer reaction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Maintainer reaction to assumption 1: the eu_leg metadata DOES declare the measure (value column isAbout OEO_00140082 greenhouse gas emission value, species per row via gas valueReferences) - the gap is that the hand-written eu_leg mapping never emits the is-about triple and the substance enum lacks the concept. Charted as wayfinder ticket WF-21; the prototype needs no code change once fixed (measures are discovered from the VKG). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Variant B (workbench rail) is the direction; C kept for other use cases;
variant 0 to be re-homed near the data view (wayfinder WF-22).
- Full width: the Registry tab drops the lg2 container; the rail grows to
viewport height.
- Comparability-aware selection window: every rail row gets a
contract-computed dot for the chosen measure (merges / aggregate first /
would block with WF-12 reason / no data) - same pure verdict function,
evaluated as "what if this table joined the selection"; legend at the
rail bottom.
- Measure-first flow: the measure select is catalog-wide ("start here")
with per-option provider counts; group-by options flag dimensions not
present in every selected source instead of silently dropping sources.
- Stale-chart UX: the hook remembers the parameters of the last run; any
change dims the chart under a "Parameters changed - Update chart"
overlay (charts keep rendering with the granularity they were run with).
Scenario-first selection stays a seam (needs the WF-14 bundle-link
harvest); comparability-beyond-unit+substance recorded as map fog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The measure select moves out of the toolbar into its own bar above the
graph filters - it is the initially required selection. The bar shows
per-scenario provider info ("Provided by 3 sources: AMIRIS Germany 2019
(2) - Ariadne (1) - 1 of 2 selected provide it") and a "Select all N
providers" button wired to the new selectProviders action.
The ontology-hierarchy measure grouping idea from the same reaction
round (OEO groups per-species emission rates under greenhouse gas
emission rate OEO_00140082) is charted as wayfinder WF-23, not built:
discovery grouping is the safe near-term use; a verdict leg needs
double-counting and CO2e-weighting guardrails first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Empty results no longer render a blank graph: a "No data for this query" panel names the exact slice (measure, unit, granularity, source count) and suggests what to change. All three variants. - Every rail row in variant B gets a peek icon: a dialog with the first 8 raw rows of the table plus an "Open table page" link to /dataedit/view/model_draft/<table> - raw-data transparency without leaving the composition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Legend never shows raw OEO ids: IRI-valued group values resolve through the TIB Terminology Service (tibTerms.js, the single-table view's resolution and cache) - TIB label first, registry enum label second, shortened IRI last. Legend hover shows the ontology definition; a term-chip row under the chart links each series to its ontology term. - echarts dataZoom (inside wheel/drag, range slider, toolbox zoom/restore) activates when the x-axis exceeds 31 buckets - makes hourly/daily series over long periods navigable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of the discussion
Describe the findings of the discussion in the issue or meeting.
Type of change (CHANGELOG.md)
Features
(#)
Changes
(#)
Bugs
(#)
Removed
(#)
Documentation updates
(#)
Workflow checklist
Automation
Closes #
PR-Assignee
CONTRIBUTING.md
CHANGELOG.md
mkdocs
Reviewer
Reviewer Guidelines