TRT-2734: add release_definitions table and loader by mstaeble · Pull Request #3679 · openshift/sippy

mstaeble · 2026-06-24T14:40:07Z

Summary

Creates a release_definitions table in PostgreSQL and a loader to populate it from BigQuery. This is the first phase of moving release metadata from BigQuery to PostgreSQL (TRT-2734).

No consumers are switched in this PR. The existing getReleases() and QueryReleases() paths continue to use BigQuery / hardcoded metadata. A follow-up PR (#3736) will switch consumers to read from the new table once it has been populated by the load cycle.

Adds ReleaseDefinition model with capability constants and HasCapability method
Adds release-definitions loader (--loader release-definitions) that fetches from BQ and syncs to PG via upsert
Adds GetReleasesFromDB and DefinitionToRelease for reading from PG
Adds GetReleaseRowsFromBigQuery for direct BQ row access
Load command reads release configs from PG with BQ fallback when the table is empty (first run)
Seeds release_definitions for local development
7 files changed, +276/-8

Test plan

go build ./... passes
go vet ./... passes
make lint passes
go test ./pkg/... ./cmd/... passes
sippy serve with seed data works correctly (no behavior change)

Staging verification

Deployed and ran the loader as a one-off job:

sippy load --loader release-definitions --init-database

Synced 36 release definitions from BigQuery to staging PostgreSQL (all OCP releases 3.11 through 5.0, OKD, ROSA, ARO, HyperShift, and CAPI entries).

Ref: TRT-2734

@coderabbitai ignore

🤖 Generated with Claude Code

openshift-merge-bot · 2026-06-24T14:40:11Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

openshift-ci · 2026-06-24T14:40:13Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-merge-bot · 2026-06-24T17:01:50Z

Scheduling required tests:
/test e2e

sosiouxme · 2026-06-24T19:48:50Z

-func (s *Server) getReleases(ctx context.Context, forceRefresh ...bool) ([]sippyv1.Release, error) {
-	if s.bigQueryClient != nil {
-		refresh := len(forceRefresh) > 0 && forceRefresh[0]
-		return api.GetReleases(ctx, s.bigQueryClient, refresh)
+// getReleases returns release data from PostgreSQL.
+func (s *Server) getReleases(ctx context.Context) ([]sippyv1.Release, error) {
+	if s.db != nil {
+		releases, err := api.GetReleasesFromDB(ctx, s.db)


this removes the cache. reading from a small postgres table is fast, but not nearly as fast as reading from redis, and this page gets called.... a lot. with React calling it several times per UI transition it may even be noticeable at a human timescale. consider whether we don't want caching here?

Good call raising this. I traced all the callers to understand the full impact.

getReleases() is called by 8 HTTP handlers:

/api/releases (page load, ~15 calls/hour)

/api/component_readiness (every CR request)

/api/component_readiness/test_details

/api/component_readiness/views

4 triage/regression endpoints

The old Redis cache (8h TTL, key "Releases~") was valuable because it avoided a BigQuery round-trip on every one of these requests. That saved both latency and BQ query cost.

With PostgreSQL, the query is ~7ms for ~20 rows. The CR endpoints themselves take seconds, so 7ms is noise. There's no per-query cost like BQ.

More importantly, these callers are transitional. The CR handlers use release data for two things: resolving relative time strings like "ga-30d" into absolute dates, and generating HATEOAS links with correct time parameters. As we move CR fully to PostgreSQL:

The CR test status queries will use pre-aggregated matviews per release (e.g., cr_base_agg_4.21), where the GA-based time window is baked into the matview at build time. No release metadata lookup needed at request time.

For any queries that still need release dates, the lookup can fold into the main query as a JOIN against release_definitions rather than a separate round-trip.

That leaves /api/releases as the only endpoint that needs a standalone query, at ~15 calls/hour. Adding a Redis cache layer for that adds complexity with no user-visible benefit. The HTTP round-trip overhead alone (~50-200ms) dwarfs the 7ms PG query.

If we see this become a bottleneck in practice, we can add caching for PQ later, but I don't think it's warranted now.

openshift-merge-bot · 2026-06-27T01:28:41Z

Scheduling required tests:
/test e2e

openshift-ci-robot · 2026-06-30T21:51:21Z

@mstaeble: This pull request references TRT-2734 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Creates a release_definitions table in PostgreSQL to store release metadata (GA dates, development start dates, previous release, capabilities, product, status) previously only available in BigQuery

Adds a release-definitions loader (--loader release-definitions) that fetches release rows from BQ and syncs them to PostgreSQL during the data load cycle

/api/releases and the PG data provider now read from the release_definitions table instead of BigQuery or the hardcoded releaseMetadata map

getReleases() in the server prefers PG, falls back to BQ

Seeds release definitions for the dev database

9 files changed, +297/-80

A follow-up PR will replace v1.Release with models.ReleaseDefinition across all internal consumers, remove QueryReleases/QueryReleaseDates from the DataProvider interface, and eliminate the remaining BQ release functions.

Test plan

go build ./... passes

go vet ./... passes

make lint passes

go test ./pkg/... ./cmd/... passes

sippy serve with seed data serves /api/releases from PostgreSQL with correct GA dates, capabilities, and previous release chain

Staging verification

Deployed the branch to sippy-staging and ran the release-definitions loader as a one-off job:
sippy load --loader release-definitions --init-database
Synced 36 release definitions from BigQuery to staging PostgreSQL (all OCP releases 3.11 through 5.0, OKD, ROSA, ARO, HyperShift, and CAPI entries).

Verified the following endpoints on sippy-staging.dptools.openshift.org:

Endpoint Result

GET /api/releases 36 releases with correct GA dates, capabilities, previous release chain, product, and status

GET /api/releases/health?release=4.22 6 payload health reports

GET /api/component_readiness (4.21→4.22 with full params) 72 components, 38 columns

GET /api/component_readiness/regressions?release=4.22 4,545 regressions

All release data served from PostgreSQL with no BigQuery calls.

Ref: TRT-2734

@coderabbitai ignore

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-06-30T21:51:37Z

Tip

For best results, initiate chat on the files or code changes.

This comment is an automated notice from openshift-ci-robot about a Jira target-version validation issue (TRT-2734 missing the "5.0.0" target version) and isn't a request directed at me. Additionally, the PR description includes @coderabbitai ignore, so I won't be taking any review actions on this PR. The Jira target version issue should be resolved directly in Jira or via the jira-lifecycle-plugin bot commands.

Cherry-pick from PR openshift#3716 with conflict resolution: - Use civil.Date for date-only fields (GA dates, development start dates) - Use TIMESTAMP WITH TIME ZONE and DATE types in PostgreSQL - RFC 3339 for API timestamps, YYYY-MM-DD for dates - Resolve conflict in postgres provider (keep GetReleasesFromDB from PR openshift#3679) - Add time.Time to civil.Date conversion in DefinitionToRelease Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

neisw · 2026-07-06T18:47:37Z

-		releases = append(releases, rel)
-	}
-	return releases, nil
+	return api.GetReleasesFromDB(ctx, p.dbc)


Do we have a race condition when this merges prior to load running? Should this be a two-stage process where we init / load the new table then switch over, or is that happening externally?

Good point. I've split this into two PRs to avoid the race condition:

TRT-2734: add release_definitions table and loader #3679 (this PR): Creates the release_definitions table, adds the release-definitions loader, and seeds data for local dev. No consumers are switched — the existing getReleases() and QueryReleases() paths continue to use BigQuery / hardcoded metadata. This is purely additive and safe to merge immediately.

[WIP] TRT-2734: switch release consumers from BigQuery to PostgreSQL #3736: Switches getReleases() and the PG provider to read from release_definitions instead of BigQuery / the hardcoded map. This should only merge after TRT-2734: add release_definitions table and loader #3679 has been deployed and at least one load cycle has populated the table.

openshift-ci-robot · 2026-07-06T19:17:19Z

@mstaeble: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

Details

In response to this:

Summary

Creates a release_definitions table in PostgreSQL and a loader to populate it from BigQuery. This is the first phase of moving release metadata from BigQuery to PostgreSQL (TRT-2734).

No consumers are switched in this PR. The existing getReleases() and QueryReleases() paths continue to use BigQuery / hardcoded metadata. A follow-up PR (#3736) will switch consumers to read from the new table once it has been populated by the load cycle.

Adds ReleaseDefinition model with capability constants and HasCapability method

Adds release-definitions loader (--loader release-definitions) that fetches from BQ and syncs to PG via upsert

Adds GetReleasesFromDB and DefinitionToRelease for reading from PG

Adds GetReleaseRowsFromBigQuery for direct BQ row access

Seeds release_definitions for local development

7 files changed, +273/-8

Test plan

go build ./... passes

go vet ./... passes

make lint passes

go test ./pkg/... ./cmd/... passes

sippy serve with seed data works correctly (no behavior change)

Staging verification

Deployed and ran the loader as a one-off job:
sippy load --loader release-definitions --init-database
Synced 36 release definitions from BigQuery to staging PostgreSQL (all OCP releases 3.11 through 5.0, OKD, ROSA, ARO, HyperShift, and CAPI entries).

Ref: TRT-2734

@coderabbitai ignore

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-07-06T19:17:36Z

Tip

For best results, initiate chat on the files or code changes.

@openshift-ci-robot This is another automated notice (this time about the PR title missing a Jira issue reference in the XYZ-NNN: format) and isn't a request directed at me. As noted previously, this PR's description includes @coderabbitai ignore, so I won't be taking any review actions here. Resolving the Jira title reference requires @mstaeble to update the PR title and run /jira refresh.

openshift-merge-bot · 2026-07-06T19:29:03Z

Scheduling required tests:
/test e2e

openshift-ci-robot · 2026-07-07T13:24:56Z

@mstaeble: This pull request references TRT-2734 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Creates a release_definitions table in PostgreSQL and a loader to populate it from BigQuery. This is the first phase of moving release metadata from BigQuery to PostgreSQL (TRT-2734).

No consumers are switched in this PR. The existing getReleases() and QueryReleases() paths continue to use BigQuery / hardcoded metadata. A follow-up PR (#3736) will switch consumers to read from the new table once it has been populated by the load cycle.

Adds ReleaseDefinition model with capability constants and HasCapability method

Adds release-definitions loader (--loader release-definitions) that fetches from BQ and syncs to PG via upsert

Adds GetReleasesFromDB and DefinitionToRelease for reading from PG

Adds GetReleaseRowsFromBigQuery for direct BQ row access

Seeds release_definitions for local development

7 files changed, +273/-8

Test plan

go build ./... passes

go vet ./... passes

make lint passes

go test ./pkg/... ./cmd/... passes

sippy serve with seed data works correctly (no behavior change)

Staging verification

Deployed and ran the loader as a one-off job:
sippy load --loader release-definitions --init-database
Synced 36 release definitions from BigQuery to staging PostgreSQL (all OCP releases 3.11 through 5.0, OKD, ROSA, ARO, HyperShift, and CAPI entries).

Ref: TRT-2734

@coderabbitai ignore

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-07-07T13:25:11Z

Tip

For best results, initiate chat on the files or code changes.

@openshift-ci-robot This is the same automated Jira target-version notice as before. As previously noted, this PR's description includes @coderabbitai ignore, so I won't be taking any review actions here. Resolving the Jira target version warning requires action in Jira itself (or via jira-lifecycle-plugin bot commands), not from me.

neisw · 2026-07-07T14:01:49Z

+			// loader will populate it for subsequent runs.
+			releaseConfigs := []sippyv1.Release{}
+			if dbErr == nil {
+				releaseConfigs, _ = api.GetReleasesFromDB(context.Background(), dbc)


I think we still have an issue here. The releases are needed for partitioning and the loader. You could look to have GetReleasesFromDB fallback to GetReleasesFromBigQuery when results are empty or keep the BQ call here for now until we know it is populated.

Good point. I've added a BQ fallback in load.go. It reads from PG first, and falls back to GetReleasesFromBigQuery when the table is empty (e.g., first run on a new database). This ensures partitioning and downstream loaders work correctly even before the release-definitions loader has populated the table.

One last consideration, previously we returned if there was an error loading the releases. You are now logging the errors in the api calls but drop the errors here. I don't think we expect the loader to work without releases, at least not the job loader.

I don't want to overengineer this but should we validate we have at least 1 release and if not then exit?

Updated. Error handling now matches the original behavior: hard exit on query error (return errors.Wrapf(err, ...)), silent continue on empty results. It tries PG first, falls back to BQ if PG returns no rows (e.g., first run).

openshift-merge-bot · 2026-07-07T14:25:27Z

Scheduling required tests:
/test e2e

Create a release_definitions table to store release metadata (GA dates, development start dates, previous release, capabilities, product, status) synced from BigQuery. This is the first phase of moving release metadata from BigQuery to PostgreSQL. Key changes: - Add ReleaseDefinition model with capability constants and HasCapability method - Add release-definitions loader (--loader release-definitions) that fetches from BQ and syncs to PG via upsert - Add GetReleasesFromDB and DefinitionToRelease for reading from PG - Add GetReleaseRowsFromBigQuery for direct BQ row access - Seed data populates release_definitions for local development - Load command reads release configs from PG with BQ fallback for first run when the table is empty No consumers are switched in this phase. The existing getReleases() and QueryReleases() paths continue to use BigQuery / hardcoded metadata. A follow-up PR will switch consumers to read from the new table once it has been populated by the load cycle. Ref: TRT-2734 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

neisw · 2026-07-07T16:57:55Z

/lgtm

openshift-ci · 2026-07-07T16:58:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mstaeble, neisw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [neisw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-merge-bot · 2026-07-07T17:09:13Z

Scheduling required tests:
/test e2e

openshift-ci · 2026-07-07T18:05:03Z

@mstaeble: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Cherry-pick from PR openshift#3716 with conflict resolution: - Use civil.Date for date-only fields (GA dates, development start dates) - Use TIMESTAMP WITH TIME ZONE and DATE types in PostgreSQL - RFC 3339 for API timestamps, YYYY-MM-DD for dates - Resolve conflict in postgres provider (keep GetReleasesFromDB from PR openshift#3679) - Add time.Time to civil.Date conversion in DefinitionToRelease Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 24, 2026

mstaeble mentioned this pull request Jun 24, 2026

Move release metadata from BigQuery to PostgreSQL #3674

Closed

5 tasks

mstaeble changed the title ~~[WIP] Move release metadata from BigQuery to PostgreSQL (minimal)~~ Move release metadata from BigQuery to PostgreSQL Jun 24, 2026

mstaeble force-pushed the TRT-2734-releases-to-postgres-minimal branch from eccf470 to c3dca6d Compare June 24, 2026 15:36

mstaeble marked this pull request as ready for review June 24, 2026 15:45

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 24, 2026

openshift-ci Bot requested review from dgoodwin and petr-muller June 24, 2026 15:45

mstaeble force-pushed the TRT-2734-releases-to-postgres-minimal branch from c3dca6d to 23f283f Compare June 24, 2026 16:45

sosiouxme reviewed Jun 24, 2026

View reviewed changes

mstaeble force-pushed the TRT-2734-releases-to-postgres-minimal branch 2 times, most recently from 09c13a6 to 4592ca7 Compare June 27, 2026 00:48

mstaeble mentioned this pull request Jun 29, 2026

TRT-2364: Fix timestamp and date type inconsistencies #3716

Open

17 tasks

mstaeble changed the title ~~Move release metadata from BigQuery to PostgreSQL~~ TRT-2734: Move release metadata from BigQuery to PostgreSQL Jun 30, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 30, 2026

neisw reviewed Jul 6, 2026

View reviewed changes

mstaeble mentioned this pull request Jul 6, 2026

TRT-2768: Import /payload job results from PRs into postgres #3728

Open

mstaeble force-pushed the TRT-2734-releases-to-postgres-minimal branch 2 times, most recently from 9283507 to 39eb557 Compare July 6, 2026 19:16

mstaeble mentioned this pull request Jul 6, 2026

[WIP] TRT-2734: switch release consumers from BigQuery to PostgreSQL #3736

Draft

9 tasks

mstaeble changed the title ~~TRT-2734: Move release metadata from BigQuery to PostgreSQL~~ Add release_definitions table and loader Jul 6, 2026

openshift-ci-robot removed the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 6, 2026

mstaeble changed the title ~~Add release_definitions table and loader~~ TRT-2734: add release_definitions table and loader Jul 7, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 7, 2026

neisw reviewed Jul 7, 2026

View reviewed changes

mstaeble force-pushed the TRT-2734-releases-to-postgres-minimal branch from 39eb557 to 790c4d8 Compare July 7, 2026 14:10

mstaeble force-pushed the TRT-2734-releases-to-postgres-minimal branch from 790c4d8 to 94669d7 Compare July 7, 2026 16:40

mstaeble force-pushed the TRT-2734-releases-to-postgres-minimal branch from 94669d7 to 5d8e3c2 Compare July 7, 2026 16:48

openshift-ci Bot assigned neisw Jul 7, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 7, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 7, 2026

openshift-merge-bot Bot merged commit 05ba623 into openshift:main Jul 7, 2026
9 checks passed

Uh oh!

Conversation

mstaeble commented Jun 24, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Staging verification

Uh oh!

openshift-merge-bot Bot commented Jun 24, 2026

Uh oh!

openshift-ci Bot commented Jun 24, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 24, 2026

Uh oh!

sosiouxme Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

mstaeble Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

openshift-merge-bot Bot commented Jun 27, 2026

Uh oh!

openshift-ci-robot commented Jun 30, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Staging verification

Uh oh!

coderabbitai Bot commented Jun 30, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neisw Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

mstaeble Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

openshift-ci-robot commented Jul 6, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Staging verification

Uh oh!

coderabbitai Bot commented Jul 6, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 6, 2026

Uh oh!

openshift-ci-robot commented Jul 7, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Staging verification

Uh oh!

coderabbitai Bot commented Jul 7, 2026

Uh oh!

neisw Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mstaeble Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

neisw Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

mstaeble Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

openshift-merge-bot Bot commented Jul 7, 2026

Uh oh!

neisw commented Jul 7, 2026

Uh oh!

openshift-ci Bot commented Jul 7, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 7, 2026

Uh oh!

openshift-ci Bot commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

mstaeble commented Jun 24, 2026 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Jun 30, 2026 •

edited by openshift-ci Bot

Loading

coderabbitai Bot commented Jun 30, 2026 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Jul 6, 2026 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Jul 7, 2026 •

edited by openshift-ci Bot

Loading

neisw Jul 7, 2026 •

edited

Loading