[fm] Guard support bundle creation with SitrepGuardedInsert.#10535
[fm] Guard support bundle creation with SitrepGuardedInsert.#10535mergeconflict wants to merge 17 commits into
Conversation
e15c2c1 to
e5d67e2
Compare
1bc2f60 to
62cc3dd
Compare
e5d67e2 to
809d7d3
Compare
62cc3dd to
3d5af68
Compare
809d7d3 to
7128ff2
Compare
a1beef9 to
e97748d
Compare
7128ff2 to
1574845
Compare
e97748d to
f3832d4
Compare
1574845 to
4d0f1dd
Compare
f3832d4 to
cbc506b
Compare
85edf6b to
a3ab6c2
Compare
cbc506b to
59680b0
Compare
Add the `SitrepGuardedInsert` Diesel combinator and the
`SitrepGuardedResource` trait: a generic primitive for FM rendezvous to
insert a resource row idempotently and guarded against stale-sitrep
execution.
The combinator wraps a caller-supplied resource INSERT in a single CTE
statement that:
- aborts (StaleSitrep) unless the executor's expected generation still
equals the latest sitrep's generation column;
- short-circuits (AlreadyExists) if a creation marker already exists for
the resource id;
- on a successful insert, atomically writes a creation marker, gated by
`WHERE EXISTS (SELECT 1 FROM new_resource)` so a marker is never
fabricated for a pre-existing row.
All spliced SQL identifiers come from the trait's `&'static str` consts, so
the query is injection-safe. The result is surfaced as a
`SitrepGuardedInsertOutcome` of Created / AlreadyExists / StaleSitrep.
59680b0 to
b6f5bfe
Compare
a3ab6c2 to
26cbbe1
Compare
b6f5bfe to
a7d6f65
Compare
26cbbe1 to
da05cf5
Compare
a7d6f65 to
aee00ec
Compare
da05cf5 to
064090e
Compare
Add the `SitrepGuardedInsert` Diesel combinator and the `SitrepGuardedResource` trait: a generic primitive for FM rendezvous to insert a resource row idempotently and guarded against stale-sitrep execution. The combinator wraps a caller-supplied resource INSERT in a single CTE statement that: - aborts (StaleSitrep) unless the executor's expected generation still equals the latest sitrep's generation column; - short-circuits (AlreadyExists) if a creation marker already exists for the resource id; - on a successful insert, atomically writes a creation marker. The result is surfaced as a `SitrepGuardedInsertOutcome` of `Created` / `AlreadyExists` / `StaleSitrep`. Context: #10248. This PR was previously #10532 but I made a mess of it. This is used in #10533 and #10535 which are split out in hopes of making the review somewhat less miserable.
Wire the alert resource through `SitrepGuardedInsert`:
- `impl SitrepGuardedResource for Alert`;
- schema: `alert_generation` on `fm_sitrep` and the
`rendezvous_alert_created` marker table (migration
fm-alert-resource-deletion);
- `alert_create`'s FM path routes through the combinator, surfacing a
stale sitrep as a monomorphic `Error::Conflict`;
- `SitrepBuilder` tracks `alert_generation`, bumping it when the
outstanding alert-request set changes; the closed-case carry-forward
filter drops fully-satisfied closed cases;
- fm_rendezvous reads the expected generation and aborts the alert loop
on a stale mismatch; fm_analysis loads existing markers to drive
carry-forward; omdb displays the new status fields and generation.
The support-bundle mirror of the preceding alert change: `impl SitrepGuardedResource for SupportBundle`, the `support_bundle_generation` column and `rendezvous_support_bundle_created` marker table (migration fm-bundle-resource-deletion), `support_bundle_create`'s FM path routed through the combinator inside its transaction, support-bundle generation tracking and carry-forward, the fm_rendezvous bundle loop, the fm_analysis bundle-marker lookup, and the omdb display. `support_bundle_generation` and `alert_generation` are tracked and guarded independently, so a stale generation for one resource aborts only that resource's rendezvous loop.
Add the `SitrepGuardedInsert` Diesel combinator and the `SitrepGuardedResource` trait: a generic primitive for FM rendezvous to insert a resource row idempotently and guarded against stale-sitrep execution. The combinator wraps a caller-supplied resource INSERT in a single CTE statement that: - aborts (StaleSitrep) unless the executor's expected generation still equals the latest sitrep's generation column; - short-circuits (AlreadyExists) if a creation marker already exists for the resource id; - on a successful insert, atomically writes a creation marker. The result is surfaced as a `SitrepGuardedInsertOutcome` of `Created` / `AlreadyExists` / `StaleSitrep`. Context: #10248. This PR was previously #10532 but I made a mess of it. This is used in #10533 and #10535 which are split out in hopes of making the review somewhat less miserable.
aee00ec to
aab93e8
Compare
064090e to
5f46981
Compare
Rename first_sitrep_with_alert_request_bumps_generation -> first_sitrep_with_alert_request_bumps_alert_generation so the alert test reads clearly alongside the support bundle generation tests added later.
| /// For [`SupportBundleProvenance::Fm`] the bundle INSERT is gated by | ||
| /// [`SitrepGuardedInsert`], which also writes the | ||
| /// `rendezvous_support_bundle_created` marker atomically. The | ||
| /// transaction encloses the guarded insert and the data-selection | ||
| /// inserts, so a marker is never durable without its data-selection | ||
| /// rows. If a bundle with the given id already exists, returns | ||
| /// [`Error::ObjectAlreadyExists`]. If the latest sitrep's | ||
| /// `support_bundle_generation` has advanced past | ||
| /// `expected_support_bundle_generation`, returns [`Error::Conflict`]; the | ||
| /// caller should skip the request rather than retrying. | ||
| /// | ||
| /// The free-dataset lookup runs before the generation guard, so a stale | ||
| /// rendezvous activation that also finds no free dataset reports | ||
| /// [`Error::insufficient_capacity`] rather than [`Error::Conflict`]. | ||
| /// This only affects status reporting and log noise: the guard still | ||
| /// prevents a stale activation from inserting anything. | ||
| pub async fn support_bundle_create( |
There was a problem hiding this comment.
hmm. similarly to the suggestion I left on #10533, I wonder if we might want to refactor this as separate support_bundle_create and fm_rendezvous_support_bundle_create methods, rather than passing in a "provenance" type that tells the method which behavior to do --- I can't really imagine having code that wants to be able to call support_bundle_create with either provenance dynamically; we're only doing one code path or the other, so it feels a bit nicer to me for them to just be separate methods.
with that said, I realize that (unlike alerts) the provenance enum was already here, so 🤷♀️
There was a problem hiding this comment.
I had a reason for keeping the switch-on-provenance thing here, but now I honestly can't remember what it was or whether it was a good reason. I'll think about this.
There was a problem hiding this comment.
Ok, I messed with this a bit and decided to stop messing with it. The direction it was going, I had three separate SupportBundleCreateParams structs: one for support_bundle_create, another for fm_rendezvous_support_bundle_create, and a third for their shared helper function (which had some super fun trait bounds for the insert query closure)... It was getting ugly, and then my efforts to de-uglify all pointed in the direction of "hey wouldn't it be nice to just switch on provenance."
| return external::Error::conflict( | ||
| "cannot create FM-originated support \ | ||
| bundle for stale sitrep", | ||
| ); |
There was a problem hiding this comment.
Similarly to what we did in #10533, I think that if we're going to split the FM and user-requested support bundle creation code paths into separate functions, it would be nicer to just have the FM one return an error enum that represents these specific error cases more explicitly, rather than an external::Error.
I was going to suggest that we could take the FmRendezvousAlertCreateError type added in #10533 and rename it to FmRendezvousCreateError or something, and remove the string "alert" from the docs/formatted output (we could say the generic "resource" or something instead). But, I realized that support bundle creation also needs to be able to represent "insufficient capacity". I suppose we could do something like
enum FmRendezvousSupportBundleCreateError {
#[error("insufficient capacity for blah blah blah whatever")]
InsufficientCapacity,
#[error(transparent)]
Create(#[from] FmRendezvousCreateError),
}or something? I dunno if that's really worth it over just having separate error types...
There was a problem hiding this comment.
Yeah, see above... I agree in principle but it's a bit of a pain here in practice, so I'm punting on this.
Wire the support bundle resource through
SitrepGuardedInsert:impl SitrepGuardedResource for SupportBundle;support_bundle_generationonfm_sitrepand therendezvous_support_bundle_createdmarker table (migration fm-bundle-resource-deletion);support_bundle_create's FM path routes through the combinator, surfacing a stale sitrep asError::Conflict;SitrepBuildertrackssupport_bundle_generation, bumping it when the outstanding support-bundle-request set changes; the closed-case carry-forward filter drops fully-satisfied closed cases and keeps those with unsatisfied support bundle requests;support_bundle_generationandalert_generationare tracked and guarded independently, so a stale generation for one resource aborts only that resource's rendezvous loop.Context: #10248, builds on #10564 and #10533.