feat: fort map-data consumer — gyms, stations & pokestops - with DNF filtering

[jfberry]

What this does

Routes fort map data — gyms, pokestops, and stations (getAll, getOne, getAvailable) — through Golbat's in-memory fort API (UnownHash/Golbat#385) when a scanner source has an endpoint, with transparent SQL fallback. Map pans stop issuing fort SQL entirely on endpoint-backed sources, and Golbat's DNF filtering returns only the forts that will render.

Architecture

• Endpoint branch per model (mem set on the source): POST /api/<type>/scan → per-record pure mapper → the existing, unchanged secondaryFilter. On any failure (503 / network / bad shape) the model logs and falls through to SQL — a dual source degrades gracefully.
• DNF is a superset narrow; secondaryFilter guarantees exactness. Pure backends (server/src/filters/fort/*) translate the map filters into Golbat DNF clauses (OR across clauses, AND within). The one hard invariant is never under-return; anything inexpressible (quest title/target conditions, gender, gym badges, time-window cutoffs) poisons to match-all or stays a client-side residual.
• Exact key semantics in the translation (form-exact pokemon pairs, per-reward-type clauses, amount-exact mega/stardust/xp, grunt-class exclusion subtraction, station liveness) so the DNF result matches what renders.
• Observability: each fetch logs the narrowing (DNF(n): X matched -> Y after secondaryFilter (-residual) | <clause shapes>), so any translation gap is visible in production logs.
• Deep-link parity: an off-viewport onlyManualId fort is fetched via the by-id endpoint and joins the candidate set, mirroring SQL's (bbox) OR id = ?.

Verified live (dual source, endpoint vs SQL)

Path	Result
Pokestop quests	2373 forts scanned → 8 returned → 8 rendered (−0 residual)
Pokestop invasions (+ exclude grunts)	2806 scanned → 107 → 107 (−0)
Pokestop lures / showcases	validated
Stations (incl. station_active, gmax)	validated; inactive-viewing unaffected (match-all preserved)
Gyms (raids/teams/badges)	validated, residual 0

Config / deploy

• Requires Golbat with feat(api): fort map-data + DNF filtering + maintained availability UnownHash/Golbat#385 (fort_in_memory on), deployed before this — unknown scan-body fields fail closed to the SQL fallback.
• Per scanner source: endpoint + secret (or HTTP auth) as for pokemon; DB config unchanged (fallback + un-migrated queries).

No test framework exists in this repo; verification was throwaway node goldens per pure module (mappers, DNF backends), eslint/prettier, and the live parity runs above.

🤖 Generated with Claude Code

[jfberry]

Validating in a test environment

Scope: this branch routes only gyms through Golbat (getAll/getOne/getAvailable). Pokestop markers/popups and stations still use SQL. Live validation needs Golbat #385 deployed with fort_in_memory = true.

1. Golbat prereqs — deploy #385, fort_in_memory = true; sanity-check (S=secret, G=golbat url):

curl -H "X-Golbat-Secret: $S" "$G/api/gym/available"
curl -H "X-Golbat-Secret: $S" -H 'Content-Type: application/json' -XPOST "$G/api/gym/scan" \
  -d '{"min":{"latitude":LAT1,"longitude":LON1},"max":{"latitude":LAT2,"longitude":LON2},"limit":1000,"filters":[]}'
curl -H "X-Golbat-Secret: $S" "$G/api/gym/id/<gym-id>"

2. ReactMap — git checkout feat/fort-consumer && yarn install, restart (server-only changes — no client rebuild).

3. Config — add endpoint + secret to the scanner DB source whose useFor includes gym (makes it a dual source). If you already did this for the Phase-1 pokestop test, gyms are already routed.

4. Confirm the endpoint path (logs):

• ReactMap, each ~15-min refresh: [GYMS] [GYM] loaded available from Golbat endpoint …/api/gym/available — …
• getAll/getOne are silent on success; a [GYM] /api/gym/scan … falling back to SQL warn = the endpoint failed → SQL.
• Golbat: available-gyms built in … each time getAvailable is hit.

5. Functional — gym + raid markers render; a popup shows team/slots/defenders and raid boss + moves + CP; the filter drawer populates teams, tiers (r), eggs (e), raid bosses.

6. Golden parity (endpoint vs SQL): curl "$R/api/v1/available/gyms?current=1" with fort_in_memory on (endpoint) vs off (SQL fallback); diff the two → expect match (bar time-sensitive active-raid keys).

7. Fallback — stop Golbat or set fort_in_memory=false → gyms should still render via SQL + the fallback warn appears (no hard endpoint dependency; a timeout falls back too).

8. Area restrictions — as an area-restricted user, confirm gyms appear only within permitted areas (exercises the new client-side filterRTree) — the one behavior that isn't a straight SQL mirror, so the most important to verify.

[jfberry]

This is now in my production and I have verified the DNFs through manual testing

[Mygod]

The endpoint-backed fort paths contain an access-control bypass and several contract, filtering, and fallback regressions. These can expose restricted station data or omit valid stations, quests, and invasions despite successful lint and build checks.

Full review comments:

• [P1] Enforce strict-area denial before endpoint station scans — src/models/Station.js:785-785
  When authentication.strictAreaRestrictions is enabled, restrictions are configured, and a user has no assigned areas, this endpoint branch returns before the SQL path's getAreaSql guard. filterRTree returns true when both lists are empty, so the user receives every station in the viewport instead of []; apply the strict-area guard before accepting endpoint rows.

• [P1] Stop sending the unsupported station_active filter — server/src/filters/fort/station.js:37-37
  For every station request except inactive mode, this builder emits station_active, but the Golbat ApiFortDnfFilter contract targeted here has no such field—the companion work adds stationed_gmax, not station_active. Huma rejects unknown request properties, so endpoint-only station sources return no markers and dual sources always fall back to SQL; add the Golbat field/evaluator or stop sending it.

• [P1] Make advertised type-20 mega rewards retrievable — server/src/models/pokestopAvailableMapper.js:100-104
  When /api/fort/available reports a type-20 TEMP_EVO_BRANCH_RESOURCE with a Pokémon, this advertises an m... filter, but buildPokestopDnfFilters admits only reward type 12 and parseRdmRewards leaves non-empty type-20 rewards as u20. Selecting the advertised mega key therefore drops the row before or during secondary filtering, including the current m150-150 fallback; normalize and accept type 20 consistently in both scan stages.

• [P1] Treat endpoint incidents as confirmed-capable — server/src/models/Pokestop.js:900-900
  For a pure endpoint pokestop source, DbManager never populates hasConfirmed, even though scan rows carry confirmed and all lineup slots. Passing undefined here makes onlyConfirmed ineffective for selected invasion filters and makes confirmed a... filters use the grunt's possible encounter pool instead of the observed slots; the endpoint context and getFilterContext should explicitly model confirmed support.

• [P1] Preserve every active incident during DNF narrowing — server/src/filters/fort/pokestop.js:277-278
  When a stop has multiple active incidents, Golbat's fort DNF index exposes only one flat IncidentCharacter/IncidentDisplayType, while with_incidents:true returns the full incident array. Narrowing on these fields drops a stop whenever the selected invasion or event is not the one held in the flat lookup, so secondary filtering never sees a valid match; poison multi-incident filters to match-all or index all active incidents.

• [P2] Keep historical rarity queries on dual DB sources — server/src/services/DbManager.js:287-287
  When an existing scanner DB schema—normally using one source for Pokémon and forts—is made dual by adding endpoint, this assignment makes its Pokémon source mem truthy. historicalRarity() then substitutes [] for that bound source and clears the historical rarity map on refresh despite the DB still being available; distinguish a pure endpoint from a dual source in that skip condition.

• [P2] Apply result limits after local fort filters — server/src/models/Gym.js:542-542
  When a DNF scan has more than queryLimits.gyms candidates—for example, a large viewport where most candidates fail the user's area polygon—Golbat stops traversal at this limit before filterRTree, freshness checks, and secondaryFilter run. Valid matches later in the traversal are lost, unlike SQL's WHERE-before-LIMIT behavior; the same pre-residual cap is sent by pokestop and station scans, so push those gates upstream or defer the final cap until after local filtering.

• [P2] Return null on endpoint-only getOne misses — server/src/models/Gym.js:854-854
  When an endpoint-only source returns 404 or no response for a stale ID, fetchFortById returns null and execution reaches this SQL query. That model has no bound knex, so it throws; DbManager.getOne uses Promise.all, causing the whole gymsSingle request to fail—the identical fallback exists in Pokestop.getOne—instead of returning no match. Return null when there is no bound DB fallback.

• [P2] Include confirmed rewards from every rocket slot — server/src/models/pokestopAvailableMapper.js:176-177
  When map.misc.fallbackRocketPokemonFiltering is false, this mapper is the source of confirmed a... availability keys, but it reads only slot 1. The SQL path adds configured rewards from slots 1, 2, and 3, so endpoint-backed sources omit filters for confirmed rewards in later slots; expose and map all three slots.

[jfberry]

Thanks for the thorough review, @Mygod. Walking through all nine — seven are addressed (across this PR and the Golbat companion #385), and two look like they were reviewed against a Golbat without #385's changes. Details below.

Fixed

• Add GraphQL Support #1 — strict-area denial before endpoint scans (P1). Confirmed bug: the endpoint paths filter with filterRTree, which returns true for empty area inputs, so a no-area user under strictAreaRestrictions was not denied the way the SQL getAreaSql denies. Added areaRestrictionsDenyAll (mirrors getAreaSql's strict-deny and empty-consolidated deny) and short-circuit each fort model's endpoint branch to []. — 15d82178
• Add Devices #3 — type-20 mega retrievable (P1). Root cause was upstream: Golbat's vendored proto didn't decode reward type 20 at all (empty info), which is what the m150-150 empty-info stopgap worked around. With Golbat Enable new Pokemon by default #382 merged (decodes TEMP_EVO_BRANCH_RESOURCE → pokemon_id/amount), type 20 is now treated as mega everywhere — SQL match/advertise/search, parseRdmRewards, and the endpoint DNF (quest_reward_type: [12, 20]) — and the GoFest stopgap is retired. — ReactMap cade9f68, Golbat c3acb26
• Refactoring #4 — endpoint incidents confirmed-capable (P1). A pure-endpoint pokestop source now reports hasConfirmed: true (the Golbat scan always returns confirmed lineup data; no DB schema check runs for a knex-less source). — f51f662e
• Add Ingress Portals #6 — historical rarity on dual sources (P2). historicalRarity() now skips only pure-endpoint sources (no bound knex) rather than any source with mem, so a dual source keeps contributing. — f51f662e
• Add weather #7 — fort scan result limit (P2). On the Golbat side, the fort scan traversal cap read Tuning.MaxPokemonResults; added a dedicated max_fort_results (default 9000) so the fort API can be tuned independently of pokemon. The per-request Limit still min()s it. — Golbat a3c82e5
• Add S2 Scan Cells #8 — getOne on endpoint-only misses (P2). DbManager.getOne now uses Promise.allSettled, so a pure-endpoint source whose by-id fetch misses (falling through to an unbound this.query()) no longer fails the whole single-fort lookup. — f51f662e
• Add Submission Cells #9 — confirmed rewards from every rocket slot (P2). Golbat's invasion availability exposed only slot 1; it now exposes slots 2/3 (FortLookupIncident/invasionKey/ApiPokestopInvasionAvailable), and the mapper advertises an a key per slot the event marks (second/thirdReward). Index cardinality is unchanged — a confirmed grunt's lineup is deterministic per character, so (character, slot1) already determines slots 2/3. — Golbat e8c4505, ReactMap fe24b481

Not reproduced (appear to predate Golbat #385)

• Add Initial Filtering Menus #2 — station_active unsupported filter. station_active is part of the Golbat ApiFortDnfFilter contract in the companion Correct Scrolling Issue #385: it's a declared field with an evaluator (isFortDnfMatch: *StationActive != (StationEndTimestamp > now)) and a dedicated test. Huma rejects only unknown properties, and this one is declared — so endpoint station sources are not rejected. Correct Scrolling Issue #385 adds both stationed_gmax and station_active.
• Add Spawnpoints #5 — multi-incident DNF drops stops. FortLookup.Incidents is a slice ([]FortLookupIncident), and isFortDnfMatch iterates all non-expired incidents with match-any semantics, not a single flat incident. ReactMap also emits incident_character and incident_display_type as separate clauses (never AND-ed within one), so a stop with the selected invasion in any active incident still matches. If you saw a drop against Correct Scrolling Issue #385's matcher I'm happy to dig into a specific repro.

All changes build/lint clean; Golbat decoder tests pass (incl. new slots-2/3 coverage). #3, #9, and the type-20 path assume a deployed Golbat ≥ #385 + #382.

[jfberry]

I've taken the change to support slot2/slot3 availability and querying (which I previously had not added because these fields are always empty). Just like AR searching, these fields are legacy fields at the moment - but slot2/3 data coming back is something we would hope for though unlikely, whereas AR is something which is likely gone forever. A focused change later to drop AR data capture & reporting will be a wider impact in golbat

[Mygod]

Type-20 workaround has been removed in 9acf867. Please merge main and resolve conflicts. Hopefully it will make things easier.

[jfberry]

Merged develop (through 9acf867) into the branch — thanks, that did make it simpler.

My type-20 handling has converged on develop's: I reverted my SQL-side type-20 generalization (it was redundant with 9acf867 + the reward-definition refactor) and took develop's TEMP_EVOLUTION_RESOURCE_REWARD_TYPES / QUEST_REWARD_FILTER_DEFINITIONS approach wholesale. The only fort-consumer-specific pieces that remain are:

• Endpoint DNF (filters/fort/pokestop.js): mega clauses emit quest_reward_type: [12, 20] so Golbat returns type-20 (temp-evo branch) mega stops for the in-memory scan — develop doesn't have this file.
• parseRdmRewards guard: the merge resolution keeps typeof quest.quest_rewards === 'string' ? JSON.parse(...) : quest.quest_rewards (+ an Array.isArray guard). The endpoint path hands parseRdmRewards an already-parsed rewards array rather than a JSON string, so develop's JSON.parse(quest.quest_rewards)[0] alone would throw on endpoint sources. Flagging it as the integration seam if parseRdmRewards gets touched again.

No conflict markers, lint clean; the DNF, type-20 key, and rocket slots-2/3 mapper pass their unit checks. This assumes deployed Golbat ≥ #385 + #382 for the type-20 / slots-2/3 paths.

[Mygod]

The endpoint Pokestop path currently returns no markers because it omits a required filter limit, and one supported pure-endpoint configuration can fail during startup. Additional endpoint capability, availability, manual-ID, performance, and source-maintainability regressions also remain.

Full review comments:

• [P1] Pass the result limit to endpoint filtering — server/src/models/Pokestop.js:939-943
  For every successful endpoint-backed Pokestop scan, this call omits secondaryFilter's final resultLimit argument. Its loop condition is filteredResults.length < resultLimit; with undefined this is false immediately, so even non-empty endpoint responses produce no markers. Pass queryLimits.pokestops as the final argument, as the SQL call below does.

• [P1] Keep pure endpoint filter context off the SQL path — server/src/services/DbManager.js:285-285
  When fallbackRocketPokemonFiltering is disabled and Pokestop uses a pure endpoint source, setting hasConfirmed here makes Pokestop.getFilterContext() bypass its !hasConfirmed return and execute this.query() on an unbound model. state.loadLocalContexts() awaits that during startup, so initialization rejects; make the filter-context path recognize mem and return endpoint capability without SQL.

• [P2] Preserve the appended manual Pokestop at the limit — server/src/models/Pokestop.js:929-931
  When the scan already returns queryLimits.pokestops candidates and onlyManualId points outside the bounding box, the by-ID record is appended to the end and then this truncation immediately removes it. Deep links therefore fail in full viewports; retain the manual row when capping or apply the final limit after secondary filtering.

• [P2] Drop incomplete type-20 rewards from availability — server/src/models/pokestopAvailableMapper.js:108-108
  When Golbat reports a type-20 tuple without both a positive pokemon_id and amount, this emits u20 or m{id}-0, but secondaryFilter treats type 20 as a dedicated mega reward and drops it when its computed mega key is empty. The drawer consequently advertises a filter that can never return a marker; match the SQL path by adding type 20 only when both fields are present.

• [P2] Mark dual endpoint Pokestops as confirmed-capable — server/src/services/DbManager.js:292-295
  When a dual source's database lacks the confirmed column, schemaCheck leaves hasConfirmed false and this overlay copies only endpoint credentials, even though subsequent getAll calls use Golbat rows carrying confirmation and lineup slots. Consequently onlyConfirmed is ignored and reward filtering falls back to possible grunt encounters instead of the actual lineup; override this capability whenever the endpoint is active.

• [P2] Do not send unsupported station_active clauses — server/src/filters/fort/station.js:37-37
  For endpoint-backed station scans, Golbat's ApiFortDnfFilter has no station_active field and its JSON decoder silently ignores unknown members. In onlyAllStations mode this therefore becomes an empty clause matching every expired station, so the intended cache and payload reduction does not occur; add the corresponding Golbat predicate or leave liveness as an explicit residual match-all.

• [P2] Replace literal NULs in the cache key — server/src/utils/fortAvailable.js:31-31
  The separators in this template literal are literal NUL bytes, causing Git to classify the entire JavaScript file as binary so normal source diffs and some text tooling cannot inspect it. Use escaped separators such as \0 in the source to preserve the runtime key while keeping the file reviewable.

Also please change the merge target to main branch, also there has been some additional changes introduced in the main branch meanwhile.

[jfberry]

Thanks — all seven addressed, plus retargeted to main and merged the latest main.

P1

• Result limit not passed to endpoint filtering — the endpoint secondaryFilter call now passes queryLimits.pokestops (its loop is filteredResults.length < resultLimit, so undefined returned zero markers). Removed the pre-truncation entirely, mirroring the SQL call.
• Pure-endpoint filter context off the SQL path — getFilterContext now recognizes mem and returns { hasConfirmedInvasions: true } without touching this.query(), so startup no longer rejects on an unbound model when fallbackRocketPokemonFiltering is off.

P2

• Manual Pokestop at the limit — fixed by the same change: no pre-truncation, so the appended off-viewport by-id row survives into secondaryFilter (parity with SQL's OR id = manualId + LIMIT).
• Incomplete type-20 in availability — questRewardKey/the mapper now emit m<id>-<amt> only when both pokemon_id and amount are present, and never u20 (type 20 has a dedicated filter). Matches what secondaryFilter keys, so no dead drawer entries.
• Dual endpoint confirmed-capability — the dual overlay now also sets hasConfirmed: true, so onlyConfirmed and confirmed rocket-reward filters use the Golbat lineup even when the bound DB lacks the confirmed column.
• Literal NULs in the cache key — switched to the \0 escape in source; git no longer classifies fortAvailable.js as binary.

station_active — I'll push back gently here: it is in the Golbat companion (#385) — a declared ApiFortDnfFilter field with an evaluator (*StationActive != (StationEndTimestamp > now)) and a test, not an unknown member. You're right that Huma silently ignores genuinely-unknown fields (no additionalProperties: false), but that's the general DNF-superset contract — any field a given Golbat doesn't recognize is ignored and secondaryFilter/the station time-gate still finalizes correctness (just without the payload reduction). Against a Golbat that predates #385's station changes it degrades to residual match-all rather than breaking. The consumer already requires #385, so it resolves optimally there. Happy to add a hard residual fallback if you'd rather not depend on the field.

Base is now main; note the diff currently carries develop's commits that aren't yet in main — those drop out once develop lands on main.

[jfberry]

Whole-PR review + fixes

Given how much moved recently (the develop/main merges, type-20 convergence, two review rounds), I ran a full review across three dimensions — correctness/SQL-parity, security/access-control, concurrency/integration — over the ~2,050-line feature diff, verifying each finding against the code. 8 issues surfaced (2 P1, 3 P2, 3 P3); 7 are fixed in 09ac0769, 1 is a documented accept.

P1 (fixed)

• getOne returned the full Golbat record — SQL getOne projects to {lat, lon}; the endpoint returned the raw record, and since the client controls the GraphQL selection, a gyms-but-not-raids user could read raid/team/detail for any fort id, outside their areas. Now projected to {lat, lon} to match SQL.
• Pokestop onlyLevels under-return — the endpoint applied the power-up gate to every row, but SQL applies it only under onlyAllPokestops (the gym sibling has the !onlyAllGyms || guard). A user with a non-all levels filter lost the whole quest/invasion/lure layer. Added the !onlyAllPokestops || guard.

P2 (fixed)

• search / submissionCells crashed for pure-endpoint sources — those fort methods have no endpoint branch, and DbManager batched them with Promise.all, so a pure-endpoint source rejected the whole batch. Switched to Promise.allSettled (mirrors the getOne fix), degrading that source to no results.
• Credentials written to disk — fetchJson dumped request options (incl. X-Golbat-Secret / Basic auth) to logs/ on any failed request. Now redacts those headers before the debug log and the payload dump.
• Availability TTL armed on empty refresh — a failed (empty) endpoint refresh armed the 60s TTL and suppressed recovery. TTL is now only armed on a non-empty result.

P3

• Fixed: encodeURIComponent on all fort by-id fetch URLs (no path traversal via id/onlyManualId); historicalRarity → allSettled so a dual DB lacking pokemon_stats doesn't blank every source's rarity map.
• Accepted (not changed): force doesn't bypass the 30s combined-availability cache — that cache deliberately caches failures to avoid hammering an old/down Golbat, and the TTL fix already cuts the drawer-empty window to ≤30s; a force-bypass needs invasive threading for a minor, drawer-only gain.

Verified solid (no action)

• All three DNF builders correctly mirror SQL as superset-narrows — layer-gating, exact quest keys (form:0/bare, amount-exactness, type-20 [12,20]), rocket→incident_character expansion, onlyGymBadges→[], station station_active. No under-returns.
• All availability mappers reproduce the SQL getAvailable keys (incl. type-20 complete-only, rocket slots 1/2/3).
• Strict-area guard is complete on all three getAll endpoints and faithfully mirrors getAreaSql; manual-id preserves area filtering; cache key is credential-isolated.
• Golbat's DNF matches both AR and non-AR quest layers (arMatch || noArMatch) — no systematic layer under-return.

node --check + ESLint clean across all changed files. As before, worth a functional smoke test on a pokestop source (quests + confirmed invasions) once pulled, given there's no server test suite. Requires Golbat ≥ #385 + #382.

[jfberry]

(running in my production, will also ask my other testers of this PR to update after a short period of no further review comments)