pdbcompat: full upstream depth-parity (v1.20.5)#29
Merged
Conversation
At depth=2 the nested reverse-set elements embedded in a parent object
must drop exactly the parent-FK fields upstream excludes, no more and no
less. Two cases diverged from peeringdb_server/serializers.py:
- CampusSerializer.fac_set excludes ["org_id", "org"], so a facility
nested under a campus keeps campus_id and drops org_id. We were doing
the inverse: stripping campus_id and leaking org_id.
- CarrierSerializer.carrierfac_set excludes ["fac"] only, so carrier_id
stays on each carrierfac element. We were stripping carrier_id.
Both were verified against live www.peeringdb.com payloads. The org and
net reverse-set cases already matched (org_id / net_id respectively) and
are unchanged.
Regenerate the campus and carrier goldens to reflect the corrected
back-references.
Upstream IXLanSerializer (peeringdb_server/serializers.py:3407) exposes a
single reverse collection on an ixlan:
net_set = nested(NetworkSerializer, source="netixlan_set_active_prefetched",
through="netixlan_set", getter="network")
— a list of flat Network objects reached through the netixlan join, one
entry per active join row (no dedup, join order). There is no
netixlan_set field on the ixlan surface at all.
The mirror was exposing the raw NetworkIxLan join rows under a
netixlan_set key and omitting net_set entirely, so a drop-in client
reading ixlan.net_set got nothing and instead found an unexpected
netixlan_set of a different shape. Resolve each active join row to its
Network and emit net_set with the flat NetworkSerializer shape; drop
netixlan_set.
Verified against live www.peeringdb.com/api/ixlan payloads (the net_set
element carries asn/info_*/fac_count — Network fields — not the
ipaddr4/speed of a join row). Regenerate the ixlan goldens.
At depth=2 upstream gives every singular FK object embedded one level down its own reverse relations as bare ID lists plus its own forward FK as a flat object — e.g. a netixlan's net carries poc_set/netfac_set/ netixlan_set (ascending IDs) and a flat org; its ixlan carries net_set/ ixpfx_set and a flat ix. The mirror stopped one level short and rendered those FK objects flat (the bounded divergence documented in v1.19.3). Generalise the existing nestedOrgMap/nestedCampusMap pattern into nestedNetMap, nestedFacMap, nestedIxMap, nestedIxLanMap and nestedCarrierMap, and route every leaf getter (poc, netfac, netixlan, ixfac, carrierfac, ixpfx) plus the top-level ixlan's ix through them. Direct reverse-set ID lists are ascending (sortedIDsOrEmpty); the two through-relation sets — an ix's fac_set via ixfac and an ixlan's net_set via netixlan — preserve join order with duplicates (intsOrEmpty), exactly as upstream's prefetch iteration does. Second-level FK objects remain flat, matching upstream's depth budget. Verified against live www.peeringdb.com payloads (2026-06-08). Regenerate the poc/netfac/netixlan/ixfac/carrierfac/ixpfx/ixlan goldens.
The detail handler only accepted ?depth=0 and ?depth=2, silently
coercing every other value (including 1, 3, 4) to the default 2. Upstream
parses ?depth= as a raw int clamped to [0, 4] for single GETs
(serializers.py:789-823) and renders three distinct shapes: 0 is the bare
row, 1 expands forward FK objects flat with reverse sets as bare ID
lists, and 2 fully expands. A client asking for depth=1 therefore got the
heavier, differently-shaped depth=2 payload.
Clamp the parsed depth to [0, 4] instead of allow-listing {0, 2}, and add
a real depth==1 branch to every getter. The depth=1 shape of a top-level
object is exactly the shape a singular FK object takes when embedded one
level down at depth=2, so the parent getters reuse the nested*Map
builders directly and the leaf getters render their FK objects flat. The
existing depth=2 expansion is unchanged. Depths >2 render the depth=2
shape; the extra sub-level nesting they would add upstream is not
reproduced (almost no client reads beyond two levels).
Verified against live www.peeringdb.com depth=0/1/2 payloads.
The depth-parity work (real ?depth=1, ixlan net_set, and the second-level nested-set expansion) grew every depth=2 detail row, so the response-budget floor in typicalRowBytes had drifted past the 20% threshold the table documents. Re-measure the Depth2 column from BenchmarkRowSize (benchtime=20x × count=3, 2× ceil-64); the leaf join entities grew most (netixlan 2752→4928, ixfac 3008→4416, ixpfx 896→2240) now that they embed each FK object's own ID-list sets. Depth0 (the list shape) is unchanged. Route ?depth=1 through the Depth2 estimate rather than the bare-row figure: it is now a served level whose reverse-relation ID lists exceed the bare row, so billing it the bare size would under-count the budget. The estimate stays a per-row floor — a single detail object never approaches the 128 MiB envelope regardless.
Upstream's FacilitySerializer exposes campus as a related field (serializers.py:1728, related_fields/list_exclude at 1816-1818): it is excluded from the bare list/depth-0 row but present at detail depth, serialized as null when the facility has no campus rather than omitted. Our nested and top-level facility getters only set campus when the edge was present, dropping the key entirely for a campus-less facility — so a carrierfac's campus-less fac at depth=2 carried no campus key where upstream carries campus:null. Emit the explicit null in both nestedFacMap and getFacWithDepth. Surfaced by the live patched-vs-upstream comparison (the only structural residual outside the intentional POC-visibility privacy difference).
Bring the documentation in line with the v1.20.5 pdbcompat depth-parity
work and trim the divergence surface to what actually diverges:
- API.md: the ?depth= parameter now documents the real 0-4 ladder
(flat / FK-flat+ID-lists / full) and a depth=1 example. The Known
Divergences table drops from 7 verbose rows to 5: the depth=2
second-level row is gone (fixed), the ?status= and ?limit=0 rows are
gone (not divergences / a duplicate Validation Note), and the two
3-hop traversal rows merge into one 2-hop-cap row. Two real entries
are added: detail depth 3-4 rendering the depth-2 shape, and the
intentional poc_set privacy stance (we omit non-Public POC ids that
upstream lists at depth=1).
- ARCHITECTURE.md: recalibrated the per-entity depth=2 row-size table
and noted that ?depth=1 bills the depth=2 estimate.
- CLAUDE.md: a concise depth-expansion note (the nested*Map builders,
direct-vs-through set ordering, ixlan net_set, campus:null) and the
PK-lookup site count bumped 26 -> 27.
- CHANGELOG: a [1.20.5] entry covering the five parity fixes and the
one retained intentional divergence.
Code Metrics Report
Code coverage of files in pull request scope (76.1%)
Reported by octocov |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Brings the pdbcompat
/api/single-object depth responses to full paritywith upstream PeeringDB. A live shape comparison of all 13 entity types at
?depth=0/1/2againstwww.peeringdb.com/api(2026-06-08), cross-checkedagainst
peeringdb/peeringdbsource, surfaced four undocumented ormis-documented divergences — all corrected here, then re-validated live.
Released as v1.20.5 (a parity bugfix).
Fixes (one bisectable commit each)
align nested back-ref FK strippingorg_id/droppedcampus_id(backwards); carrier.carrierfac_set strippedcarrier_id. Now matches upstream's per-serializerexclude=lists.expose ixlan net_set instead of netixlan_setnet_set(Networks viagetter="network"); we exposed the raw join rows under the wrong key and omittednet_set.render second-level nested sets at depth=2net) now carry their own reverse relations as ID lists, matching upstream's recursive depth budget. Removes the bounded divergence deferred in v1.19.3.implement a real ?depth=1 and honour the full depth range?depth=0/2and silently coerced everything else (incl.1) to2. Depth is now clamped to[0,4];depth=1expands forward FKs flat with reverse_setfields as bare ID lists.emit campus:null for campus-less facilitiesFacilitySerializer.campusis a related field present at detail depth, null when absent; we omitted the key.Plus:
recalibrate depth=2 row-size floor(budget estimates grew with thericher rows;
?depth=1bills the depth=2 estimate) and a docs pass thatshrinks the Known Divergences table from 7 verbose rows to 5.
Validation
Deployed to the fleet and compared the patched live instance vs a fresh
upstream fetch, by-ID aligned, all 13 types × depth 0/1/2 + 3 filters:
every structural divergence dropped to zero (before: org depth=1 had 157
extra paths; depth=2 leaf entities had 31–76 missing paths each).
One intentional non-parity is retained and documented: anonymous
poc_setID lists omit non-
PublicPOC ids that upstream lists at?depth=1(upstream hides them only on expansion). Matching upstream there would leak
the existence of non-
Publiccontacts, contradicting the row-levelpoc.visibleprivacy policy.Tests
internal/pdbcompat/depth_test.gogains depth=1, clamp, ixlannet_set,second-level, back-ref-strip, and campus-null parity tests, all written
test-first against the live upstream shapes. Goldens regenerated.
go build,go vet,go test -race ./internal/pdbcompat/...,golangci-lint, and thego generatedrift check all pass.🤖 Generated with Claude Code