Skip to content

feat(algo): add GDS_PAGE_RANK backed by icebug (NetworKit, zero-copy Arrow) - #30

Open
zachwinter wants to merge 1 commit into
LadybugDB:mainfrom
zachwinter:feat/gds-icebug-pagerank
Open

feat(algo): add GDS_PAGE_RANK backed by icebug (NetworKit, zero-copy Arrow)#30
zachwinter wants to merge 1 commit into
LadybugDB:mainfrom
zachwinter:feat/gds-icebug-pagerank

Conversation

@zachwinter

Copy link
Copy Markdown

Summary

First icebug (NetworKit) bridge for the algo extension: CALL GDS_PAGE_RANK('G') RETURN node, rank delegates PageRank to libnetworkit instead of the hand-rolled implementation. This is the start of wiring icebug into the CALL GDS_* paths we discussed on Discord (deprecating the hand-rolled algos in favor of icebug's zero-copy Arrow columnar engine).

Per that plan, the icebug-backed functions coexist with the existing algos under GDS_* names for one release cycle, then replace their internals in place. PAGE_RANK/PR are untouched.

How it works

projected graph → InMemGraph CSR (Louvain's fwd/bwd scan idiom)
  → arrow::UInt64Array → NetworKit::GraphR (zero-copy) → NetworKit::PageRank
  → scores streamed back through the existing GDSResultVertexCompute pipeline

The bind/register scaffolding is copied verbatim from page_rank.cpp (GDSFunction::initSharedState/getLogicalPlan/getPhysicalPlan); only the tableFunc body changes — it builds the CSR, hands it to icebug, and maps pr.scores() back to (node, rank) rows. Node offsets are dense in both worlds, so the mapping is identity.

Files

  • algo/src/function/gds_page_rank.cpp — the bridge.
  • algo/src/include/function/algo_function.h, algo/src/main/algo_extension.cpp — declare + register GDS_PAGE_RANK.
  • algo/cmake/download_icebug.cmake — fetches the prebuilt icebug release (icebug-<os>-<arch> tag 12.8) into vendor/. Set -DICEBUG_SOURCE_DIR=<icebug checkout> to link a local source build instead (dev).
  • algo/CMakeLists.txt, algo/src/function/CMakeLists.txt — link libnetworkit + Arrow::arrow_shared + OpenMP::OpenMP_CXX, with rpath to the vendored lib.
  • algo/test/test_files/gds_page_rank.test.

Build requirements (needs your input)

The build now needs Apache Arrow C++ and OpenMP available (Arrow is a public dependency of libnetworkit). One thing to confirm for CI: the prebuilt icebug 12.8 links Arrow 24.x — the extension build needs a matching Arrow. Happy to adjust the vendoring (pin Arrow, or a newer icebug release built against the Arrow the CI uses) however you prefer. Locally I validated the full path with a source build of icebug against the system Arrow.

Testing

gds_page_rank.test (star graph): the hub ranks highest, and the ordering matches PAGE_RANK. Absolute values differ because NetworKit returns a sum-to-1 probability distribution (0.480 / 0.173…) vs the existing implementation's normalization — expected, and NetworKit's is the standard convention.

Notes / follow-ups

  • Single-node-table graphs for now (multi-table is a small follow-up).
  • GDS_LOUVAIN, GDS_NODE2VEC (ARRAY output → the vector extension's HNSW index), and personalized PageRank (icebug's built-in personalization/forSources) all follow this same pattern — one bridge, the whole NetworKit catalog.

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist

  • I have added tests.
  • I have requested a review from a maintainer.
  • Documentation — deferred; happy to add once the surface/build approach is settled.

…Arrow)

First icebug bridge for the algo extension. `CALL GDS_PAGE_RANK('G')` delegates
PageRank to libnetworkit instead of the hand-rolled implementation: the projected
graph's adjacency is materialized as CSR (reusing the Louvain fwd/bwd scan idiom),
handed to a NetworKit::GraphR over Arrow UInt64 buffers, run through
NetworKit::PageRank, and streamed back via the existing GDS result pipeline.

Per the plan to deprecate the hand-rolled algos: the icebug-backed functions live
alongside the existing ones under GDS_* names for one release cycle, then replace
their internals in place.

- gds_page_rank.cpp: the bridge (bind/register scaffold copied from page_rank.cpp).
- cmake/download_icebug.cmake: fetches the prebuilt icebug release into vendor/;
  -DICEBUG_SOURCE_DIR=<checkout> links a local source build instead (dev).
- CMake: link libnetworkit + Arrow::arrow_shared + OpenMP::OpenMP_CXX, with rpath
  to the vendored lib. Requires Apache Arrow C++ (matching icebug's pinned version)
  and OpenMP available to the build.
- Test: gds_page_rank.test (star graph; NetworKit PageRank is a sum-to-1 distribution,
  so the hub ranks highest — same ordering as PAGE_RANK, different normalization).

Notes / follow-ups: single-node-table graphs for now (multi-table next); GDS_LOUVAIN,
GDS_NODE2VEC, and personalized PageRank (icebug's built-in personalization) follow the
same pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@adsharma

Copy link
Copy Markdown
Contributor

Pin arrow works on other platforms, but not macosx. Bugscope has the same issue.

We'll likely have to update ladybug, icebug and all platform in sync unless we can figure out how to pin on homebrew.

@zachwinter

Copy link
Copy Markdown
Author

I understand your comment in Discord now!

For now, I -SKIP'd gds_page_rank.test for the OFF build. Would you prefer a proper conditional (maybe a SKIP_ICEBUG_DISABLED token mirroring SKIP_COMPRESSION_DISABLED) so it runs automatically once icebug is enabled? I'll open a PR in lbug core if so.

@adsharma

Copy link
Copy Markdown
Contributor

@zachwinter - icebug 13.0 out. Can you see if it fixes the problem here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants