feat(algo): add GDS_PAGE_RANK backed by icebug (NetworKit, zero-copy Arrow) - #30
Open
zachwinter wants to merge 1 commit into
Open
feat(algo): add GDS_PAGE_RANK backed by icebug (NetworKit, zero-copy Arrow)#30zachwinter wants to merge 1 commit into
zachwinter wants to merge 1 commit into
Conversation
…Arrow)
First icebug bridge for the algo extension. `CALL GDS_PAGE_RANK('G')` delegates
PageRank to libnetworkit instead of the hand-rolled implementation: the projected
graph's adjacency is materialized as CSR (reusing the Louvain fwd/bwd scan idiom),
handed to a NetworKit::GraphR over Arrow UInt64 buffers, run through
NetworKit::PageRank, and streamed back via the existing GDS result pipeline.
Per the plan to deprecate the hand-rolled algos: the icebug-backed functions live
alongside the existing ones under GDS_* names for one release cycle, then replace
their internals in place.
- gds_page_rank.cpp: the bridge (bind/register scaffold copied from page_rank.cpp).
- cmake/download_icebug.cmake: fetches the prebuilt icebug release into vendor/;
-DICEBUG_SOURCE_DIR=<checkout> links a local source build instead (dev).
- CMake: link libnetworkit + Arrow::arrow_shared + OpenMP::OpenMP_CXX, with rpath
to the vendored lib. Requires Apache Arrow C++ (matching icebug's pinned version)
and OpenMP available to the build.
- Test: gds_page_rank.test (star graph; NetworKit PageRank is a sum-to-1 distribution,
so the hub ranks highest — same ordering as PAGE_RANK, different normalization).
Notes / follow-ups: single-node-table graphs for now (multi-table next); GDS_LOUVAIN,
GDS_NODE2VEC, and personalized PageRank (icebug's built-in personalization) follow the
same pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Pin arrow works on other platforms, but not macosx. Bugscope has the same issue. We'll likely have to update ladybug, icebug and all platform in sync unless we can figure out how to pin on homebrew. |
Author
|
I understand your comment in Discord now! For now, I -SKIP'd gds_page_rank.test for the OFF build. Would you prefer a proper conditional (maybe a SKIP_ICEBUG_DISABLED token mirroring SKIP_COMPRESSION_DISABLED) so it runs automatically once icebug is enabled? I'll open a PR in lbug core if so. |
Contributor
|
@zachwinter - icebug 13.0 out. Can you see if it fixes the problem here? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First icebug (NetworKit) bridge for the
algoextension:CALL GDS_PAGE_RANK('G') RETURN node, rankdelegates PageRank tolibnetworkitinstead of the hand-rolled implementation. This is the start of wiring icebug into theCALL GDS_*paths we discussed on Discord (deprecating the hand-rolled algos in favor of icebug's zero-copy Arrow columnar engine).Per that plan, the icebug-backed functions coexist with the existing algos under
GDS_*names for one release cycle, then replace their internals in place.PAGE_RANK/PRare untouched.How it works
The bind/register scaffolding is copied verbatim from
page_rank.cpp(GDSFunction::initSharedState/getLogicalPlan/getPhysicalPlan); only thetableFuncbody changes — it builds the CSR, hands it to icebug, and mapspr.scores()back to(node, rank)rows. Node offsets are dense in both worlds, so the mapping is identity.Files
algo/src/function/gds_page_rank.cpp— the bridge.algo/src/include/function/algo_function.h,algo/src/main/algo_extension.cpp— declare + registerGDS_PAGE_RANK.algo/cmake/download_icebug.cmake— fetches the prebuilt icebug release (icebug-<os>-<arch>tag 12.8) intovendor/. Set-DICEBUG_SOURCE_DIR=<icebug checkout>to link a local source build instead (dev).algo/CMakeLists.txt,algo/src/function/CMakeLists.txt— linklibnetworkit+Arrow::arrow_shared+OpenMP::OpenMP_CXX, with rpath to the vendored lib.algo/test/test_files/gds_page_rank.test.Build requirements (needs your input)
The build now needs Apache Arrow C++ and OpenMP available (Arrow is a public dependency of libnetworkit). One thing to confirm for CI: the prebuilt icebug 12.8 links Arrow 24.x — the extension build needs a matching Arrow. Happy to adjust the vendoring (pin Arrow, or a newer icebug release built against the Arrow the CI uses) however you prefer. Locally I validated the full path with a source build of icebug against the system Arrow.
Testing
gds_page_rank.test(star graph): the hub ranks highest, and the ordering matchesPAGE_RANK. Absolute values differ because NetworKit returns a sum-to-1 probability distribution (0.480 / 0.173…) vs the existing implementation's normalization — expected, and NetworKit's is the standard convention.Notes / follow-ups
GDS_LOUVAIN,GDS_NODE2VEC(ARRAY output → the vector extension's HNSW index), and personalized PageRank (icebug's built-inpersonalization/forSources) all follow this same pattern — one bridge, the whole NetworKit catalog.Types of changes
Checklist