This repository has moved to pieteams/piekbs

WikiLoop

A knowledge search engine for agents — distill raw docs into structured Markdown wiki, search and read via MCP

English | 简体中文 | 繁體中文 | Русский | Deutsch | Français | Español | 한국어

WikiLoop is a local-first knowledge search engine for agents. It distills raw documents into a structured, reviewable Markdown wiki, then exposes two MCP tools — kb_search and kb_page — that let agents search and deep-read at their own pace.

Design Philosophy

WikiLoop is built around one observation: agents use external knowledge tools the same way humans use search engines — they issue multiple queries from different angles, follow links, and synthesize their own conclusions. They do not want a pre-packaged answer; they want the raw materials to form their own.

This means WikiLoop's job is not to answer questions. It is to make sure that when an agent searches for something, it finds the right documents — and can read them in full.

wikiloop-kb/
  raw/                  Source of truth — original materials in any format.
                        Drop files here; the watcher auto-distills them.

  wiki/                 Structured Markdown knowledge layer (LLM-maintained).
    source-notes/       One distilled note per raw document. FTS search target.
    concepts/           Cross-document synthesis: concepts and methodologies.
    comparisons/        Cross-document synthesis: side-by-side comparisons.
    decisions/          Cross-document synthesis: technical decisions.
    _draft/             Synthesized pages with < 2 sources (not indexed yet).

  schema/               KB-local authoring rules and page templates.
                        Edit these to customize the distilled page format.

  index/                Generated artifacts (SQLite FTS index, query logs).
                        Managed automatically — do not edit manually.

How Agents Use WikiLoop

Agents interact with WikiLoop through three MCP tools:

kb_search(query, limit?) — Search with a keyword or phrase. Returns up to 5 source-notes and 3 concept/comparison/decision pages per call. Each result includes a related field listing linked documents for navigation. Use multiple searches with different keywords to cover a topic from multiple angles.

kb_page(ids, full?) — Fetch full content of one or more pages by ID (from kb_search results). Pass up to 5 IDs to scan several documents at once, or full=true with a single ID to get the complete untruncated text.

kb_add(filename, content, source_url?) — Add a text document to the knowledge base. Writes content to raw/<filename> and triggers incremental indexing. Distillation runs asynchronously in the background. Use the converted/ prefix for agent-extracted PDF/Word/Excel/EPUB content.

The recommended agent workflow:

kb_search("keyword A")          → discover relevant documents
kb_search("keyword B")          → cover a different angle
kb_page(["id1", "id2", "id3"]) → deep-read the most relevant ones
Agent synthesizes its own answer from what it found

Agents are expected to search iteratively, follow related links, cross-verify across sources, and form their own conclusions. WikiLoop does not generate answers.

WikiLoop vs RAG

Traditional RAG retrieves context and hands it to the LLM to answer. WikiLoop hands the agent raw materials and lets the agent do the reasoning.

RAG:       user question → retrieve context → LLM answers
WikiLoop:  agent searches → agent reads → agent synthesizes

	RAG	WikiLoop
Knowledge form	Implicit (vectors or chunks)	Explicit (Markdown, auditable)
Agent role	Passive receiver of context	Active searcher and reader
Answer source	System-generated	Agent-synthesized
Auditable	No	Yes — git diff, lint, conflict links
Multi-hop reasoning	LLM-dependent	Graph expansion via related links
Embedding	Required	Not required (pure FTS)

WikiLoop bundles are conformant with OKF v0.1.

Knowledge Pipeline

Raw documents flow through a distillation pipeline before agents can search them:

Step 1 — Distill (automatic)

Drop any Markdown file into raw/. The wikiloop serve watcher automatically runs distill + index. The LLM extracts structured source-notes into wiki/source-notes/, including:

• key_claims with inlined aliases and cross-language equivalents (ALIAS RULE) — ensures FTS matches all query variants
• Named entity annotations in 【entity|type】 format
• related_to, supports, contradicts links — powers the related field in search results
• authority (1–5) and doc_type metadata

Step 2 — Synthesize (on-demand)

wikiloop synthesize --topic "RAG"

Generates concept / comparison / decision pages from source-notes when enough sources on a topic accumulate. Pages with fewer than 2 source references go to wiki/<type>/_draft/ and are not indexed until more sources are added.

Step 3 — Search

Agents use kb_search + kb_page via MCP. Search is pure FTS (SQLite FTS5 with BM25 scoring). No vector model required.

Installation

Download the latest release:

Platform	File
macOS Apple Silicon (ARM64)	WikiLoop-<version>-macos-arm64.dmg
Linux x86_64	wikiloop-<version>-linux-amd64.tar.gz
Linux ARM64	wikiloop-<version>-linux-arm64.tar.gz
Windows x86_64	wikiloop-<version>-windows-amd64.zip

macOS Intel (x86_64): No pre-built release. GitHub Actions dropped the Intel macOS runner in April 2025. Build from source on your Intel Mac: CGO_ENABLED=1 go build -tags fts5 -o wikiloop ./cmd/wikiloop/

macOS: Open the DMG and drag WikiLoop to Applications. The app runs as a menubar icon.

Linux:

tar -xzf wikiloop-<version>-linux-amd64.tar.gz -C /path/to/install/
sudo ln -sf /path/to/install/wikiloop /usr/local/bin/wikiloop

Windows: Extract the zip and run wikiloop.exe serve (or wikiloop.exe stdio for MCP). Add the directory to PATH for convenience. No CGO required — pure Go binary.

HarmonyOS PC (community, experimental): WikiLoop is not officially released for HarmonyOS PC. However, since the core binary requires no CGO (pure Go + SQLite), it can be built natively on HarmonyOS using the community Harmonybrew package manager. See ohos_go_cgo for a guide on setting up Go + CGO on HarmonyOS PC.

# On HarmonyOS PC (after installing Go via Harmonybrew)
CGO_ENABLED=0 go build -tags fts5 -o wikiloop ./cmd/wikiloop/
wikiloop serve

Building from Source

Requires Go 1.25+. No CGO required.

# macOS / Linux
go build -tags fts5 -o wikiloop ./cmd/wikiloop/

# Windows
go build -tags fts5 -o wikiloop.exe ./cmd/wikiloop/

Or use the multi-platform build script:

./scripts/build.sh [version] [target...]

Target	Output	Platform
darwin-arm64	dist/WikiLoop-<version>-macos-arm64.dmg	macOS Apple Silicon
linux-amd64	dist/wikiloop-<version>-linux-amd64.tar.gz	Linux x86_64
linux-arm64	dist/wikiloop-<version>-linux-arm64.tar.gz	Linux ARM64
windows-amd64	dist/wikiloop-<version>-windows-amd64.zip	Windows x86_64

Repository Structure

wikiloop/
  cmd/wikiloop/        # main entry point
  internal/
    kb/                # FTS indexing, search, graph expansion, page fetch
    mcp/               # MCP server (stdio + HTTP)
    watcher/           # file watcher for auto-distill + reindex
    distill/           # LLM distillation pipeline
    synthesize/        # concept/comparison/decision page generation
    convert/           # raw file conversion
    service/           # OS service manager (launchd / systemd)
    webui/             # web UI
    tray/              # macOS system tray (darwin only)
    config/            # KB config (config.yaml)
  scripts/
    build.sh           # multi-platform build script

Schema & Templates

wikiloop init populates the KB's schema/ directory with bundled authoring rules and page templates:

• schema/templates/: Markdown templates for source-note / concept / comparison / decision pages.
• schema/references/: authoring rules — page types, citation rules, conflict rules, directory structure.

The distill/synthesize prompts read these templates, so editing them customizes the generated wiki format per-KB.

Quick Start

export WIKILOOP_KB=/path/to/your-kb

wikiloop init           # scaffold KB dirs and copy schema/templates
wikiloop serve          # start server: MCP + Web UI + file watcher
wikiloop index          # build/update FTS index
wikiloop status         # index stats
wikiloop lint           # health-check wiki pages

Command Reference

All commands accept a global --kb <path> flag (defaults to $WIKILOOP_KB, then ~/wikiloop-kb).

Command	Description
wikiloop init [--force]	Scaffold KB dirs and copy bundled schema/templates.
wikiloop serve	Start the long-running server: HTTP MCP (/mcp) + Web UI + file watcher. Default when no subcommand is given.
wikiloop index	Build/update the FTS index from wiki/ and raw/ markdown.
wikiloop search <query>	FTS keyword search; prints ranked hits with paths and snippets.
wikiloop synthesize [--topic X] [--full]	Generate concept/comparison/decision pages from source-notes.
wikiloop synthesize --gaps --topic X	Knowledge-gap analysis for a topic.
wikiloop import-lark <URL>	Import a Lark/Feishu Wiki page and its embedded tables into raw/lark/. Requires a logged-in lark-cli.
wikiloop lint	Health-check wiki pages: missing frontmatter fields, broken source links.
wikiloop status	Print index stats (document counts, index size).
wikiloop service <install|uninstall|start|stop|status|logs>	Manage the OS service (launchd / systemd).

LLM config (config.yaml under KB root, distill section) is required for distill and synthesize.

MCP Server

WikiLoop exposes KB tools via the MCP protocol.

Available tools: kb_search, kb_page, kb_add

Admin operations (status, reindex, lint) are available via the Web UI or CLI (wikiloop status, wikiloop index, wikiloop lint).

---

Scenario 1: Local Multi-Agent Sharing

HTTP mode is recommended: one WikiLoop process shared by all agents — Claude Code, Cursor, VS Code (Copilot), Windsurf, Trae, Codex, Hermes, OpenClaw, and others.

Step 1: Start WikiLoop

export WIKILOOP_KB=/path/to/wikiloop-kb
wikiloop serve

On macOS, double-click WikiLoop.app to launch as a menubar icon.

Step 2: Configure HTTP MCP in each agent

Add to ~/.claude.json under mcpServers:

{
  "mcpServers": {
    "wikiloop": {
      "type": "http",
      "url": "http://127.0.0.1:8766/mcp",
      "headers": {
        "x-api-key": "${WIKILOOP_API_KEY}"
      }
    }
  }
}

x-api-key corresponds to server.api_key in config.yaml. Omit headers if no api_key is set.

---

Scenario 2: Hosted Agent Environments

In hosted environments (Hermes, OpenClaw, etc.), install WikiLoop on the persistent volume and invoke via stdio — WikiLoop starts as a subprocess of the agent host, with the watcher running in the background automatically.

Example (NAS-mounted OpenClaw/Hermes, mount point /root/.openclaw):

1. Install to persistent volume (one-time):

tar -xzf wikiloop-linux-amd64.tar.gz -C /root/.openclaw/wikiloop/
chmod +x /root/.openclaw/wikiloop/wikiloop

2. Install markitdown (recommended):

markitdown enables conversion of PDF, Word, Excel, PPT, and HTML files to Markdown before distillation. Without it, only .md and .txt files are distilled; binary files are indexed by filename only.

pip install markitdown
# verify
markitdown --version

Verified working on OpenClaw/Hermes (path: /root/.openclaw/workspace/bin/markitdown). Add workspace/bin to PATH or set the full path in your environment.

If markitdown is unavailable, agents can extract text themselves (using LLM vision or other tools) and write the result directly to $WIKILOOP_KB/raw/converted/<slug>.md — the watcher picks it up automatically.

3. MCP configuration:

Hermes (mcp_servers in agent config):

mcp_servers:
  wikiloop:
    command: /root/.openclaw/wikiloop/wikiloop
    args: [stdio]
    env:
      WIKILOOP_KB: /root/.openclaw/wikiloop-kb
      PATH: /root/.openclaw/workspace/bin:/usr/local/bin:/usr/bin:/bin

The KB directory is created automatically on first launch. No manual init needed.

4. Adding content to the knowledge base:

Agents with write_file access can write directly into the KB — the watcher detects changes and triggers indexing and distillation automatically.

Content type	Write to
Articles, notes, references (Markdown/text)	$WIKILOOP_KB/raw/<your-category>/<slug>.md
Agent-converted PDF / Word / Excel / EPUB content	$WIKILOOP_KB/raw/converted/<slug>.md

Files in raw/converted/ are treated as already-converted and go straight to distillation, skipping the markitdown step. All other paths under raw/ are processed through the full pipeline (convert → index → distill).

Organize subdirectories however makes sense for your content — WikiLoop does not enforce a fixed structure under raw/.

System Service (optional)

wikiloop serve includes a built-in watcher that automatically monitors the KB directory, triggers distill, and rebuilds the index. No additional setup required.

To make WikiLoop start on boot and run in the background, install it as a system service (macOS launchd / Linux systemd):

wikiloop service install --kb /path/to/your-kb
wikiloop service status
wikiloop service uninstall

Logs: {WIKILOOP_KB}/index/watcher.log