Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions docs/api/pytest-plugin.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,27 @@ RAMPART's pytest integration. Activates automatically when installed.
members:
- RampartSession
- TrialGroupResult

## Parallel Execution Hooks

When `pytest-xdist` is installed, the plugin registers `pytest_testnodedown` (as an optional hook) to merge worker results into the controller session. See [Parallel Execution](../usage/xdist.md) for the data flow and trust boundary.

::: rampart.pytest_plugin._xdist
options:
members:
- SCHEMA_VERSION
- WORKEROUTPUT_KEY
- SIZE_LIMIT_OPTION
- DEFAULT_SIZE_LIMIT_BYTES
- WorkerOutputError
- SchemaVersionError
- SizeLimitError
- is_xdist_worker
- is_xdist_controller
- get_dist_mode
- get_worker_count
- serialize_worker_data
- deserialize_worker_data
- finalize_worker
- handle_testnodedown
- discover_sinks_from_conftest
9 changes: 9 additions & 0 deletions docs/contributing/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,15 @@ This allows the same evaluator (e.g., `ToolCalled`) to be used in both attack an

Subclasses implement only `_execute_async` and `strategy_name`. They should **not** catch `InfrastructureError` — the base class handles it.

### Pytest Plugin

`pytest_plugin/` integrates RAMPART with pytest:

- `plugin.py` — hook registrations (configure, collection, sessionfinish, terminal summary, optional `pytest_testnodedown`).
- `_session.py` — session-scoped state container (`RampartSession`), trial-group aggregates, sink registry, idempotency and incomplete-run flags.
- `_collection.py` — per-test `ResultCollector` and the `ContextVar`-based handler that captures results from executions.
- `_xdist.py` — pytest-xdist support: detection helpers, JSON-safe serialization of `Result` objects, controller-side merge, and conftest-scanning sink discovery. Workers serialize their results into `config.workeroutput`; the controller deserializes via `pytest_testnodedown` and emits a single unified report. See [Parallel Execution](../usage/xdist.md) for the data flow and trust boundary.

### PyRIT Bridge

PyRIT is RAMPART's upstream dependency for converters and prompt generation. Its import chain is heavy, so:
Expand Down
1 change: 1 addition & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,5 @@ You provide an **adapter** that connects your agent to the framework. RAMPART pr
- **Execution strategies** — orchestrate injection, triggering, and evaluation lifecycles
- **Evaluators** — detect conditions in agent responses (tool calls, text patterns, side effects)
- **pytest integration** — markers for harm categorization and statistical trials, automatic result collection, terminal summaries
- **Parallel execution** — run tests across worker processes with `pytest-xdist`; RAMPART produces a single unified report
- **Reporting** — structured JSON output for CI dashboards
30 changes: 29 additions & 1 deletion docs/usage/ci-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,17 @@ RAMPART tests interact with real or simulated agents and may take longer than un
pytest tests/ -v --timeout=300
```

### Parallel Execution

For faster CI runs, use [`pytest-xdist`](xdist.md):

```bash
pip install pytest-xdist
pytest tests/ -n auto
```

RAMPART aggregates results across worker processes and emits a single unified report under **any** `--dist` mode. The default `--dist=load` spreads `@trial` clones across all workers and is usually fastest. Add `--dist=loadgroup` only when a trial group needs to stay on one worker (e.g. clones share a session fixture or per-group worker state). See [Choosing `loadgroup` vs `load`](xdist.md#choosing-loadgroup-vs-load) for details and security considerations.

---

## Trial Markers for Statistical Confidence
Expand Down Expand Up @@ -57,11 +68,28 @@ def rampart_sinks() -> list[ReportSink]:

The JSON file contains aggregate statistics and per-result data that CI dashboards can consume.

!!! tip "Running in parallel"
Under [`pytest-xdist`](xdist.md), prefer the `pytest_rampart_sinks` hook over the fixture — it is resolved on the controller, so it works the same in single-process and parallel CI runs. See [Registering Sinks](pytest-integration.md#pytest_rampart_sinks-hook).

---

## Pytest Options

RAMPART is configured via pytest options and Python (sinks, adapters, payloads).

### `--rampart-xdist-max-bytes`

Maximum size in bytes of a worker's serialized result payload when running under [`pytest-xdist`](xdist.md). Defaults to `67108864` (64 MB). Workers that exceed the cap log a warning and the controller marks the run as incomplete. Also configurable via the `rampart_xdist_max_bytes` ini option.

```bash
pytest -n auto --rampart-xdist-max-bytes=134217728 # 128 MB
```

---

## Environment Variables

RAMPART itself does not read environment variables. Your adapter and test configuration typically do. Setting them locally for ad-hoc runs:
Your adapter and test configuration typically read environment variables. Setting them locally for ad-hoc runs:

=== "Linux / macOS"

Expand Down
10 changes: 10 additions & 0 deletions docs/usage/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,16 @@ RAMPART's configurable components: [`LLMConfig`][rampart.core.llm.LLMConfig] for

---

## Parallel-execution tuning

RAMPART exposes one pytest option for parallel-execution tuning. Other components (LLM endpoints, agent configuration) typically have their own configuration conventions.

| Option | Default | Description |
|--------|---------|-------------|
| `--rampart-xdist-max-bytes` (CLI) / `rampart_xdist_max_bytes` (ini) | `67108864` (64 MB) | Maximum size of a worker's serialized result payload when running under [`pytest-xdist`](xdist.md). Workers exceeding the cap are recorded as incomplete in `TestRunReport.metadata`. |

---

## LLMConfig

Immutable configuration for an LLM endpoint. Used by [`LLMDriver`][rampart.drivers.llm.LLMDriver] and [`Payloads.generate_async()`][rampart.payloads.Payloads.generate_async].
Expand Down
44 changes: 44 additions & 0 deletions docs/usage/pytest-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ async def test_with_threshold(adapter):
- `ERROR` results count against the pass rate (they are not `SAFE`)
- The trial group aggregate appears in the terminal summary

!!! tip "Running trials in parallel"
Under [`pytest-xdist`](xdist.md), aggregation is correct under any `--dist` mode. The default `--dist=load` spreads trial clones across all workers and is usually fastest; use `--dist=loadgroup` only when a trial group must stay on one worker (shared session fixture or per-group worker state). See [Choosing `loadgroup` vs `load`](xdist.md#choosing-loadgroup-vs-load).

---

## Fixtures
Expand Down Expand Up @@ -98,6 +101,47 @@ def rampart_sinks() -> list[ReportSink]:
]
```

!!! warning "xdist compatibility"
Under [`pytest-xdist`](xdist.md), the controller process discovers fixture-based sinks by calling `rampart_sinks` directly. Fixtures that depend on other fixtures (e.g., `tmp_path_factory`, `request`) cannot be resolved on the controller and are skipped with a warning. Use a parameterless fixture or a module-level list to remain compatible:

```python
# Resolved on the xdist controller (controller-only — single-process
# discovery needs the fixture form above, or the hook below)
rampart_sinks = [JsonFileReportSink(output_dir=Path(".report"))]
```

For sinks that need configuration or dependencies, prefer the
`pytest_rampart_sinks` hook below — it is resolved on the controller and works
identically in single-process and parallel runs.

---

### `pytest_rampart_sinks` hook

For sinks that need configuration — or to register sinks in a way that behaves
identically in single-process and `pytest-xdist` runs — implement the
`pytest_rampart_sinks` hook in your `conftest.py`:

```python
# conftest.py
from pathlib import Path

from rampart.reporting import JsonFileReportSink


def pytest_rampart_sinks(config):
return [JsonFileReportSink(output_dir=Path(".report"))]
```

The hook receives the active `pytest.Config`, so you can build
sinks from CLI/ini options or environment variables. Multiple implementations are
supported; RAMPART emits to the **union** of every returned sink.

**Precedence:** when any `pytest_rampart_sinks` implementation exists, it is
authoritative and the `rampart_sinks` fixture path is skipped entirely (so a
project that defines both does not double-register). The fixture remains the
single-process fallback when no hook implementation is present.

---

## Automatic Result Collection
Expand Down
3 changes: 3 additions & 0 deletions docs/usage/results-and-reporting.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,9 @@ class MyDatabaseSink:

Define the `rampart_sinks` fixture in your `conftest.py`. See [pytest Markers & Fixtures](pytest-integration.md#rampart_sinks) for the setup and examples with multiple sinks.

!!! note "Parallel execution"
Under [`pytest-xdist`](xdist.md), workers send their results to the controller, which emits sinks **once** with a unified [`TestRunReport`][rampart.reporting.sink.TestRunReport]. For sinks that need configuration, prefer the `pytest_rampart_sinks` hook, which is resolved on the controller and works the same in single-process and parallel runs. The `rampart_sinks` fixture is still supported as a single-process fallback, but on the controller it cannot depend on other fixtures. See [Registering Sinks](xdist.md#registering-sinks-the-pytest_rampart_sinks-hook) for details.

---

## TestRunReport
Expand Down
Loading
Loading