[META]: Migrate pyright to ty by spencrr · Pull Request #86 · microsoft/RAMPART

spencrr · 2026-06-17T00:03:07Z

Description

Migrate type checker from pyright to ty to align with PyRIT (microsoft/PyRIT#1319).

Typing fixes prompted by ty

InjectionHandle and Session now inherit AbstractAsyncContextManager (rampart/core/injection.py, rampart/core/adapter.py). ty rejected AsyncExitStack.enter_async_context(handle) because a Protocol with __aenter__ -> Self doesn't unify with AbstractAsyncContextManager[Handle, _ExitT_co]'s covariant _T_co without an explicit subtype relationship.

class InjectionHandle(AbstractAsyncContextManager["InjectionHandle", None], Protocol):
class Session(AbstractAsyncContextManager["Session"], Protocol):

Notes:

typeshed declares AbstractAsyncContextManager as both ABC and Protocol (stdlib/contextlib.pyi) - it's on the type-checker allowlist, so a Protocol may legally inherit from it.
Base order matters for C3 linearization: the ABC must precede Protocol.
Generic args [InjectionHandle, None]: _T_co = __aenter__ return; _ExitT_co = None because cleanup never suppresses exceptions (closing/nullcontext family, not suppress).
__aexit__ aligned with stdlib conventions: positional-only /, canonical exc_value/traceback names.

Attacks.xpia and coerce_driver narrowing reordered (rampart/attacks/init.py, rampart/drivers/_utils.py). ty cannot yet narrow element type through isinstance(x, list) over list[A] | list[B]. Reordering to check the protocol/scalar branch first eliminates the need for cast and # type: ignore.

ResponseContains branch order tightened (rampart/evaluators/response_contains.py). callable() doesn't narrow against a union containing str (every str is callable for ty's purposes here). Reordering to re.Pattern -> str -> callable and using if found is True: makes the dispatch unambiguous.

pytest_plugin/plugin.py (rampart/pytest_plugin/plugin.py): replaced a register_default_handler_factory lambda with a typed module-level _default_handler_factory so list[ExecutionEventHandler] aligns under list invariance. pytest.Item user-attribute writes now use # ty: ignore[unresolved-attribute].

Test changes

tests/unit/attacks/test_xpia.py: _mock_handle and _mock_evaluator now use spec=InjectionHandle / spec=Evaluator. With the reordered narrowing in Attacks.xpia (isinstance(inject, InjectionHandle) first), bare AsyncMock() becomes ambiguous because runtime-checkable protocols treat any Mock as matching (every hasattr returns True). spec= makes isinstance deterministic.
tests/unit/core/test_adapter.py: @abstractmethod added to a protocol-conformance test fixture so ty doesn't flag the ... body.
Test-only # pyright: ignore -> # ty: ignore (e.g., tests/unit/core/test_llm.py).

Docs

Five contributing pages updated to reference ty: docs/contributing/index.md, development-setup.md, code-style.md, architecture.md, pull-requests.md.

Breaking changes

None for users. Contributors will need to run uv sync to install ty (pyright is removed from the dev group) and update editor integrations - call out in development-setup.md.

Checklist

pre-commit run --all-files passes
Tests added or updated for changes - tests/unit/attacks/test_xpia.py mocks gain spec= for deterministic protocol isinstance; pyright ignore strings ported in tests/unit/core/test_llm.py, test_execution.py, test_protocols.py, test_types.py, etc.
Documentation updated - 5 contributing pages

Verification

uv run ty check: All checks passed
uv run ruff check . && uv run ruff format --check .: clean
uv run pytest: 491 passed, 5 skipped
uv run pre-commit run --all-files: all hooks pass

Aligns RAMPART with sister project PyRIT (#1319), which moved off mypy to ty for type checking. RAMPART previously used pyright (strict). ty is Astral's Rust-based type checker; the migration adopts the same shape PyRIT settled on but tightened for a smaller codebase. Changes: - pyproject.toml: drop pyright dev dep, add ty>=0.0.38; replace [tool.pyright] config with [tool.ty.environment], [tool.ty.src], and [tool.ty.rules]. Add [[tool.ty.overrides]] for tests/** that disables six rule categories where the tests intentionally feed MagicMock objects to typed pytest hooks, construct objects with deliberately wrong args to exercise validation paths, and define Protocol stub classes with '...' bodies. - .pre-commit-config.yaml: replace pyright hook with ty hook. - .github/workflows/ci.yml: replace 'uv run pyright' step with 'uv run ty check'. - Source fixes in rampart/: - attacks/__init__.py: annotate 'handles: list[InjectionHandle]' so the union-of-list branches collapse cleanly; ty cannot yet narrow element type through isinstance(x, list), so the one remaining branch carries a targeted suppression. - pytest_plugin/plugin.py: replace the lambda passed to register_default_handler_factory with a typed module-level function so the Protocol return type aligns (list invariance). Replace pyright suppressions for dynamic pytest.Item attributes with ty equivalents. - attacks/_xpia.py, drivers/_utils.py, evaluators/response_contains.py: add narrow # ty: ignore comments with explanations for ty's current inference gaps (Self-typed __aenter__, list[A]|list[B] isinstance narrowing, callable() narrowing against str-containing unions). - Strip all '# pyright: ignore[...]' comments project-wide; tests/** overrides cover the equivalent test-only cases. - Update contributor docs (development-setup, code-style, architecture, pull-requests, contributing index) to reference ty. Verification: - uv run ty check: All checks passed - uv run ruff check . / ruff format --check .: clean - uv run pytest: 448 passed - uv run pre-commit run --all-files: all hooks pass

ty rejected `AsyncExitStack.enter_async_context(handle)` because `InjectionHandle` was structurally close to but not assignable to `AbstractAsyncContextManager[InjectionHandle, None]` — `__aenter__` returning `Self` doesn't unify with the parent's covariant `_T_co` when there's no explicit subtype relationship. Inheriting from `AbstractAsyncContextManager["InjectionHandle", None]` fixes this. Notes: - typeshed declares `AbstractAsyncContextManager` as both `ABC` and `Protocol` (it's on the type-checker allowlist), so a Protocol can legally inherit from it. See: https://github.com/python/typeshed/blob/main/stdlib/contextlib.pyi - Base order matters: the ABC must come before `Protocol` for C3 linearization to succeed. - Generic args `[InjectionHandle, None]`: `_T_co` = what `__aenter__` returns, `_ExitT_co` = `None` (we never suppress exceptions). - `__aenter__ -> Self` is kept as a covariant refinement of the parent's `_T_co`; subclasses get their precise self-type. - `__aexit__` aligned with stdlib conventions: positional-only `/` marker and canonical `exc_value`/`traceback` parameter names.

Use exc_value/traceback parameter names and add positional-only / marker, matching the InjectionHandle.__aexit__ signature (typeshed/contextlib.pyi). Cosmetic — ty accepts both forms; the asymmetry was just confusing for readers. Addresses PR microsoft#86 review.

InjectionHandle is not in rampart.attacks.__all__ — it was only importable from there as an incidental runtime side-effect of the new isinstance check in Attacks.xpia. Import from rampart.core.injection directly so the test doesn't depend on a re-export that could quietly disappear. Addresses PR microsoft#86 review.

The `if found is True` check was a behavioral regression: callable targets can return any truthy value (e.g., `re.Match` from `lambda t: re.search(pat, t)`), and `Match is True` is False. That caused silent NOT_DETECTED where the previous `if found:` correctly returned DETECTED — a false negative in a safety detector. ty doesn't require `is True` either: after the isinstance reorder, `found` is statically `bool`, so plain `if found:` type-checks fine. The isinstance reordering above is kept. Addresses PR microsoft#86 review.

Two stragglers from the pyright->ty migration: the 'tooling' bullet (line 35) describing strict-mode Pyright, and the pre-commit summary (line 160) naming Pyright as the type checker. Both now reference ty. Not addressing the .gitignore portion of the review comment: leaving `pyrightconfig.json` ignored is intentional (default from the Python gitignore template). Addresses PR microsoft#86 review.

…-to-ty

Remove the [[tool.ty.overrides]] block that silenced invalid-argument-type across all of tests/**. That was too broad — it hid genuine wrong-arg-type bugs anywhere in the suite, not just the intentional negative tests. Two narrow categories of fix replace it: - MagicMock passed to a concrete (non-Protocol) param: cast at the call site to the expected type (pytest.Config / TerminalReporter / pytest.Session). The variable stays a MagicMock so .assert_*() and attribute mutation still work. MagicMock already satisfies Protocol params structurally (via __getattr__ -> Any), so only concrete-class params needed this. - Intentional wrong types asserting a runtime error (None config fields, 'not a function', NotASink) inside pytest.raises: per-line # ty: ignore[invalid-argument-type]. One non-mock case (register_default_handler_factory(lambda: [handler])) failed on list invariance, not a mock — fixed with a typed local list[ExecutionEventHandler] instead of a cast or ignore. Verification: uv run ty check clean; ruff clean; 491 passed, 5 skipped.

…ocol stubs `@abstractmethod` on a plain (non-ABC) class is a runtime no-op, and these classes are instantiated immediately for `isinstance(..., Protocol)` checks. The decorator read as the opposite of intent — it was only there to quiet the type checker about the `...` body not returning `Session`. `raise NotImplementedError` is clearer, works with any return type, and drops the now -unused `abc.abstractmethod` import. Applied in both the conforming-adapter stub (test_adapter.py) and the structural-subtyping stub (test_protocols.py). Addresses PR microsoft#86 review (comment 3432228031).

spencrr added 5 commits June 16, 2026 16:14

fixup how type narrowing is handled to avoid ignore

ae73c6c

reorder type narrow to avoid cast

157ef50

add spec to mock to ensure Attack.__init__ narrows properly

eb34f8e

spencrr changed the title ~~Dev/spencrr/migrate to ty~~ [META]: Migrate pyright to ty Jun 17, 2026

spencrr marked this pull request as ready for review June 17, 2026 00:12

spencrr requested a review from a team June 17, 2026 00:12

nina-msft reviewed Jun 18, 2026

View reviewed changes

spencrr added 8 commits June 18, 2026 14:04

uv lock update

85a2fb8

Merge remote-tracking branch 'upstream/main' into dev/spencrr/migrate…

cab9ab5

…-to-ty

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[META]: Migrate pyright to ty#86

[META]: Migrate pyright to ty#86
spencrr wants to merge 13 commits into
microsoft:mainfrom
spencrr:dev/spencrr/migrate-to-ty

spencrr commented Jun 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

spencrr commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Typing fixes prompted by ty

Test changes

Docs

Breaking changes

Checklist

Verification

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

spencrr commented Jun 17, 2026 •

edited

Loading