Skip to content

fix: fail closed when LLM provider is unavailable#291

Draft
tuodijihua wants to merge 1 commit into
NVIDIA:mainfrom
tuodijihua:agent/fail-closed-unavailable-provider
Draft

fix: fail closed when LLM provider is unavailable#291
tuodijihua wants to merge 1 commit into
NVIDIA:mainfrom
tuodijihua:agent/fail-closed-unavailable-provider

Conversation

@tuodijihua

Copy link
Copy Markdown

Summary

Prevent a requested deep scan from returning SAFE when the configured LLM provider is unavailable but analyzer nodes record mixed success/failure telemetry.

When this condition is detected, the report now:

  • records an explicit degradation notice
  • floors an otherwise SAFE recommendation to CAUTION
  • preserves existing scoring and higher-severity recommendations

Fixes #290.

Root cause

report() previously inferred degradation only from aggregate call-log counts. Some analyzer nodes can catch a provider exception and still record successful fallback work, so the log may contain both successes and failures even though is_llm_available() is false. That prevented the existing all-calls-failed degradation path from activating.

This change combines the provider availability check with evidence of at least one real LLM call failure. The guard is intentionally narrow: it only applies when deep scanning was requested, the provider is unavailable, and the call log contains a failure.

User impact

Without this guard, automation can treat a static-only result as a successful deep scan and allow installation based on a SAFE recommendation. The new CAUTION floor makes the degradation visible to callers without overriding DANGER or BLOCK results.

Validation

  • uv run pytest tests/nodes/test_report.py -q — 61 passed
  • uv run ruff check src/skillspector/nodes/report.py tests/nodes/test_report.py — passed
  • git diff origin/main...HEAD --check — passed
  • Full suite in this environment: 1368 passed, 12 skipped, 6 xfailed, 7 failed
    • The seven failures are URL/SSRF tests caused by GitHub/GitLab resolving to private addresses in this environment.
    • The same focused URL/SSRF test set produces the same 7 failures on an unmodified worktree at the base commit.

Scope

This patch does not attempt to redesign analyzer telemetry. It adds a report-layer fail-closed guard and a regression test for the mixed-telemetry case.

Signed-off-by: tuodijihua <158809980+tuodijihua@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deep scan can report SAFE when provider is unavailable

1 participant