Add nightly application validation for top kernels

[msaroufim]

Summary

• add nightly application validation for the best submission from each of the current top 10 users
• keep the problem contract minimal: version plus a problem-owned script
• trust entries that already reached the top 10; application validation does not run KernelGuard again
• run at 22:00 America/Los_Angeles with an atomic daily claim and at most two Modal jobs concurrently
• persist shape coverage and synchronized-wall speedup for the exact submission, GPU, and contract version
• expose the existing sweep through both a manual admin endpoint and kernelbot-admin validate-top10

Behavior

The candidate query first selects each user's fastest eligible submission, ranks
those per-user winners, and then validates up to 10 distinct users. Validation is
an annotation only: it never removes or reorders leaderboard entries.

The Cholesky workload requires each shape to produce a valid factor, converge in
the mini-training loop, and be at least as fast as the PyTorch baseline.
Implementations may use custom kernels, PyTorch, or a hybrid of both; fallback
is not a validation gate. Kernelboard can render VALIDATED, X/Y VALIDATED,
or VALIDATION ERROR.

Modal already consumes MODAL_ENVIRONMENT; this PR does not add a second
environment setting or custom lookup path. APPLICATION_VALIDATION_ENABLED=false
is the scheduler kill switch.

Validation

• uv run ruff check . --exclude examples/ --line-length 120 --fix
• PYTEST_MODAL_ENV=pytest uv run pytest tests/ -v — 257 passed, 7 skipped
• fresh current-tip end-to-end run with migrated Postgres, local Kernelbot API,
  the cholesky-validation-debug Modal environment, and local Kernelboard API/UI
• the local API was started without KERNELGUARD_ENABLED, confirming there is
  no second KernelGuard pass
• latest attached Seraphim B200 current-tip run: 5/8 shapes validated and 2.82x
  geomean synchronized-wall speedup
• CLI coverage verifies enqueue, blocking --wait, URL encoding, and required
  admin authentication

Coordinated rollout

Merge/deploy in this order:

1. this PR
2. Add Cholesky mini-training validation contract reference-kernels#168
3. Show per-submission application validation badges kernelboard#261

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/libkernelbot
application_validation.py					26, 42-45, 51-61, 64-71, 81, 89, 105-110, 113-115, 138, 141, 152, 160, 166, 194-214, 233, 238, 293-301
leaderboard_db.py					926, 958, 976
task.py					37, 247
utils.py
validation_runtime.py					37, 49, 54, 70, 72, 74, 111-112
Project Total

This report was generated by python-coverage-comment-action