PickySteve

Picky about what he loads into context, including what he refuses to load.

https://github.com/user-attachments/assets/8750946b-36be-4c48-bf73-79513451d1f5

# from the repo root (uv 0.10+; on Windows the venv python is .venv/Scripts/python.exe — substitute it throughout) uv venv --python 3.11 .venv uv pip install --python .venv/bin/python -r requirements.txt # choose your model — local Ollama, OpenAI, Claude, OpenRouter, or any OpenAI-compatible endpoint .venv/bin/python -m pickysteve.setup # calibrate the reranker floor on the labeled set .venv/bin/python eval/calibrate.py # run one request .venv/bin/python -m pickysteve "review my Rust endpoint for security and REST design"

python -m pickysteve.connectors.install --list # see which of 18 agents are detected python -m pickysteve.connectors.install --all # wire every detected agent (backs up configs first)

SupportsClaude Code, Codex, Cursor, Windsurf, Cline, Roo Code, Gemini CLI, Qwen Code, Goose, OpenHands, GitHub Copilot, Kimi Code, OpenCode, ZeroClawvia MCP stdio, andAider, Hermes, OpenClaw, NanoClawvia an OpenAI-compatible proxy on:8077/v1. Full per-agent config snippets and the connectivity matrix are inINTEGRATIONS.md.

flowchart TD A[Request] --> B[Security Gate\nscan raw request] B -->|clean| C[Router\ncheap model → search query] B -->|injection| X1[Abort] C --> D[Retrieval\nBM25 + embeddings, RRF fused] D --> E[Security Gate\nscan every retrieved doc] E -->|clean| F[Rerank\ncross-encoder vs original request] E -->|poisoned| X2[Abort / drop candidate] F --> G[Floor + Dedupe\nbelow floor → clarify, don't guess] G --> H[Knowledge Graph\nconfused_with edges + distinguishers] H --> I[Judge\nLLM reads full skill bodies + KG notes] I --> J[Compat Check\nflag conflicts, don't merge] J --> K[Assembly\nnonce-wrapped untrusted-data boundary] K --> L[Execution\ncapable model does the work] L --> M[Log\nfull trace to logs/runs.jsonl]

Ten stages: gate, route, retrieve, gate again on retrieved content, rerank, floor/dedupe, knowledge-graph context, judge, compat-check, assemble, execute, log. The second gate pass scans every retrieved candidate, not just the user's request. Most similar projects skip that pass, and it is the highest-risk surface: a poisoned skill doc is attacker-controlled content sitting right next to your execution model.

Total Phase-1 dependencies:stackone-defender,rank-bm25,sentence-transformers,openai,numpy. That is the minimal set the spec prescribes.

Two decisions the spec left open (decided and documented)

- Retrieval unit (§2.3):each markdown file is one retrieval unit. A skill folder with several files (seeregistry/rag-architecture/) yields multiple units sharing askill_id. After reranking, units from the same skill collapse to the best one in assembly, so the execution model never receives three chunks of one skill.
- Gate policy on a poisoned retrieval (§2.1):defaultRETRIEVED_INJECTION_POLICY=abort. If a retrieved candidate trips the gate (high-risk), the whole request aborts. The documented alternative isdrop, which discards just that candidate and continues. For allowed-but- sanitized content, the pipeline uses the Tier-1-sanitized text downstream (defense in depth) and logs that sanitization happened.

Refinements after a 21-agent adversarial review

The first validation surfaced three failures. Fixing them, and adversarially reviewing the fixes, added these mechanisms. SeeFINDINGS.mdfor the full before/after.

- Tier-3 escalation (gate, request path only):a legitimate question about prompt injection was being blocked. The request gate now enables the defender's Tier-3 LLM hook over the gray band0.64, 0.85), just above the model's calibrated 0.64 block threshold. A cheap adjudicator can rescue a would-be block but never flip a would-be allow, while near-certain attacks (≥0.85) still hard-block without consulting it. Retrieved third-party content never escalates (strict gate).
- Multi-intent router with §2.4-safe rescue:the router emits sub-queries and retrieval unions across them for recall. Reranking stays governed by the original request (§2.4). Only a genuinely compound request (two or more distinct sub-intents) also maxes over its sub-queries, to surface a secondary intent the full-request score would bury.
- Relative-dominance gate:a secondary skill is kept only if it scores at leastDOMINANCE_RATIO(0.08) times the top skill. This keeps PickySteve picky instead of dumping marginal tag-alongs.
- Honest #13 fix:a correct skill that the reranker under-scored was fixed by enriching the skill doc with real symptom vocabulary, not by lowering the floor onto leaked data. The floor is recalibrated on a leakage-free labeled set with hard-negatives.

On a 40-request held-out accuracy set with no calibration overlap (TEST_REPORT.md):90% overall correct,100% top-1 accuracy (30/30 answerable),96.7% full recall,MRR 1.000,100% off-domain rejection(haiku/recipe requests correctly getno_confident_match).

Security, red-team detection(SECURITY_AUDIT.md,[TEST_REPORT.md):

- 180-payload corpus (129 attack / 51 benign, 14 evasion families):100% attack detection, zero bypassesafter hardening. The baseline was 86%.
- Separate 115-attack corpus:97.6%detection on the request path,96.5%on retrieved content, up from 87.1%. Benign false-positive rate held at0.0%throughout.
- On the adversarial 180-payload corpus, benign-allow rate is 61% (39% false-positive on deliberately tricky security-flavored prompts). On the real skill registry, false positives are0/43, verified by a startup-time warm pass the server refuses to serve without.

…