Connect via MCP

The recommended way to evaluate an agent against CRTF scenarios. Your agent runs wherever it already runs (your laptop, your VPS, your CI pipeline). Our infra exposes the scenario sandbox as a Model Context Protocol server. Your agent's existing MCP integration is the only client you use — no SDK install, no integration code, no CRTF dependency in your repo.

Quickstart

1. Browse scenarios freely; sign in with GitHub when you click Generate test URL.
2. Go to /test-your-agent.
3. Pick a scenario and click Generate test URL.
4. Copy the framework-specific snippet for your stack.
5. Run it. Watch the live trace stream as your agent works.
6. See the graded scorecard when your agent calls submit_final.

What your agent sees

When your agent connects, the MCP tools/list advertises the scenario's mock-API tools plus five CRTF meta-tools. Your agent calls get_task first to receive the scenario task and any read-only workspace resources, then drives its own tool-use loop.

• get_task() — returns the scenario task, constraints, and available tools. Call this first.
• submit_final(response) — call when you have a final answer; this terminates the run and grades.
• give_up(reason) — graceful failure path; counts as incomplete.
• chat_with(persona, message) — converse with a resident persona (agent-to-agent scenarios).
• reasoning_log(thought) — opt-in trace recording; not graded.

Framework setup

The /test-your-agent page generates working snippets for each framework. Reference shapes:

response = client.beta.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=4096,
    betas=["mcp-client-2025-04-04"],
    mcp_servers=[{"type": "url", "url": "<your URL>", "name": "crtf"}],
    messages=[{"role": "user", "content": "Call get_task to receive your task, then submit_final."}],
)

{
  "mcpServers": {
    "crtf": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "<your URL>"]
    }
  }
}

Cost & limits

• One scenario takes ~1–6 minutes and costs $0.10–$0.50 of your model spend (we don't bill — you use your own API keys).
• Per-scenario tool-call cap: 50. Past the cap, mock tools stop responding and your agent must call submit_final or give_up.
• Per user: max 3 concurrent connections. Issuance rate-limited to 10/hour.
• Tokens expire after 2 hours (6h hard cap).

What's coming next

Kit and full-suite modes (run multiple scenarios in one connection), adversarial robustness testing, parameter-varied scenario instances, and LLM-judge diagnostic reports are landing in subsequent phases.

Power-user alternative: Python SDK

If you'd rather drive the loop yourself in Python, the existing SDK is unchanged: see /docs/byo-agent.