paperclipai/paperclip · error · Error

Claude Managed evals require exact model claude-sonnet-5

Error message

Claude Managed evals require exact model claude-sonnet-5

What it means

Eval sessions pin the model per provider for reproducibility. When provider is "claude_managed", parseEvalSessionRequest requires request.model to be exactly "claude-sonnet-5"; any other value (including aliases or other Claude versions) throws this error.

Source

Thrown at packages/paperclip-runner/src/cli/eval-session-contract.ts:290

    );
  }
  if (input.includeCollaborationModeInstructions === false) {
    throw new Error(
      "current CapabilityLiveSessionService requires collaboration-mode instructions",
    );
  }

  const runnerd = object(input.runnerd, "request.runnerd");
  const digest = text(runnerd.sha256, "request.runnerd.sha256");
  if (!/^(?:sha256:)?[a-f0-9]{64}$/i.test(digest)) {
    throw new Error("request.runnerd.sha256 must be a SHA-256 digest");
  }
  const limits = object(input.limits, "request.limits");
  const sessionInput = object(input.session, "request.session");
  const session = sessionInput as unknown as CreateCapabilityLiveSessionInput;
  const model = text(input.model, "request.model");
  if (provider === "claude_managed" && model !== "claude-sonnet-5") {
    throw new Error("Claude Managed evals require exact model claude-sonnet-5");
  }
  if (
    provider === "aws_agentcore" &&
    model !== "global.anthropic.claude-sonnet-4-6"
  ) {
    throw new Error(
      "AWS AgentCore evals require exact model global.anthropic.claude-sonnet-4-6",
    );
  }
  if (sessionInput.provider !== undefined && sessionInput.provider !== provider) {
    throw new Error("request.session.provider must match request.provider");
  }
  if (
    session.requestedModel !== undefined &&
    session.requestedModel !== model
  ) {
    throw new Error("request.session.requestedModel must match request.model");
  }

View on GitHub (pinned to 01ad858492)

Solutions

1. Set request.model to exactly "claude-sonnet-5" for claude_managed evals.
2. Remove or correct a stale model env var/CLI override feeding the request builder.
3. If another model is intended, switch provider to a non-claude_managed provider that permits it (aws_agentcore requires global.anthropic.claude-sonnet-4-6).

Example fix

// before
{ "provider": "claude_managed", "model": "claude-sonnet-4-5", ... }
// after
{ "provider": "claude_managed", "model": "claude-sonnet-5", ... }

const REQUIRED = { claude_managed: "claude-sonnet-5", aws_agentcore: "global.anthropic.claude-sonnet-4-6" };
if (REQUIRED[req.provider] && req.model !== REQUIRED[req.provider]) throw new Error(`provider ${req.provider} requires model ${REQUIRED[req.provider]}`);

function isPinnedClaudeManagedRequest(input) { return input.provider !== "claude_managed" || input.model === "claude-sonnet-5"; }

try { return parseEvalSessionRequest(raw); } catch (e) { if (String(e.message).includes("require exact model")) { raw.model = "claude-sonnet-5"; return parseEvalSessionRequest(raw); } throw e; }

When it happens

Trigger: Calling parseEvalSessionRequest with provider "claude_managed" and model set to anything other than "claude-sonnet-5", e.g. "claude-sonnet-4-5", "claude-3-7-sonnet", a dated snapshot id, or an environment-default model.

Common situations: Reusing a request template written for a different provider/model; a config where model comes from an env var or CLI flag that defaults to another model; upgrading models in shared config without updating claude_managed eval entries; typos in the model id.

Related errors

• ${label} is not a regular file at ${canonical}.
• AWS AgentCore evals require exact model global.anthropic.cla
• request.session.provider must match request.provider
• request.session.requestedModel must match request.model
• Live roster ${rosterId} has no cases

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-02). Data as JSON: /api/errors/13c4dfc318da0ab0. Report an issue: GitHub.