{"record":{"id":"13c4dfc318da0ab0","repo":"paperclipai/paperclip","slug":"claude-managed-evals-require-exact-model-claude-so","errorCode":null,"errorMessage":"Claude Managed evals require exact model claude-sonnet-5","messagePattern":"Claude Managed evals require exact model claude-sonnet-5","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/paperclip-runner/src/cli/eval-session-contract.ts","lineNumber":290,"sourceCode":"    );\n  }\n  if (input.includeCollaborationModeInstructions === false) {\n    throw new Error(\n      \"current CapabilityLiveSessionService requires collaboration-mode instructions\",\n    );\n  }\n\n  const runnerd = object(input.runnerd, \"request.runnerd\");\n  const digest = text(runnerd.sha256, \"request.runnerd.sha256\");\n  if (!/^(?:sha256:)?[a-f0-9]{64}$/i.test(digest)) {\n    throw new Error(\"request.runnerd.sha256 must be a SHA-256 digest\");\n  }\n  const limits = object(input.limits, \"request.limits\");\n  const sessionInput = object(input.session, \"request.session\");\n  const session = sessionInput as unknown as CreateCapabilityLiveSessionInput;\n  const model = text(input.model, \"request.model\");\n  if (provider === \"claude_managed\" && model !== \"claude-sonnet-5\") {\n    throw new Error(\"Claude Managed evals require exact model claude-sonnet-5\");\n  }\n  if (\n    provider === \"aws_agentcore\" &&\n    model !== \"global.anthropic.claude-sonnet-4-6\"\n  ) {\n    throw new Error(\n      \"AWS AgentCore evals require exact model global.anthropic.claude-sonnet-4-6\",\n    );\n  }\n  if (sessionInput.provider !== undefined && sessionInput.provider !== provider) {\n    throw new Error(\"request.session.provider must match request.provider\");\n  }\n  if (\n    session.requestedModel !== undefined &&\n    session.requestedModel !== model\n  ) {\n    throw new Error(\"request.session.requestedModel must match request.model\");\n  }","sourceCodeStart":272,"sourceCodeEnd":308,"githubUrl":"https://github.com/paperclipai/paperclip/blob/01ad8584922b5d85292b1723cae71fa0d9b07a19/packages/paperclip-runner/src/cli/eval-session-contract.ts#L272-L308","documentation":"Eval sessions pin the model per provider for reproducibility. When provider is \"claude_managed\", parseEvalSessionRequest requires request.model to be exactly \"claude-sonnet-5\"; any other value (including aliases or other Claude versions) throws this error.","triggerScenarios":"Calling parseEvalSessionRequest with provider \"claude_managed\" and model set to anything other than \"claude-sonnet-5\", e.g. \"claude-sonnet-4-5\", \"claude-3-7-sonnet\", a dated snapshot id, or an environment-default model.","commonSituations":"Reusing a request template written for a different provider/model; a config where model comes from an env var or CLI flag that defaults to another model; upgrading models in shared config without updating claude_managed eval entries; typos in the model id.","solutions":["Set request.model to exactly \"claude-sonnet-5\" for claude_managed evals.","Remove or correct a stale model env var/CLI override feeding the request builder.","If another model is intended, switch provider to a non-claude_managed provider that permits it (aws_agentcore requires global.anthropic.claude-sonnet-4-6)."],"exampleFix":"// before\n{ \"provider\": \"claude_managed\", \"model\": \"claude-sonnet-4-5\", ... }\n// after\n{ \"provider\": \"claude_managed\", \"model\": \"claude-sonnet-5\", ... }","handlingStrategy":"validation","validationCode":"const REQUIRED = { claude_managed: \"claude-sonnet-5\", aws_agentcore: \"global.anthropic.claude-sonnet-4-6\" };\nif (REQUIRED[req.provider] && req.model !== REQUIRED[req.provider]) throw new Error(`provider ${req.provider} requires model ${REQUIRED[req.provider]}`);","typeGuard":"function isPinnedClaudeManagedRequest(input) { return input.provider !== \"claude_managed\" || input.model === \"claude-sonnet-5\"; }","tryCatchPattern":"try { return parseEvalSessionRequest(raw); } catch (e) { if (String(e.message).includes(\"require exact model\")) { raw.model = \"claude-sonnet-5\"; return parseEvalSessionRequest(raw); } throw e; }","preventionTips":["Pin the model constant per provider in shared config instead of passing free-form strings.","Ignore model env overrides in eval request builders.","Add a unit test asserting each provider's pinned model id."],"tags":["validation","model-pin","config"],"backgroundTag":"model-not-allowed-for-provider","analyzedSha":"01ad8584922b5d85292b1723cae71fa0d9b07a19","analyzedAt":"2026-09-02T18:44:00.616Z","contentChangedAt":"2026-09-02T18:44:00.616Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}