{"record":{"id":"0496a2405fc6ba01","repo":"paperclipai/paperclip","slug":"aws-agentcore-evals-require-exact-model-global-ant","errorCode":null,"errorMessage":"AWS AgentCore evals require exact model global.anthropic.claude-sonnet-4-6","messagePattern":"AWS AgentCore evals require exact model global\\.anthropic\\.claude-sonnet-4-6","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/paperclip-runner/src/cli/eval-session-contract.ts","lineNumber":296,"sourceCode":"  }\n\n  const runnerd = object(input.runnerd, \"request.runnerd\");\n  const digest = text(runnerd.sha256, \"request.runnerd.sha256\");\n  if (!/^(?:sha256:)?[a-f0-9]{64}$/i.test(digest)) {\n    throw new Error(\"request.runnerd.sha256 must be a SHA-256 digest\");\n  }\n  const limits = object(input.limits, \"request.limits\");\n  const sessionInput = object(input.session, \"request.session\");\n  const session = sessionInput as unknown as CreateCapabilityLiveSessionInput;\n  const model = text(input.model, \"request.model\");\n  if (provider === \"claude_managed\" && model !== \"claude-sonnet-5\") {\n    throw new Error(\"Claude Managed evals require exact model claude-sonnet-5\");\n  }\n  if (\n    provider === \"aws_agentcore\" &&\n    model !== \"global.anthropic.claude-sonnet-4-6\"\n  ) {\n    throw new Error(\n      \"AWS AgentCore evals require exact model global.anthropic.claude-sonnet-4-6\",\n    );\n  }\n  if (sessionInput.provider !== undefined && sessionInput.provider !== provider) {\n    throw new Error(\"request.session.provider must match request.provider\");\n  }\n  if (\n    session.requestedModel !== undefined &&\n    session.requestedModel !== model\n  ) {\n    throw new Error(\"request.session.requestedModel must match request.model\");\n  }\n  if (session.acpxAgent === \"pi\") {\n    throw new Error(\"The Pi ACPX profile is not available\");\n  }\n\n  return {\n    schema: EVAL_SESSION_REQUEST_SCHEMA,","sourceCodeStart":278,"sourceCodeEnd":314,"githubUrl":"https://github.com/paperclipai/paperclip/blob/01ad8584922b5d85292b1723cae71fa0d9b07a19/packages/paperclip-runner/src/cli/eval-session-contract.ts#L278-L314","documentation":"parseEvalSessionRequest enforces that eval-session requests using provider 'aws_agentcore' pin the model to exactly 'global.anthropic.claude-sonnet-4-6' (eval-session-contract.ts:283-290). This fail-closed guard exists because the eval harness only qualifies that single AgentCore model ID; any other model string would produce non-comparable eval results or an AWS-side invocation failure. The check compares request.model literally, so even case or region-prefix variants are rejected.","triggerScenarios":"Calling parseEvalSessionRequest (directly or via the eval-session CLI/runner 'request' path) with request.provider='aws_agentcore' and request.model set to anything other than 'global.anthropic.claude-sonnet-4-6', e.g. 'anthropic.claude-sonnet-4-5', 'us.anthropic.claude-sonnet-4-6', or an omitted/mistyped model ID.","commonSituations":"Copy-pasting a model ID from AWS docs that includes a regional prefix (us./eu./apac.), reusing a request JSON authored for claude_managed or codex providers without updating the model, upgrading to a newer Sonnet version and assuming the harness accepts it, or omitting request.model and getting a schema error surfaced as this mismatch.","solutions":["Set request.model to exactly \"global.anthropic.claude-sonnet-4-6\" in the eval-session request JSON.","If you intended a different model, switch request.provider away from aws_agentcore (e.g. acpx or claude_managed) where that model is supported.","If you need a newer AgentCore model, update the qualifier/contract (expected model constant and evals qualification) rather than bypassing the check.","Regenerate the request file from a known-good template for aws_agentcore evals and re-run."],"exampleFix":"// before\n{\n  \"provider\": \"aws_agentcore\",\n  \"model\": \"us.anthropic.claude-sonnet-4-6\",\n  ...\n}\n// after\n{\n  \"provider\": \"aws_agentcore\",\n  \"model\": \"global.anthropic.claude-sonnet-4-6\",\n  ...\n}","handlingStrategy":"validation","validationCode":"const AGENTCORE_MODEL = \"global.anthropic.claude-sonnet-4-6\";\nfunction assertAgentCoreModel(req: { provider?: string; model?: string }): void {\n  if (req.provider === \"aws_agentcore\" && req.model !== AGENTCORE_MODEL) {\n    throw new Error(`aws_agentcore evals require model ${AGENTCORE_MODEL}, got ${req.model}`);\n  }\n}","typeGuard":"function isAgentCoreRequest(req: { provider?: string; model?: string }): boolean {\n  return req.provider !== \"aws_agentcore\" || req.model === \"global.anthropic.claude-sonnet-4-6\";\n}","tryCatchPattern":"try {\n  const parsed = parseEvalSessionRequest(rawRequest);\n  // proceed\n} catch (err) {\n  if (err instanceof Error && err.message.includes(\"AWS AgentCore evals require exact model\")) {\n    // fix request.model to \"global.anthropic.claude-sonnet-4-6\" and retry once\n  } else throw err;\n}","preventionTips":["Keep a single MODEL constant per provider and derive both request.model and session.requestedModel from it.","Never hand-copy model IDs from AWS docs — regional prefixes (us./eu.) will fail the exact match.","Validate the request JSON against the eval-session contract before invoking the CLI.","Pin the model in your eval config template so upgrades are an explicit change."],"tags":["validation","config","evals","aws-agentcore"],"backgroundTag":"invalid-model-id","analyzedSha":"01ad8584922b5d85292b1723cae71fa0d9b07a19","analyzedAt":"2026-09-02T18:44:00.616Z","contentChangedAt":"2026-09-02T18:44:00.616Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}