paperclipai/paperclip · error · Error

AWS AgentCore evals require exact model global.anthropic.cla

Error message

AWS AgentCore evals require exact model global.anthropic.claude-sonnet-4-6

What it means

parseEvalSessionRequest enforces that eval-session requests using provider 'aws_agentcore' pin the model to exactly 'global.anthropic.claude-sonnet-4-6' (eval-session-contract.ts:283-290). This fail-closed guard exists because the eval harness only qualifies that single AgentCore model ID; any other model string would produce non-comparable eval results or an AWS-side invocation failure. The check compares request.model literally, so even case or region-prefix variants are rejected.

Source

Thrown at packages/paperclip-runner/src/cli/eval-session-contract.ts:296

  }

  const runnerd = object(input.runnerd, "request.runnerd");
  const digest = text(runnerd.sha256, "request.runnerd.sha256");
  if (!/^(?:sha256:)?[a-f0-9]{64}$/i.test(digest)) {
    throw new Error("request.runnerd.sha256 must be a SHA-256 digest");
  }
  const limits = object(input.limits, "request.limits");
  const sessionInput = object(input.session, "request.session");
  const session = sessionInput as unknown as CreateCapabilityLiveSessionInput;
  const model = text(input.model, "request.model");
  if (provider === "claude_managed" && model !== "claude-sonnet-5") {
    throw new Error("Claude Managed evals require exact model claude-sonnet-5");
  }
  if (
    provider === "aws_agentcore" &&
    model !== "global.anthropic.claude-sonnet-4-6"
  ) {
    throw new Error(
      "AWS AgentCore evals require exact model global.anthropic.claude-sonnet-4-6",
    );
  }
  if (sessionInput.provider !== undefined && sessionInput.provider !== provider) {
    throw new Error("request.session.provider must match request.provider");
  }
  if (
    session.requestedModel !== undefined &&
    session.requestedModel !== model
  ) {
    throw new Error("request.session.requestedModel must match request.model");
  }
  if (session.acpxAgent === "pi") {
    throw new Error("The Pi ACPX profile is not available");
  }

  return {
    schema: EVAL_SESSION_REQUEST_SCHEMA,

View on GitHub (pinned to 01ad858492)

Solutions

1. Set request.model to exactly "global.anthropic.claude-sonnet-4-6" in the eval-session request JSON.
2. If you intended a different model, switch request.provider away from aws_agentcore (e.g. acpx or claude_managed) where that model is supported.
3. If you need a newer AgentCore model, update the qualifier/contract (expected model constant and evals qualification) rather than bypassing the check.
4. Regenerate the request file from a known-good template for aws_agentcore evals and re-run.

Example fix

// before
{
  "provider": "aws_agentcore",
  "model": "us.anthropic.claude-sonnet-4-6",
  ...
}
// after
{
  "provider": "aws_agentcore",
  "model": "global.anthropic.claude-sonnet-4-6",
  ...
}

const AGENTCORE_MODEL = "global.anthropic.claude-sonnet-4-6";
function assertAgentCoreModel(req: { provider?: string; model?: string }): void {
  if (req.provider === "aws_agentcore" && req.model !== AGENTCORE_MODEL) {
    throw new Error(`aws_agentcore evals require model ${AGENTCORE_MODEL}, got ${req.model}`);
  }
}

function isAgentCoreRequest(req: { provider?: string; model?: string }): boolean {
  return req.provider !== "aws_agentcore" || req.model === "global.anthropic.claude-sonnet-4-6";
}

try {
  const parsed = parseEvalSessionRequest(rawRequest);
  // proceed
} catch (err) {
  if (err instanceof Error && err.message.includes("AWS AgentCore evals require exact model")) {
    // fix request.model to "global.anthropic.claude-sonnet-4-6" and retry once
  } else throw err;
}

When it happens

Trigger: Calling parseEvalSessionRequest (directly or via the eval-session CLI/runner 'request' path) with request.provider='aws_agentcore' and request.model set to anything other than 'global.anthropic.claude-sonnet-4-6', e.g. 'anthropic.claude-sonnet-4-5', 'us.anthropic.claude-sonnet-4-6', or an omitted/mistyped model ID.

Common situations: Copy-pasting a model ID from AWS docs that includes a regional prefix (us./eu./apac.), reusing a request JSON authored for claude_managed or codex providers without updating the model, upgrading to a newer Sonnet version and assuming the harness accepts it, or omitting request.model and getting a schema error surfaced as this mismatch.

Related errors

• request.session.provider must match request.provider
• request.session.requestedModel must match request.model
• ${label} is not a regular file at ${canonical}.
• request.agentCoreProfile.eventExpiryDays must be 90
• Claude Managed evals require exact model claude-sonnet-5

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-02). Data as JSON: /api/errors/0496a2405fc6ba01. Report an issue: GitHub.