paperclipai/paperclip · critical · Error

Public replay contains credential or private reference mater

Error message

Public replay contains credential or private reference material

What it means

During the recursive scan, visit() tests every string value against the SECRET_TEXT regex patterns imported from public-eval-chat.mjs (API keys, tokens, private URLs/paths, credentials). Any match causes "Public replay contains credential or private reference material" to be thrown, preventing secrets or credential-shaped strings from being published in the public report.

Source

Thrown at packages/paperclip-runner/scripts/public-eval-viewer.mjs:150

      check,
      "id kind passed detail evidenceRefs title description definition anchor",
    );
    fields(check.definition, "id kind");
    fields(check.anchor, "kind id");
    if (check.evidenceRefs.length)
      throw new Error("Public replay contains raw evidence references");
  }
  const visit = (value, key = "") => {
    if (typeof value === "string") {
      if (
        /(?:sessionId|providerSessionId)$/i.test(key) &&
        !["public-report", "unknown", "redacted"].includes(value)
      )
        throw new Error("Public replay contains a private session identity");
      for (const pattern of SECRET_TEXT) {
        pattern.lastIndex = 0;
        if (pattern.test(value))
          throw new Error(
            "Public replay contains credential or private reference material",
          );
      }
    } else if (value && typeof value === "object") {
      for (const [name, child] of Object.entries(value)) {
        if (
          /^(?:managedProfile|acpxProfile|providerTrace|mockState|stateHistory|trace|environment|env|apiKey|accessToken|password|secret)$/i.test(
            name,
          ) &&
          child != null
        )
          throw new Error("Public replay contains a private field");
        visit(child, name);
      }
    }
  };
  visit(payload);
  for (const section of [

View on GitHub (pinned to 01ad858492)

Solutions

Find the matching string via the validation error context and remove/replace it in the source replay, then regenerate the payload.
Scrub or redact credential-like text in the projection step (mask with "[redacted]") before validation.
Fix the fixture/prompt so real secrets never enter runs; use placeholder credentials in tests.
If a legitimate string is newly flagged, review whether SECRET_TEXT in public-eval-chat.mjs needs refinement — do so deliberately, not to leak secrets.

Example fix

// before
detail: "auth failed for key sk-live-9f2k...";
// after
detail: "auth failed for key [redacted]";

Defensive patterns

Strategy: validation

Validate before calling

import { SECRET_TEXT } from "./public-eval-chat.mjs";
const hasSecret = (s) => SECRET_TEXT.some((p) => { p.lastIndex = 0; return p.test(s); });
JSON.stringify(payload).match(/[^"]+/g)?.forEach((s) => { if (hasSecret(s)) console.warn("secret-like string present"); });

Type guard

const containsSecret = (s) => SECRET_TEXT.some((p) => { p.lastIndex = 0; return p.test(s); });

Try / catch

try {
  validatePublicChatPayload(payload);
} catch (err) {
  if (err.message === "Public replay contains credential or private reference material") {
    throw new Error("A string in the replay matches SECRET_TEXT; scrub the source replay and regenerate");
  }
  throw err;
}

Prevention

Never use real credentials in fixtures or prompts; use obviously fake placeholders
Scrub tool output for token-shaped strings before it enters a replay
Run the SECRET_TEXT scan on internal payloads in CI so leaks surface before projection
When SECRET_TEXT patterns change, re-scan all retained fixtures

When it happens

Trigger: Calling validatePublicChatPayload when any string anywhere in the payload matches a SECRET_TEXT pattern — e.g. an API key pasted into a user_message turn, a bearer token in a tool call detail, a private endpoint URL in a notice, or a key-like string in check details.

Common situations: Real credentials accidentally included in test fixtures or prompt content; tool output echoed tokens into agent messages; a new SECRET_TEXT pattern was added that now matches previously accepted text (e.g. after a key-format change); copy/pasted environment values in scenario descriptions.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-10). Data as JSON: /api/errors/e8c5468838d306d4. Report an issue: GitHub.