{"record":{"id":"e8c5468838d306d4","repo":"paperclipai/paperclip","slug":"public-replay-contains-credential-or-private-refer","errorCode":null,"errorMessage":"Public replay contains credential or private reference material","messagePattern":"Public replay contains credential or private reference material","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"critical","filePath":"packages/paperclip-runner/scripts/public-eval-viewer.mjs","lineNumber":150,"sourceCode":"      check,\n      \"id kind passed detail evidenceRefs title description definition anchor\",\n    );\n    fields(check.definition, \"id kind\");\n    fields(check.anchor, \"kind id\");\n    if (check.evidenceRefs.length)\n      throw new Error(\"Public replay contains raw evidence references\");\n  }\n  const visit = (value, key = \"\") => {\n    if (typeof value === \"string\") {\n      if (\n        /(?:sessionId|providerSessionId)$/i.test(key) &&\n        ![\"public-report\", \"unknown\", \"redacted\"].includes(value)\n      )\n        throw new Error(\"Public replay contains a private session identity\");\n      for (const pattern of SECRET_TEXT) {\n        pattern.lastIndex = 0;\n        if (pattern.test(value))\n          throw new Error(\n            \"Public replay contains credential or private reference material\",\n          );\n      }\n    } else if (value && typeof value === \"object\") {\n      for (const [name, child] of Object.entries(value)) {\n        if (\n          /^(?:managedProfile|acpxProfile|providerTrace|mockState|stateHistory|trace|environment|env|apiKey|accessToken|password|secret)$/i.test(\n            name,\n          ) &&\n          child != null\n        )\n          throw new Error(\"Public replay contains a private field\");\n        visit(child, name);\n      }\n    }\n  };\n  visit(payload);\n  for (const section of [","sourceCodeStart":132,"sourceCodeEnd":168,"githubUrl":"https://github.com/paperclipai/paperclip/blob/01ad8584922b5d85292b1723cae71fa0d9b07a19/packages/paperclip-runner/scripts/public-eval-viewer.mjs#L132-L168","documentation":"During the recursive scan, visit() tests every string value against the SECRET_TEXT regex patterns imported from public-eval-chat.mjs (API keys, tokens, private URLs/paths, credentials). Any match causes \"Public replay contains credential or private reference material\" to be thrown, preventing secrets or credential-shaped strings from being published in the public report.","triggerScenarios":"Calling validatePublicChatPayload when any string anywhere in the payload matches a SECRET_TEXT pattern — e.g. an API key pasted into a user_message turn, a bearer token in a tool call detail, a private endpoint URL in a notice, or a key-like string in check details.","commonSituations":"Real credentials accidentally included in test fixtures or prompt content; tool output echoed tokens into agent messages; a new SECRET_TEXT pattern was added that now matches previously accepted text (e.g. after a key-format change); copy/pasted environment values in scenario descriptions.","solutions":["Find the matching string via the validation error context and remove/replace it in the source replay, then regenerate the payload.","Scrub or redact credential-like text in the projection step (mask with \"[redacted]\") before validation.","Fix the fixture/prompt so real secrets never enter runs; use placeholder credentials in tests.","If a legitimate string is newly flagged, review whether SECRET_TEXT in public-eval-chat.mjs needs refinement — do so deliberately, not to leak secrets."],"exampleFix":"// before\ndetail: \"auth failed for key sk-live-9f2k...\";\n// after\ndetail: \"auth failed for key [redacted]\";","handlingStrategy":"validation","validationCode":"import { SECRET_TEXT } from \"./public-eval-chat.mjs\";\nconst hasSecret = (s) => SECRET_TEXT.some((p) => { p.lastIndex = 0; return p.test(s); });\nJSON.stringify(payload).match(/[^\"]+/g)?.forEach((s) => { if (hasSecret(s)) console.warn(\"secret-like string present\"); });","typeGuard":"const containsSecret = (s) => SECRET_TEXT.some((p) => { p.lastIndex = 0; return p.test(s); });","tryCatchPattern":"try {\n  validatePublicChatPayload(payload);\n} catch (err) {\n  if (err.message === \"Public replay contains credential or private reference material\") {\n    throw new Error(\"A string in the replay matches SECRET_TEXT; scrub the source replay and regenerate\");\n  }\n  throw err;\n}","preventionTips":["Never use real credentials in fixtures or prompts; use obviously fake placeholders","Scrub tool output for token-shaped strings before it enters a replay","Run the SECRET_TEXT scan on internal payloads in CI so leaks surface before projection","When SECRET_TEXT patterns change, re-scan all retained fixtures"],"tags":["security","secrets","data-leak","validation"],"backgroundTag":"schema-validation-failed","analyzedSha":"01ad8584922b5d85292b1723cae71fa0d9b07a19","analyzedAt":"2026-09-10T03:14:50.855Z","contentChangedAt":"2026-09-10T03:14:50.855Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}