{"record":{"id":"eedb6866557284a2","repo":"paperclipai/paperclip","slug":"chat-evalbook-must-not-mix-in-legacy-plain-attempt","errorCode":null,"errorMessage":"Chat Evalbook must not mix in legacy plain attempt pages","messagePattern":"Chat Evalbook must not mix in legacy plain attempt pages","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/paperclip-runner/scripts/publish-runner-protocol-eval-history.mjs","lineNumber":228,"sourceCode":"        ...clean.split(\"/\"),\n      );\n      const rel = relative(root, target).split(sep).join(\"/\");\n      if (\n        !isPublicProtocolEvalPath(rel) ||\n        !(await lstat(target).catch(() => null))?.isFile()\n      ) {\n        throw new Error(\n          `Public report contains a broken or unsafe link in ${file}: ${href}`,\n        );\n      }\n    }\n  }\n  if (viewer) {\n    for (const file of viewer.files.keys())\n      if (!files.includes(file))\n        throw new Error(`Missing public viewer asset: ${file}`);\n    if (files.some((file) => /^attempts\\/[^/]+\\.html$/.test(file)))\n      throw new Error(\n        \"Chat Evalbook must not mix in legacy plain attempt pages\",\n      );\n  }\n  const campaign = await loadObject(join(root, \"campaign.json\"));\n  if (\n    campaign.schema !== \"paperclip.runner-protocol-eval.campaign/v1\" ||\n    !SAFE_CAMPAIGN.test(String(campaign.campaignId ?? \"\"))\n  ) {\n    throw new Error(\"Public report campaign metadata is invalid\");\n  }\n  return { files, campaign };\n}\n\nexport async function createProtocolEvalBundleManifest(\n  reportRoot,\n  campaignId,\n  { viewerRoot } = {},\n) {","sourceCodeStart":210,"sourceCodeEnd":246,"githubUrl":"https://github.com/paperclipai/paperclip/blob/01ad8584922b5d85292b1723cae71fa0d9b07a19/packages/paperclip-runner/scripts/publish-runner-protocol-eval-history.mjs#L210-L246","documentation":"The public protocol-eval publisher validates that a report intended for public hosting uses only the current Chat Evalbook format. When the report contains the new rich chat attempt pages (attempts/<id>/index.html), the publisher loads the trusted viewer build, and legacy flat attempt pages (attempts/<id>.html) are forbidden so the published site never mixes old and new attempt renderings. This guard runs at the end of validatePublicProtocolEvalReport after viewer assets are checked.","triggerScenarios":"Calling validatePublicProtocolEvalReport or publishProtocolEvalHistory with a reportRoot whose files include both attempts/<id>/index.html (chat-rich attempts, which force viewer loading) and legacy attempts/<id>.html pages from an older report generator.","commonSituations":"Stale report directory left over from a previous version of the eval generator merged with newly generated chat attempt pages; a report rebuilt incrementally where old attempt HTML files were never cleaned out; hand-copying files from an old public report into a new one.","solutions":["Clean the report output directory and regenerate it fully from the current eval runner so only attempts/<id>/index.html pages exist","Delete any legacy attempts/<id>.html files (flat files, not directories) from reportRoot before publishing","Check the generator version that produced the report; regenerate with a toolchain that no longer emits plain attempt pages","If legacy pages are needed, publish them as a separate campaign rather than mixing them into the chat Evalbook"],"exampleFix":"// before: stale legacy file remains in report\n// attempts/gha-123-45.html  (old flat page)\n// attempts/gha-123-45/index.html (new rich page)\n\n// after: remove the legacy flat page before publishing\nrm reports/runner-protocol-eval-public-report/attempts/*.html","handlingStrategy":"validation","validationCode":"const files = fs.readdirSync(reportRoot, { recursive: true });\nconst hasLegacy = files.some((f) => /^attempts\\/[^/]+\\.html$/.test(f.replaceAll(\"\\\\\", \"/\")));\nif (hasLegacy) throw new Error(\"remove legacy attempts/<id>.html pages before publishing\");","typeGuard":"const isLegacyAttemptPage = (f) => /^attempts\\/[^/]+\\.html$/.test(f);\nconst hasLegacyAttempts = (files) => files.some(isLegacyAttemptPage);","tryCatchPattern":"try {\n  await publishProtocolEvalHistory({ reportRoot, destination, viewerRoot });\n} catch (e) {\n  if (String(e.message).includes(\"legacy plain attempt pages\")) {\n    for (const f of listLegacyAttemptPages(reportRoot)) fs.rmSync(path.join(reportRoot, f));\n    // retry once\n  } else throw e;\n}","preventionTips":["Wipe the report output directory at the start of every generation instead of writing into it incrementally","Never copy files from older public reports into a new report directory","Pin the eval generator version and regenerate reports wholesale when upgrading"],"tags":["validation","publishing","legacy-format"],"backgroundTag":"incompatible-source-type","analyzedSha":"01ad8584922b5d85292b1723cae71fa0d9b07a19","analyzedAt":"2026-09-10T03:14:50.855Z","contentChangedAt":"2026-09-10T03:14:50.855Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}