paperclipai/paperclip · error · Error

Chat Evalbook must not mix in legacy plain attempt pages

Error message

Chat Evalbook must not mix in legacy plain attempt pages

What it means

The public protocol-eval publisher validates that a report intended for public hosting uses only the current Chat Evalbook format. When the report contains the new rich chat attempt pages (attempts/<id>/index.html), the publisher loads the trusted viewer build, and legacy flat attempt pages (attempts/<id>.html) are forbidden so the published site never mixes old and new attempt renderings. This guard runs at the end of validatePublicProtocolEvalReport after viewer assets are checked.

Source

Thrown at packages/paperclip-runner/scripts/publish-runner-protocol-eval-history.mjs:228

        ...clean.split("/"),
      );
      const rel = relative(root, target).split(sep).join("/");
      if (
        !isPublicProtocolEvalPath(rel) ||
        !(await lstat(target).catch(() => null))?.isFile()
      ) {
        throw new Error(
          `Public report contains a broken or unsafe link in ${file}: ${href}`,
        );
      }
    }
  }
  if (viewer) {
    for (const file of viewer.files.keys())
      if (!files.includes(file))
        throw new Error(`Missing public viewer asset: ${file}`);
    if (files.some((file) => /^attempts\/[^/]+\.html$/.test(file)))
      throw new Error(
        "Chat Evalbook must not mix in legacy plain attempt pages",
      );
  }
  const campaign = await loadObject(join(root, "campaign.json"));
  if (
    campaign.schema !== "paperclip.runner-protocol-eval.campaign/v1" ||
    !SAFE_CAMPAIGN.test(String(campaign.campaignId ?? ""))
  ) {
    throw new Error("Public report campaign metadata is invalid");
  }
  return { files, campaign };
}

export async function createProtocolEvalBundleManifest(
  reportRoot,
  campaignId,
  { viewerRoot } = {},
) {

View on GitHub (pinned to 01ad858492)

Solutions

1. Clean the report output directory and regenerate it fully from the current eval runner so only attempts/<id>/index.html pages exist
2. Delete any legacy attempts/<id>.html files (flat files, not directories) from reportRoot before publishing
3. Check the generator version that produced the report; regenerate with a toolchain that no longer emits plain attempt pages
4. If legacy pages are needed, publish them as a separate campaign rather than mixing them into the chat Evalbook

Example fix

// before: stale legacy file remains in report
// attempts/gha-123-45.html  (old flat page)
// attempts/gha-123-45/index.html (new rich page)

// after: remove the legacy flat page before publishing
rm reports/runner-protocol-eval-public-report/attempts/*.html

const files = fs.readdirSync(reportRoot, { recursive: true });
const hasLegacy = files.some((f) => /^attempts\/[^/]+\.html$/.test(f.replaceAll("\\", "/")));
if (hasLegacy) throw new Error("remove legacy attempts/<id>.html pages before publishing");

const isLegacyAttemptPage = (f) => /^attempts\/[^/]+\.html$/.test(f);
const hasLegacyAttempts = (files) => files.some(isLegacyAttemptPage);

try {
  await publishProtocolEvalHistory({ reportRoot, destination, viewerRoot });
} catch (e) {
  if (String(e.message).includes("legacy plain attempt pages")) {
    for (const f of listLegacyAttemptPages(reportRoot)) fs.rmSync(path.join(reportRoot, f));
    // retry once
  } else throw e;
}

When it happens

Trigger: Calling validatePublicProtocolEvalReport or publishProtocolEvalHistory with a reportRoot whose files include both attempts/<id>/index.html (chat-rich attempts, which force viewer loading) and legacy attempts/<id>.html pages from an older report generator.

Common situations: Stale report directory left over from a previous version of the eval generator merged with newly generated chat attempt pages; a report rebuilt incrementally where old attempt HTML files were never cleaned out; hand-copying files from an old public report into a new one.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• Report campaign ID does not match publication target
• Invalid status '${String(rawStatus)}'. Must be one of: ${PLU
• "tool" is required and must be a string
• "runContext" is required and must be an object
• "runContext" must include agentId, runId, companyId, and pro

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-10). Data as JSON: /api/errors/eedb6866557284a2. Report an issue: GitHub.