paperclipai/paperclip · error · Error

Chat Evalbook must not mix in legacy plain attempt pages

Error message

Chat Evalbook must not mix in legacy plain attempt pages

What it means

The public protocol-eval publisher validates that a report intended for public hosting uses only the current Chat Evalbook format. When the report contains the new rich chat attempt pages (attempts/<id>/index.html), the publisher loads the trusted viewer build, and legacy flat attempt pages (attempts/<id>.html) are forbidden so the published site never mixes old and new attempt renderings. This guard runs at the end of validatePublicProtocolEvalReport after viewer assets are checked.

Solutions

1. Clean the report output directory and regenerate it fully from the current eval runner so only attempts/<id>/index.html pages exist
2. Delete any legacy attempts/<id>.html files (flat files, not directories) from reportRoot before publishing
3. Check the generator version that produced the report; regenerate with a toolchain that no longer emits plain attempt pages
4. If legacy pages are needed, publish them as a separate campaign rather than mixing them into the chat Evalbook

Example fix

// before: stale legacy file remains in report
// attempts/gha-123-45.html  (old flat page)
// attempts/gha-123-45/index.html (new rich page)

// after: remove the legacy flat page before publishing
rm reports/runner-protocol-eval-public-report/attempts/*.html

const files = fs.readdirSync(reportRoot, { recursive: true });
const hasLegacy = files.some((f) => /^attempts\/[^/]+\.html$/.test(f.replaceAll("\\", "/")));
if (hasLegacy) throw new Error("remove legacy attempts/<id>.html pages before publishing");

const isLegacyAttemptPage = (f) => /^attempts\/[^/]+\.html$/.test(f);
const hasLegacyAttempts = (files) => files.some(isLegacyAttemptPage);

try {
  await publishProtocolEvalHistory({ reportRoot, destination, viewerRoot });
} catch (e) {
  if (String(e.message).includes("legacy plain attempt pages")) {
    for (const f of listLegacyAttemptPages(reportRoot)) fs.rmSync(path.join(reportRoot, f));
    // retry once
  } else throw e;
}

When it happens

Trigger: Calling validatePublicProtocolEvalReport or publishProtocolEvalHistory with a reportRoot whose files include both attempts/<id>/index.html (chat-rich attempts, which force viewer loading) and legacy attempts/<id>.html pages from an older report generator.

Common situations: Stale report directory left over from a previous version of the eval generator merged with newly generated chat attempt pages; a report rebuilt incrementally where old attempt HTML files were never cleaned out; hand-copying files from an old public report into a new one.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• Report campaign ID does not match publication target
• A full lowercase source SHA is required.
• A reusable lease cannot be replaced and reacquired in the…
• A reusable lease handoff requires an execution workspace…
• A safe, unique --revision is required

AI-assisted analysis of paperclipai/paperclip@01ad858492 (2026-09-10). Data as JSON: /api/errors/eedb6866557284a2. Report an issue: GitHub.

Appendix: source

Thrown at packages/paperclip-runner/scripts/publish-runner-protocol-eval-history.mjs:228

        ...clean.split("/"),
      );
      const rel = relative(root, target).split(sep).join("/");
      if (
        !isPublicProtocolEvalPath(rel) ||
        !(await lstat(target).catch(() => null))?.isFile()
      ) {
        throw new Error(
          `Public report contains a broken or unsafe link in ${file}: ${href}`,
        );
      }
    }
  }
  if (viewer) {
    for (const file of viewer.files.keys())
      if (!files.includes(file))
        throw new Error(`Missing public viewer asset: ${file}`);
    if (files.some((file) => /^attempts\/[^/]+\.html$/.test(file)))
      throw new Error(
        "Chat Evalbook must not mix in legacy plain attempt pages",
      );
  }
  const campaign = await loadObject(join(root, "campaign.json"));
  if (
    campaign.schema !== "paperclip.runner-protocol-eval.campaign/v1" ||
    !SAFE_CAMPAIGN.test(String(campaign.campaignId ?? ""))
  ) {
    throw new Error("Public report campaign metadata is invalid");
  }
  return { files, campaign };
}

export async function createProtocolEvalBundleManifest(
  reportRoot,
  campaignId,
  { viewerRoot } = {},
) {

View on GitHub (pinned to 01ad858492)